Field Notes: Migrating a Self-managed Kubernetes Cluster on Amazon EC2 to Amazon EKS
AWS Architecture BlogField Notes: Migrating a Self-managed Kubernetes Cluster on Amazon EC2 to Amazon EKSby Ahmed Bham and Marcelo Boeira | on20 OCT 2020| inAmazon EC2,Amazon Elastic Kubernetes Service,Architecture,Technical How-to|Permalink|ShareAWS customers from startups to enterprises have been successfully running Kubernetes clusters on Amazon EC2 instances since 2015, well beforeAmazon Elastic Kubernetes Service(Amazon EKS), was launched in 2018. As a fully managed Kubernetes service, Amazon EKS customers can run Kubernetes on AWS without needing to install, operate, and maintain their own Kubernetes control plane. Since its launch, many existing and new customers are building and running their Kubernetes clusters on Amazon EKS.At re:Invent 2019, AWS announcedAWS Fargate for Amazon EKS, which is serverless compute for containers. Then, in January 2020, AWS announced a price reduction of Amazon EKS  by 50% to $0.10 per hour, per cluster. These developments, coupled with realization of management and cost overhead of Kubernetes control plane operations at scale, made more customers look into migrating their self-managed Kubernetes clusters to Amazon EKS.The “how” of this migration is the focus of this blog.Overview of SolutionFor most customers migrating from self-managed Kubernetes clusters on Amazon EC2 to Amazon EKS can usually be accomplished with minimal or no downtime. However, for large clusters involving hundreds of nodes and thousands of pods, this requires more planning and testing, and it is recommended to engage AWS Support for guidance.ConsiderationsThere are certain considerations to ensure a successful Amazon EKS migration and operational excellence.SecurityAccess ControlAmazon EKS uses AWS Identity and Access Management (IAM) to provide authentication to your Kubernetes cluster, but it still relies on native KubernetesRole Based Access Control(RBAC) for authorization. It’s important to plan for the creation and governance of IAM users, roles, or groups, for Kubernetes cluster administration.You canenable private access to the Kubernetes API serverso that all communication between your nodes and the API server stays within your VPC. You can limit the IP addresses that can access your API server from the internet, or completely disable internet access to the API server.IAM Role for Service AccountWithIAM roles for service accountson Amazon EKS clusters, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. With this feature, you no longer need to provide extended permissions to the node IAM role so that pods on that node can call AWS APIs.Security groups for podsSecurity groups for podsintegrate Amazon EC2 security groups with Kubernetes pods. You can use Amazon EC2 security groups to define rules that allow inbound and outbound network traffic to and from pods. These pods are deployed to nodes running on many Amazon EC2 instance types.NetworkingAmazon EKS supports native VPC networking with the Amazon VPC Container Network Interface (CNI) plugin for Kubernetes. Using this plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.VPC CNI plugin uses IP addresses for the pods from the VPC CIDR ranges, and specifically from the subnet where the worker node is hosted. Therefore, customers must ensure the VPC and subnets that will host their Amazon EKS cluster have sufficient IP addresses available for the expected number of pods running at a time. Additionally, IP addresses are allocated to theElastic Network Interface(ENI) attached to the EC2 instances.  The EC2 instance selection for the worker nodes should take into account thenumber of ENI attachments supported by the instance type.Compute OptionsAn Amazon EKS cluster can schedule pods on any combination ofself-managed nodes,managed node groups, andAWS Fargate. Thistablein the Amazon EKS User Guide provides several criteria to evaluate when deciding which option best meets your requirements.Kubernetes VersionsAmazon EKS supports four major Kubernetes versions at a time,  which you can review in the availableAWS documentation, along with a calendar for future Amazon EKS releases.If you are currently running a non-supported Kubernetes cluster, or would like to migrate to a newer version on Amazon EKS, consider the following:Reviewrelease notesfor specific Kubernetes version you want to migrate to, and make necessary updates to Kubernetes manifest files.Update your Kubernetes add-ons (CNI plugin, CoreDNS, Kube-Proxy) to compatible versions, as listed in theUpdating an Amazon EKS Cluster guide.Prerequisites1. Create an IAM Role for the creator and/or administrator of the Amazon EKS cluster. Specify this role when creating the Amazon EKS cluster.2. If using an existing VPC and subnets to host Amazon EKS cluster:You will need subnets in at least two Availability ZonesAll public subnets should have the propertyMapPublicIpOnLaunch enabled(that is, Auto-assign public IPv4 address in the AWS Management Console) to host self-managed and managed node groups.3. If your pods are currently accessing AWS resources, and if you would like to use IAM roles for service accounts, then:Create service accounts in Kubernetes to be used by your pods.Follow thesestepsto create IAM roles, and assign to service account created.Update your pod manifest files to specify the newly created service account and role ARN annotation. Remove any existing code for storing or passing IAM credentials.4. If you are planning to useAWS Fargateto run your pods, you need to create the appropriateFargate Profileandpod execution role.Application and Data MigrationFor stateless workloads, apply your resource definitions (YAML or Helm) to the new cluster, and make sure everything works as expected. This includes the connection to resources external to the cluster.For stateful workloads:You will need to carefully plan your migration to avoid data loss or unexpected downtime.If you are currently using shared persistent file storage based onAmazon EFSorAmazon FSx for Lustre, they can be mounted to Amazon EKS pods concurrently. Just make sure that pods don’t write to the same files concurrently.For pods using EBS volumes, and for other persistent storage types, you can use a Kubernetes backup and restore tool,Velero.Traffic MigrationIf you have an entire domain migration that you would like to smoothly migrate, you can take advantage of Amazon Route 53’s Weighted Routing (as shown in the following diagram). With Weighted Routing, you are able to have a progressive transition from your existing cluster to the new one with zero downtime by splitting the traffic at the DNS level.Your customers are slowly being transferred to your new cluster as their cached TTL expires. The split could start with a small share of your customers, for example, 10% being pointed to the new Amazon EKS cluster and 90% still on the old one. As soon as traffic is confirmed to be working as expected on the new cluster, that percentage of clients pointed to the new one can be increased.This implementation is flexible, it can be tied to Load Balancers, EC2 Instances, and even to external on-premises infrastructure.ConclusionIn this blog post, we showed how to migrate your live-traffic serving self-hosted Kubernetes Cluster to Amazon EKS. Amazon EKS offers a cost-effective and highly available option for running Kubernetes clusters in the cloud. Since Amazon EKS is upstream Kubernetes compliant, you can migrate existing self-managed Kubernetes workloads to Amazon EKS, with multiple options to minimize or avoid service disruption. To create your first Amazon EKS cluster, visitGetting started with Amazon EKS.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Field NotesAhmed BhamAhmed Bham is a Solutions Architect with AWS. He works with AWS Startup customers to help them architect the most cost-effective, secure, reliable, and performant solutions on AWS. In his free time, he enjoys running while listening to non-fiction Audible books.Marcelo BoeiraMarcelo Boeira is a Solutions Architect at AWS, working on scaling Startups. Before joining AWS, Marcelo worked as a Site Reliability Engineer, handling infrastructure scalability and operations for almost a decade. Marcelo is passionate about distributed systems and engineering culture.
AWS customers from startups to enterprises have been successfully running Kubernetes clusters on Amazon EC2 instances since 2015, well before Amazon Elastic Kubernetes Service (Amazon EKS), was launched in 2018. As a fully managed Kubernetes service, Amazon EKS customers can run Kubernetes on AWS without needing to install, operate, and maintain their own Kubernetes control […]
Ahmed Bham
Amazon EC2 Amazon Elastic Kubernetes Service Architecture Technical How-to