Optimizing Cloud Infrastructure Cost and Performance with Starburst on AWS
AWS Architecture BlogOptimizing Cloud Infrastructure Cost and Performance with Starburst on AWSby Kinnar Kumar Sen, Kyle Payne, Shaun Van Staden, and Antony Prasad Thevaraj | on17 SEP 2021| inAmazon EC2,Amazon Elastic Kubernetes Service,Architecture,Auto Scaling,AWS CloudFormation,Customer Solutions|Permalink|ShareAmazon Web Services (AWS) Cloud is elastic, convenient to use, easy to consume, and makes it simple to onboard workloads. Because of this simplicity, the cost associated with onboarding workloads is sometimes overlooked.There is a notion that when an organization moves its workload to the cloud, agility, scalability, performance, and cost issues will disappear. While this may be true for agility and scalability, you must optimize your workload. You can do this with services likeAmazon EC2 Auto ScalingviaAmazon Elastic Compute Cloud (Amazon EC2)andAmazon EC2 Spot Instancesto realize the performance and cost benefits the cloud offers.In this blog post, we show you howStarburst Enterprise (Starburst)addressed a sudden increase in cost for their data analytics platform as they grew their internal teams and scaled out their infrastructure. After reviewing their architecture and deployments with AWS specialist architects, Starburst and AWS concluded that they could take the following steps to greatly reduce costs:Use Spot Instances to run workloads.Add Amazon EC2 Auto Scaling into their training and demonstration environments, because the Starburst platform is designed toelastically scale up and down.For analytics workloads, when you rein in costs, you typically rein in performance. Starburst and AWS worked together to balance the cost and performance of Starburst’s data analytics platform while also harnessing the flexibility, scalability, security, and performance of the cloud.What is Starburst Enterprise?Starburst provides a Massively Parallel Processing SQL (MPPSQL) engine based on open sourceTrino. It is an analytics platform that provides the cornerstone in customers’ intelligent data mesh and offers the following benefits and services:The platform gives you asingle point to access, monitor, and secure your data mesh.The platform gives you options for your data compute. You no longer have to wait on data migrations or extract, transform, and load (ETL), there is no vendor lock-in, and there isno need to swap out your existing analytics tools.Starburst Stargate (Stargate)ensures that large jobs are completed within each data domain of your data mesh. Only the result set is retrieved from the domain.Stargate reduces data output, which reduces costs and increases performance.Data governance policiescan also be applied uniquely in each data domain, ensuring security compliance and federation.As shown in Figure 1, there are many connectors for input and output that ensure you experience improved performance and security.Figure 1. Starburst platformIntegrating Starburst Enterprise with AWSAs shown in Figure 2, Starburst Enterprise uses AWS services to deliver elastic scaling and optimize cost. The platform is architected with decoupled storage and compute. This allows the platform to scale as needed to analyze petabytes of data.The platform can be deployed viaAWS CloudFormationorAmazon Elastic Kubernetes Service (Amazon EKS). Starburst on AWS allows you to run analytic queries across AWS data sources and on-premises systems such as Teradata and Oracle.Figure 2. Deployment architecture of Starburst platform on AWSAmazon EC2 Auto ScalingEnterprises have diverse analytic workloads; their compute and memory requirements vary with time. Starburst uses Amazon EKS and Amazon EC2 Auto Scaling to elastically scale compute resources to meet the demands of their analytics workloads.Amazon EC2 Auto Scaling ensures that you have the compute capacity for your workloads to handle the load elastically. It is used to architect sophisticated, elastic, and resilient applications on the AWS Cloud.Starburst uses the scheduled scaling feature of Amazon EC2 Auto Scaling to scale the cluster up/down based on time. Thus, they incur no costs when the cluster is not in use.Amazon EKS is a fully managed Kubernetes service that allows you to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane.Scaling cloud resource consumption on demand has a major impact on controlling cloud costs. Starburst supports scaling down elastically, which means removing compute resources doesn’t impact the underlying processes.Amazon EC2 Spot InstancesSpot Instances let you take advantage of unused EC2 capacity in the AWS Cloud. They are available at up to a 90% discount compared to On-Demand Instance prices. If EC2 needs capacity for On-Demand Instance usage, Spot Instances can be interrupted by Amazon EC2 with a two-minute notification. There are many ways to handle the interruption to ensure that the application is well architected for resilience and fault tolerance.Starburst has integrated Spot Instances as a part of theAmazon EKS managed node groupsto cost optimize the analytics workloads. This best practice ofinstance diversificationis implemented by using the integration eksctl and instance selector with dry-run flag. This creates a list of instances of same size (vCPU/Mem ratio) and uses them in the underlying node groups.Same size instances are required to make best use ofKubernetes Cluster Autoscaler, which is used to manage the size of the cluster.Scaling down, handling interruptions, and provisioning compute“Scaling in” an active application is tricky, but Starburst was built with resiliency in mind, and it can effectively manage shut downs.Spot Instances are an ideal compute option because Starburst can handle potential interruptions natively. Starburst also uses Amazon EKS managed node groups to provision nodes in the cluster. This requires significantly less operational effort compared to using self-managed node groups. It allows Starburst to enforce best practices likecapacity optimized allocation strategy,capacity rebalancing, andinstance diversification.When you need to “scale out” the deployment, Amazon EKS and Amazon EC2 Auto Scaling help to provision capacity, as depicted in Figure 3.Figure 3. Depicting “scale out” in a Starburst deploymentBenefits realized from using AWS servicesIn a short period, Starburst was able to increase the number of people working on AWS. They added five times the number of Solutions Architects, they have previously. Additionally, in their initial tests of their new deployment architecture, their Solutions Architects were able to complete up to three times the amount of work than they had been able to previously. Even after the workload increased more than 15 times, with two simple changes they only had a slight increase in total cost.This cost and performance optimization allows Starburst to be more productive internally and realize value for each dollar spent. This further justified investing more into building out infrastructure footprint.ConclusionIn building their architecture with AWS, Starburst realized the importance of having a robust and comprehensive cloud administration plan that they can implement and manage going forward. They are now able to balance the cloud costs with performance and stability, even after considering the SLA requirements. Starburst is planning to teach their customers about the Spot Instance and Amazon EC2 Auto Scaling best practices to ensure they maintain a cost and performance optimized cloud architecture.If you want to see the Starburst data analytics platform in action, you can get a free trial in the AWS Marketplace:Starburst Data Free Trial.TAGS:Customer SolutionsKinnar Kumar SenKinnar Kumar Sen is a Sr. Solutions Architect at Amazon Web Services (AWS) focusing on Big Data and Analytics. As a part of the EC2 Spot team, he works with customers to cost optimize the big data workloads running on AWS. Kinnar has more than 15 years of industry experience working in research, consultancy, engineering, and architecture.Kyle PayneKyle Payne is a Solutions Architect at Starburst Data. As Technical Architect, Kyle spent a little over 20 years in post-sales delivery and consulting. Near the end of that time, he took on the role of Technical Lead to enable a team of Delivery Engineers. A little over 9 years ago he moved into pre-sales and is currently a Solutions Architect. Whether in post-sales or pre-sales, his career has been first and foremost about customer success.Shaun Van StadenShaun Van Staden, Partner Solutions Architect at Starburst Data, is a seasoned Solutions Architect, consultant, and technology professional with 22 years of software industry experience working in and with business development, sales, pre-sales, professional services, engineering, product management, and product marketing teams.Antony Prasad ThevarajAntony Prasad Thevaraj is a Sr. Partner Solutions Architect in Data and Analytics at AWS. He has over 10 years of experience as a Big Data Engineer, and has worked on building complex ETL and ELT pipelines for various business units. He helps partners build joint solutions using AWS services.
Amazon Web Services (AWS) Cloud is elastic, convenient to use, easy to consume, and makes it simple to onboard workloads. Because of this simplicity, the cost associated with onboarding workloads is sometimes overlooked. There is a notion that when an organization moves its workload to the cloud, agility, scalability, performance, and cost issues will disappear. […]
Kinnar Kumar Sen
Amazon EC2 Amazon Elastic Kubernetes Service Architecture Auto Scaling AWS CloudFormation Customer Solutions