Reduce Operational Load using AWS Managed Services for your Data Solutions
AWS Architecture BlogReduce Operational Load using AWS Managed Services for your Data Solutionsby Andrew Chen, Dave Geyer, and Jerry Mullis | on23 APR 2021| inAmazon RDS,Amazon Redshift,Architecture,AWS Glue,AWS Lake Formation,Intermediate (200)|Permalink|ShareAs the volume of customers’ data grows, companies are realizing the benefits that data has for their business.Amazon Web Services (AWS)offers many database and analytics services, which give companies the ability to build complex data management workloads. At the same time, these services can reduce the operational overhead compared to traditional operations.Using core managed AWS analytics services will enable you to spend more time focusing onhowto use your data, and less time managing and maintaining the underlying infrastructure. Let’s review the core data and analytics services you can use to support these needs.Scale and Update Your Transactional WorkloadsMany companies have workloads that require thousands of writes to a database. These workloads often result in large numbers ofinserts,updates, anddeletes,which can be input/output intensive. As these workloads increase, self-managed databases require maintenance, backups, patching, and hardware provisioning to keep up with this demand.Amazon Relational Database Service (RDS)allows you to set up and quickly scale familiar database engines such as MySQL, PostgreSQL, and Microsoft SQL Server. At the same time, it alleviates database administrators from time-consuming operations.Figure 1. Amazon RDS Database Engine SelectionMoving Data from Transactional Databases to Data LakesAmazon RDS supports online transactional processing databases (OLTP), which efficiently handle create, read, update, and delete (CRUD) operations. Moving your data from a transactional database to a data lake will reduce storage stress on the transactional database and optimize computation-heavy analytics queries. This also will provide you with a centralized repository capable of combining mixed data structures from different data sources for analytics and storage.AWS Lake Formationmakes it possible to set up a data lake in a matter of days, and is built onAmazon Simple Storage Service (Amazon S3). AWS Lake Formation allows customers to integrate their centralized storage with most database engines supported by RDS through AWS Lake Formation’s blueprints feature. By using AWS Lake Formation to create a data lake, customers have the flexibility to hydrate their data lake with a full copy of the database through a database snapshot, or through incremental data loads.Figure 2. AWS Lake Formation Blueprints SelectionExtracting, Transforming, and Loading DataStoring data in a data lake provides the flexibility to store mixed data structures from disparate sources. This data may require additional processing such as standardization and normalization, for use in AWS storage and analytics services.The ability for customers to catalog and process their data ensures accuracy, reliability, and consistency of this data across multiple data stores.Customers can take advantage ofAWS Glueto catalog, process, and combine their data. AWS Glue is a serverless data integration service, and provides visual and code-based interfaces for data integration. Customers can automatically detect and discover metadata and data schemas across different data sources, by using theAWS Glue Crawlerfeature. You can use theAWS Glue Jobsfeature to automate your Extract, Transform, and Load (ETL) workloads. Use AWS Glue, when your goal is to catalog and organize many data stores, or to use the Apache Spark framework for your ETL workloads.Figure 3. AWS Glue Job VisualizerCustomers can also useAWS Lambdato write and develop custom ETL scripts in languages, such as Node.js and Python, while maintaining control of the underlying runtime environment. AWS Lambda is a serverless compute service that allows users to run code without provisioning servers. Lambda can run code for almost any workload and allows for importing of custom packages. If you run your ETL workloads with common Python modules such as Pandas and numPy, you can package and include these modules in your AWS Lambda environment. Use AWS Lambda when you have smaller workloads and require more flexibility in building out your code and have custom package dependencies.Figure 4. Selection of AWS Lambda Runtime EnvironmentData Warehouse ServicesCustomers rely on reports, dashboards, and analytics tools to extract insights from their data, monitor business performance, and support decision making. Customers can use a data warehouse to improve the performance of computation-heavy analytics requirements across petabytes of data.Amazon Redshiftis a fully managed data warehouse that provides exceptional performance for analytics queries. Amazon Redshift reduces the operational overhead required with traditional data warehouses by automating tasks such as patching, backups, and hardware provisioning. You can configure an Amazon Redshift cluster where you can customize the infrastructure and performance baselines for your data warehouse. Amazon Redshift also provides Redshift Spectrum, which allows you to use Amazon Redshift cluster resources to query and join data with Amazon S3.Running Queries on Data in Data LakesWhen storing data on Amazon S3, customers benefit from cost efficiency and durability. You can use standard SQL to query data within Amazon S3 without having to preprocess or index the data for analysis. This allows customers to reap the cost savings that Amazon S3 provides, in addition to deriving insights without moving the data.You can run queries usingAmazon Athena, which is a serverless query service that uses standard SQL to query data in Amazon S3. There is no infrastructure to manage with Amazon Athena. You can take advantage of high-performance queries while realizing storage cost savings associated with the data lake built on Amazon S3. Because Amazon Athena is serverless, you only pay for the resources consumed. You are charged based on the amount of data that is scanned with each query.In the situation where you have provisioned an Amazon Redshift data warehouse,Amazon Redshift Spectrumis an additional service that queries data in Amazon S3. Redshift Spectrum is associated with an Amazon Redshift cluster and provides more control over performance. If you want to increase query performance, you can allocate additional compute resources. The cost of Redshift Spectrum is similar to that of Athena, in which you are charged for the amount of data that is scanned.Putting it all TogetherUsing the concepts discussed in this blog post, customers can implement the following architecture to build the foundation of a data platform. This platform will reduce your operational load by using managed services in AWS:Figure 5. The Full Architectural DiagramTo learn more about analytics services on AWS visit:Analytics on AWSTAGS:data architecture,data lakes,data warehousing,ETL,federated queryAndrew ChenAndrew Chen is an Associate Solutions Architect with an interest in data analytics, machine learning, and virtualization of infrastructure. Andrew has previous experience in management consulting in which he worked as a technical lead for various cloud migration projects. In his free time, Andrew enjoys fishing, hiking, kayaking, and keeping up with financial markets.Dave GeyerDave Geyer is an Associate Solutions Architect at AWS. He has a background in data management and organizational design, and is interested in data analytics and infrastructure security. Dave has advised and worked for customers in the commercial and public sectors, providing them with architectural best practices and recommendations. Dave is interested in the aerospace and financial services industries. Outside of work, he is an adrenaline junkie, and is passionate about mountaineering and high altitudes.Jerry MullisJerry Mullis is an Associate Solutions Architect at AWS. His interests are in data migration, machine learning, and device automation. Jerry has previous experience in machine learning research and healthcare management. Jerry’s certifications include: AWS Solutions Architect Pro, AWS Developer Associate, AWS Sysops Admin Associate and AWS Certified Cloud Practitioner. In his free time, Jerry enjoys hiking, playing basketball and spending time with his wife.
As the volume of customers’ data grows, companies are realizing the benefits that data has for their business. Amazon Web Services (AWS) offers many database and analytics services, which give companies the ability to build complex data management workloads. At the same time, these services can reduce the operational overhead compared to traditional operations. Using […]
Andrew Chen
Amazon RDS Amazon Redshift Architecture AWS Glue AWS Lake Formation Intermediate (200)