Optimize your modern data architecture for sustainability: Part 2 – unified data governance, data movement, and purpose-built analytics
AWS Architecture BlogOptimize your modern data architecture for sustainability: Part 2 – unified data governance, data movement, and purpose-built analyticsby Sam Mokhtari, Dr. Ali Khoshkbar, and Sandipan Bhaumik | on25 NOV 2022| inAmazon Redshift,Amazon Simple Storage Service (S3),Architecture,AWS Glue,AWS Well-Architected,Sustainability|Permalink|ShareIn the first part of this blog series,Optimize your modern data architecture for sustainability: Part 1 – data ingestion and data lake, we focused on the 1) data ingestion, and 2) data lake pillars of themodern data architecture. In this blog post, we will provide guidance and best practices to optimize the components within the 3) unified data governance, 4) data movement, and 5) purpose-built analytics pillars.Figure 1 shows the different pillars of the modern data architecture. It includes data ingestion, data lake, unified data governance, data movement, and purpose-built analytics pillars.Figure 1. Modern Data Analytics Reference Architecture on AWS3. Unified data governanceA centralized Data Catalog is responsible for storing business and technical metadata about datasets in the storage layer. Administrators apply permissions in this layer and track events for security audits.Data discoveryTo increase data sharing and reduce data movement and duplication, enable data discovery and well-defined access controls for different user personas. This reduces redundant data processing activities. Separate teams within an organization can rely on this central catalog. It provides first-party data (such as sales data) or third-party data (such as stock prices, climate change datasets). You’ll only need access data once, rather than having to pull from source repeatedly.AWS GlueData Catalog can simplify the process for adding and searching metadata. Use AWS Glue crawlers to update the existing schemas and discover new datasets. Carefully plan schedules to reduce unnecessary crawling.Data sharingEstablish well-defined access control mechanisms for different data consumers using services such asAWS Lake Formation. This will enable datasets to be shared between organizational units with fine-grained access control, which reduces redundant copying and movement. UseAmazon Redshift data sharingto avoid copying the data across data warehouses.Well-defined datasetsCreate well-defined datasets and associated metadata to avoid unnecessary data wrangling and manipulation. This will reduce resource usage that might result from additional data manipulation.4. Data movementAWS Glue provides serverless,pay-per-usedata movement capability, without having to stand up and manage servers or clusters. Set up ETL pipelines that can process tens of terabytes of data.To minimize idle resources without sacrificing performance, useauto scaling for AWS Glue.You can create and share AWS Glue workflows for similar use cases by usingAWS Glue blueprints, rather than creating an AWS Glue workflow foreachuse case.AWS Glue job bookmarkcan track previously processed data.Consider usingGlue Flex Jobsfor non-urgent or non-time sensitive data integration workloads such as pre-production jobs, testing, and one-time data loads. With Flex, AWS Glue jobs run on spare compute capacity instead of dedicated hardware.Joins between several dataframes is a common operation in Spark jobs. To reduce shuffling of data between nodes, usebroadcast joinswhen one of the merged dataframes is small enough to be duplicated on all the executing nodes.The latestAWS Glue versionprovides more new and efficient features for your workload.5. Purpose-built analyticsData Processing modesReal-time data processing options need continuous computing resources and require more energy consumption. For the most favorable sustainability impact, evaluate trade-offs and choose the optimal batch data processing option.Identify the batch and interactive workload requirements and designtransient clustersinAmazon EMR. Using Spot Instances and configuringinstance fleetscan maximize utilization.To improve energy efficiency,Amazon EMR Serverlesscan help you avoid over- or under-provisioning resources for your data processing jobs. Amazon EMR Serverless automatically determines the resources that the application needs, gathers these resources to process your jobs, and releases the resources when the jobs finish.Amazon Redshift RA3 nodescan improve compute efficiency. With RA3 nodes, you can scale compute up and down without having to scale storage. You can chooseAmazon Redshift Serverlessto intelligently scale data warehouse capacity. This will deliver faster performance for the most demanding and unpredictable workloads.Energy efficient transformation and data model designData processing and data modeling best practices can reduce your organization’s environmental impact.To avoid unnecessary data movement between nodes in an Amazon Redshift cluster, followbest practices for table design.You can also useautomatic table optimization (ATO)for Amazon Redshift to self-tune tables based on usage patterns.Use the EXPLAIN feature inAmazon AthenaorAmazon Redshiftto tune and optimize the queries.TheAmazon Redshift Advisorprovides specific, tailored recommendations to optimize the data warehouse based on performance statistics and operations data.Consider migrating Amazon EMR orAmazon OpenSearch Serviceto a more power-efficient processor such asAWS Graviton.AWS Graviton 3delivers 2.5–3 times better performance over other CPUs. Graviton 3-based instances use up to 60% less energy for the same performance than comparable EC2 instances.Minimize idle resourcesUse auto scaling features inEMR Clustersor employAmazon Kinesis Data Streams On-Demandto minimize idle resources without sacrificing performance.AWS Trusted Advisorcan help you identify underutilizedAmazon Redshift Clusters. Pause Amazon Redshift clusters when not in use and resume when needed.Energy efficient consumption patternsConsider querying the data in place with Amazon Athena orAmazon Redshift Spectrumfor one-off analysis, rather than copying the data to Amazon Redshift.Enable acaching layer for frequent queriesas needed. This is in addition to the result caching that comes built-in with services such as Amazon Redshift. Also, useAmazon Athena Query Result Reusefor every query where the source data doesn’t change frequently.Usematerialized viewscapabilities available in Amazon Redshift orAmazon AuroraPostgres to avoid unnecessary computation.Use federated queries across data stores powered byAmazon Athena federated queryorAmazon Redshift federated queryto reduce data movement. For querying across separate Amazon Redshift clusters, consider usingAmazon Redshift data sharingfeature that decreases data movement between these clusters.Track and assess improvement for environmental sustainabilityThe optimal way to evaluate success in optimizing your workloads for sustainability is to useproxy measures and unit of work KPI. This can be GB per transaction for storage, or vCPU minutes per transaction for compute.In Table 1, we list certain metrics you could collect on analytics services as proxies to measure improvement. These fall under each pillar of the modern data architecture covered in this post.PillarMetricsUnified data governanceCloudTrail events– for monitoring crawler runs and job runsData movementDPU-hourfor AWS Glue jobsPurpose-built AnalyticsRedshift cluster performance data– CPUUtilization, percentage disk space used, read throughput, write throughput, query duration, query throughputRedshift query history(optimize queries) – query runtime, CPUUtilization, storage capacity usedAmazon Redshift Spectrum queries– System views: SVL_S3QUERY, SVL_S3QUERY_SUMMARYCloudWatch metrics for Amazon EMR– IsIdle, HDFSUtilization, S3BytesRead, S3BytesWrittenCloudWatch metrics for Amazon OpenSearch(Cluster metrics) – CPUUtilization, FreeStorageSpace, ClusterUsedSpace, JVMMemoryPressure, DiskThroughputThrottleCloudWatch metrics for Amazon Athena– ProcessedBytes, QueryQueueTime, TotalExecutionTimeCloudWatch metrics for Amazon SageMaker– CPUUtilization, GPUUtilization, GPUMemoryUtilization, MemoryUtilization, and DiskUtilizationKinesis Data Analytics application metrics– CPUUtilization, containerCPUUtilization, containerDiskUtilization, idleTimeMsPerSecondTable 1. Metrics for the Modern data architecture pillarsConclusionIn this blog post, we provided best practices to optimize processes under the unified data governance, data movement, and purpose-built analytics pillars of modern architecture.If you want to learn more, check out theSustainability Pillar of the AWS Well-Architected Frameworkand other blog posts onarchitecting for sustainability.If you are looking for more architecture content, refer to theAWS Architecture Centerfor reference architecture diagrams, vetted architecture solutions,Well-Architectedbest practices, patterns, icons, and more.Sam MokhtariDr. Sam Mokhtari is leading the sustainability pillar of AWS Well-Architected framework. His main area of depth is data and analytics, and he has published more than 30 influential articles in this field.Dr. Ali KhoshkbarDr. Ali Khoshkbar is a Senior Cloud Architect with Amazon Web Services (AWS) Professional Services. He is passionate about helping customers build scalable and high-performance data analytics solutions in the cloud.Sandipan BhaumikSandipan Bhaumik is a Senior Analytics Specialist Solutions Architect based in London. He helps customers modernise their traditional data platforms by building modern data architecture in the cloud to perform analytics at scale.
In the first part of this blog series, Optimize your modern data architecture for sustainability: Part 1 – data ingestion and data lake, we focused on the 1) data ingestion, and 2) data lake pillars of the modern data architecture. In this blog post, we will provide guidance and best practices to optimize the components […]
Sam Mokhtari
Amazon Redshift Amazon Simple Storage Service (S3) Architecture AWS Glue AWS Well-Architected Sustainability