Optimize your modern data architecture for sustainability: Part 1 – data ingestion and data lake
AWS Architecture BlogOptimize your modern data architecture for sustainability: Part 1 – data ingestion and data lakeby Sam Mokhtari, Dr. Ali Khoshkbar, and Sandipan Bhaumik | on04 NOV 2022| inAmazon Redshift,Amazon Simple Storage Service (S3),Architecture,AWS Glue,AWS Well-Architected,Sustainability|Permalink|ShareThe modern data architecture on AWS focuses on integrating a data lake and purpose-built data services to efficiently build analytics workloads, which provide speed and agility at scale. Using the right service for the right purpose not only provides performance gains, but facilitates the right utilization of resources. ReviewModern Data Analytics Reference Architecture on AWS, see Figure 1.In this series of two blog posts, we will cover guidance from theSustainability Pillarof theAWS Well-Architected Frameworkon optimizing your modern data architecture for sustainability. Sustainability in the cloud is an ongoing effort focused primarily on energy reduction and efficiency across all components of a workload. This will achieve the maximum benefit from the resources provisioned and minimize the total resources required.Modern data architecture includes five pillars or capabilities: 1) data ingestion, 2) data lake, 3) unified data governance, 4) data movement, and 5) purpose-built analytics. In the first part of this blog series, we will focus on the data ingestion and data lake pillars of modern data architecture. We’ll discuss tips and best practices that can help you minimize resources and improve utilization.Figure 1. Modern Data Analytics Reference Architecture on AWS1. Data ingestionThe data ingestion process in modern data architecture can be broadly divided into two main categories: batch, and real-time ingestion modes.To improve the data ingestion process, see the following best practices:Avoid unnecessary data ingestionWork backwards from your business needs and establish the right datasets you’ll need. Evaluate if you can avoid ingesting data from source systems by using existing publicly available datasets inAWS Data ExchangeorOpen Data on AWS. Using these cleaned and curated datasets will help you to avoid duplicating the compute and storage resources needed to ingest this data.Reduce the size of data before ingestionWhen you design your data ingestion pipelines, use strategies such as compression, filtering, and aggregation to reduce the size of ingested data. This will permit smaller data sizes to be transferred over network and stored in the data lake.To extract and ingest data from data sources such as databases, usechange data capture(CDC) or date range strategies instead of full-extract ingestion. UseAWS Database Migration Service (DMS) transformation rulesto selectively include and exclude the tables (from schema) and columns (from wide tables, for example) for ingestion.Consider event-driven serverless data ingestionAdopt an event-driven serverless architecture for your data ingestion so it only provisions resources when work needs to be done. For example, when you useAWS Glue jobsandAWS Step Functionsfor data ingestion and pre-processing, you pass the responsibility and work of infrastructure optimization to AWS.2. Data lakeAmazon Simple Storage Service (S3)is an object storage service which customers use to store any type of data for different use cases as a foundation for a data lake. To optimize data lakes on Amazon S3, follow these best practices:Understand data characteristicsUnderstand the characteristics, requirements, and access patterns of your workload data in order to optimally choose the right storage tier. You can classify your data into categories shown in Figure 2, based on their key characteristics.Figure 2. Data CharacteristicsAdopt sustainable storage optionsBased on your workload data characteristics, use the appropriate storage tier to reduce the environmental impact of your workload, as shown in Figure 3.Figure 3. Storage tiering on Amazon S3Implement data lifecycle policies aligned with your sustainability goalsBased on your data classification information, you can move data to more energy-efficient storage or safely delete it. Manage the lifecycle of all your data automatically usingAmazon S3 Lifecycle policies.Amazon S3 Storage Lensdelivers visibility into storage usage, activity trends, and even makes recommendations for improvements. This information can be used to lower the environmental impact of storing information on S3.Select efficient file formats and compression algorithmsUse efficient file formats such asParquet, where a columnar format provides opportunities for flexible compression options and encoding schemes. Parquet also enables more efficient aggregation queries, as you can skip over the non-relevant data. Using an efficient way of storage and accessing data is translated into higher performance with fewer resources.Compress your data to reduce the storage size. Remember, you will need to trade off compression level (storage saved on disk) against the compute effort required to compress and decompress. Choosing the right compression algorithm can be beneficial as well. For instance,ZStandard (zstd)provides a better compression ratio compared withLZ4orGZip.Use data partitioning and bucketingPartitioning and bucketingdivides your data and keeps related data together. This can help reduce the amount of data scanned per query, which means less compute resources needed to service the workload.Track and assess the improvement for environmental sustainabilityThe best way for customers to evaluate success in optimizing their workloads for sustainability is to useproxy measures and unit of work KPIs. For storage, this is GB per transaction, and for compute, it would be vCPU minutes per transaction. To use proxy measures to optimize workloads for energy efficiency, readSustainability Well-Architected LabonTurning the Cost and Usage Report into Efficiency Reports.In Table 1, we have listed certain metrics to use as a proxy metric to measure specific improvements. These fall under each pillar of modern data architecture covered in this post. This is not an exhaustive list, you could use numerous other metrics to spot inefficiencies. Remember, just tracking one metric may not explain the impact on sustainability. Use an analytical exercise of combining the metric with data, type of attributes, type of workload, and other characteristics.PillarMetricsData ingestionDMS Replication Instance and Task Metrics– CPUAllocated, CPUUtilization, WriteThroughput, ReadThroughputCloudWatch Metrics – Amazon Kinesis– ReadProvisionedThroughputExceeded, WriteProvisionedThroughputExceededCloudWatch Metrics – Amazon MSK– BurstBalance, CpuIdle, CpuIoWait, KafkaAppLogsDiskUsed, KafkaDataLogsDiskUsedData lakeCloudWatch Metrics for Amazon S3– BucketSizeBytesTable 1. Metrics for the Modern data architecture pillarsConclusionIn this post, we have provided guidance and best practices to help reduce the environmental impact of the data ingestion and data lake pillars of modern data architecture.In thePart 2, we cover best practices for sustainability for the unified governance, data movement, and purpose-built analytics and insights pillars.Further reading:To learn more, check out theSustainability Pillar of the AWS Well-Architected Frameworkand other blog posts onarchitecting for sustainability.For more architecture content, refer toAWS Architecture Centerfor reference architecture diagrams, vetted architecture solutions,Well-Architectedbest practices, patterns, icons, and more.Sam MokhtariDr. Sam Mokhtari is leading the sustainability pillar of AWS Well-Architected framework. His main area of depth is data and analytics, and he has published more than 30 influential articles in this field.Dr. Ali KhoshkbarDr. Ali Khoshkbar is a Senior Cloud Architect with Amazon Web Services (AWS) Professional Services. He is passionate about helping customers build scalable and high-performance data analytics solutions in the cloud.Sandipan BhaumikSandipan Bhaumik is a Senior Analytics Specialist Solutions Architect based in London. He helps customers modernise their traditional data platforms by building modern data architecture in the cloud to perform analytics at scale.
The modern data architecture on AWS focuses on integrating a data lake and purpose-built data services to efficiently build analytics workloads, which provide speed and agility at scale. Using the right service for the right purpose not only provides performance gains, but facilitates the right utilization of resources. Review Modern Data Analytics Reference Architecture on […]
Sam Mokhtari
Amazon Redshift Amazon Simple Storage Service (S3) Architecture AWS Glue AWS Well-Architected Sustainability