Optimizing your AWS Infrastructure for Sustainability, Part II: Storage
AWS Architecture BlogOptimizing your AWS Infrastructure for Sustainability, Part II: Storageby Katja Philipp, Aleena Yunus, Otis Antoniou, and Ceren Tahtasiz | on22 SEP 2021| inAmazon CloudWatch,Amazon Simple Storage Service (S3),Architecture,Sustainability|Permalink|ShareInPart I of this series, we introduced you to strategies to optimize the compute layer of your AWS architecture for sustainability. We provided you with success criteria, metrics, and architectural patterns to help you improve resource and energy efficiency of your AWS workloads.This blog post focuses on the storage layer of your AWS infrastructure and provides recommendations that you can use to store your data sustainably.Optimizing the storage layer of your AWS infrastructureManaging your data lifecycle and using different storage tiers are key components to optimizing storage for sustainability. When you consider different storage mechanisms, remember that you’re introducing a trade-off between resource efficiency, access latency, and reliability. This means you’ll need to select your management pattern accordingly.Reducing idle resources and maximizing utilizationStoring and accessing data efficiently, in addition to reducing idle storage resources results in a more efficient and sustainable architecture.Amazon CloudWatchoffers storage metrics that can be used to assess storage improvements, as listed in the following table.ServiceMetricSourceAmazon Simple Storage Service (Amazon S3)BucketSizeBytesMetrics and dimensionsS3 Object AccessLogging requests using server access loggingAmazon Elastic Block Store (Amazon EBS)VolumeIdleTimeAmazon EBS metricsAmazon Elastic File System (Amazon EFS)StorageBytesAmazon CloudWatch metrics for Amazon EFSAmazon FSx for LustreFreeDataStorageCapacityMonitoring Amazon FSx for LustreAmazon FSx for Windows File ServerFreeStorageCapacityMonitoring with Amazon CloudWatchAmazon FSx for NetApp ONTAPStorageCapacity / StorageUsedFile system metricsAmazon FSx for OpenZFSStorageCapacity / UsedStorageCapacityMonitoring with Amazon CloudWatchYou can monitor these metrics with the architecture shown in Figure 1. CloudWatch provides a unified view of your resource metrics.Figure 1. CloudWatch for monitoring your storage resourcesIn the following sections, we present four concepts to reduce idle resources and maximize utilization for your AWS storage layer.Analyze data access patterns and use storage tiersChoosing the right storage tier after analyzing data access patterns gives you more sustainable storage options in the cloud.By storing less volatile data on technologies designed for efficient long-term storage, you will optimize your storage footprint. More specifically, you’ll reduce the impact you have on the lifetime of storage resources by storing slow-changing or unchanging data on magnetic storage, as opposed to solid state memory. For archiving data or storing slow-changing data, consider usingAmazon EFS Infrequent Access,Amazon EBS Cold HDDvolumes, andAmazon S3 Glacier.To store your data efficiently throughout its lifetime, create anAmazon S3 Lifecycle configurationthat automatically transfers objects to a different storage class based on your pre-defined rules. TheExpiring Amazon S3 Objects Based on Last Accessed Date to Decrease Costsblog post shows you how to create custom object expiry rules for Amazon S3 based on the last accessed date of the object.For data with unknown or changing access patterns, useAmazon S3 Intelligent-Tieringto monitor access patterns and move objects among tiers automatically. In general, you have to make a trade-off between resource efficiency, access latency, and reliability when considering these storage mechanisms. Figure 2 shows an overview of data access patterns for Amazon S3 and the resulting storage tier. For example, inS3 One Zone-IA, energy and server capacity are reduced, because data is stored only within one Availability Zone.Figure 2. Data access patterns for Amazon S3Use columnar data formats and compressionColumnar data formats like Parquet and ORC require less storage capacity compared to row-based formats like CSV and JSON.Parquet consumes up tosix timesless storage in Amazon S3 compared to text formats. This is because of features such as column-wise compression, different encodings, or compression based on data type, as shown in theTop 10 Performance Tuning Tips for Amazon Athenablog post.You can improve performance and reduce query costs ofAmazon Athenaby30–90 percentby compressing, partitioning, and converting your data into columnar formats. Using columnar data formats and compressions reduces the amount of data scanned.Reduce unused storage resourcesRight size or delete unused storage volumesAs shown in theCost Optimization on AWSvideo, right-sizing storage by data type and usage reduces your associated costs by up to 50 percent.A straightforward way to reduce unused storage resources is to delete unattached EBS volumes. If the volume needs to be quickly restored later on, you can store anAmazon EBS snapshotbefore deletion.You can also useAmazon Data Lifecycle Managerto retain and delete EBS snapshots and Amazon EBS-backed Amazon Machine Images (AMIs) automatically. This further reduces the storage footprint of stale resources.To avoid over-provisioning volumes, see theAutomating Amazon EBS Volume-resizingblog post. It demonstrates an automated workflow that can expand a volume every time it reaches a capacity threshold. These Amazon EBS elastic volumes extend a volume when needed, as shown in theAmazon EBS Updateblog post.Another way to optimize block storage is to identify volumes that are underutilized and downsize them. Or you can change the volume type, as shown in theAWS Storage Optimizationwhitepaper.Modify the retention period of CloudWatch LogsBy default,CloudWatch Logsare kept indefinitely and never expire. You can adjust the retention policy for each log group to be between one day and 10 years. For compliance reasons,export log data to Amazon S3and use archival storage such as Amazon S3 Glacier.Deduplicate dataLarge datasets often have redundant data, which increases your storage footprint.By turning ondata deduplicationfor yourAmazon FSx for Windows File Server, you will optimize data storage. For general-purpose file shares, storage space can be reduced by 50–60 percent through deduplication.If you have datasets residing in Amazon S3, you can automatically get rid of duplicates by using theFindMatches transformprovided byAWS Lake Formation. See theIntegrate and deduplicate datasets using AWS Lake Formation FindMatchesblog post for more information on how to set it up.ConclusionIn this blog post, we discussed data storing techniques to increase your storage efficiency. These include right-sizing storage volumes; choosing storage tiers depending on different data access patterns; and compressing and converting data.These techniques allow you to optimize your AWS infrastructure for environmental sustainability.This blog post is the second post in the series, you can find the first part of the series linked in the following section. In the next part of this blog post series, we will show you how you can optimize the networking part of your IT infrastructure for sustainability in the cloud!Other blog posts in this seriesOptimizing your AWS Infrastructure for Sustainability, Part I: ComputeOptimizing your AWS Infrastructure for Sustainability, Part III: NetworkingRelated informationAWS Architecture Monthly magazine, Sustainability issue, August 2021TAGS:Optimizing your AWS Infrastructure for Sustainability Series,SustainabilityKatja PhilippKatja Philipp is an Associate Solutions Architect based in Munich, Germany. With a background in M. Sc. Information Systems, she joined AWS in September 2020 with the TechU Graduate program. She enables her customers in the Power & Utilities vertical with best practices around their cloud journey. Katja is passionate about sustainability and how technology can be leveraged to solve current challenges for a better future.Aleena YunusAleena Yunus is an Associate Startup Solutions Architect in DACH. She has a Masters in Computer Science majoring in distributed systems and network security. She enjoys developing, helping her customers innovate on the cloud, and apply architectural best practices. Aleena is an avid reader and her favorite author is John Steinbeck.Otis AntoniouOtis Antoniou is an Associate Solutions Architect based in London. He is part of the UK and Ireland SMB team and he enjoys helping customers innovate and solve business problems using AWS Services. Outside of work, Otis enjoys playing team sports, racket sports, and he is passionate about music.Ceren TahtasizCeren Tahtasiz is a Startup Solutions Architect based in London. She helps startups grow by understanding their goals and challenges, and guiding them on how they can get started on AWS. She’s passionate about enabling customers to launch cloud-native products that are resilient and scalable. Her core focus areas are serverless technologies and sustainability in cloud.
In Part I of this series, we introduced you to strategies to optimize the compute layer of your AWS architecture for sustainability. We provided you with success criteria, metrics, and architectural patterns to help you improve resource and energy efficiency of your AWS workloads. This blog post focuses on the storage layer of your AWS infrastructure and provides […]
Katja Philipp
Amazon CloudWatch Amazon Simple Storage Service (S3) Architecture Sustainability