Optimize AI/ML workloads for sustainability: Part 2, model development
AWS Architecture BlogOptimize AI/ML workloads for sustainability: Part 2, model developmentby Benoit de Chateauvieux, Eddie Pick, Dan Ferguson, and Brendan Sisson | on21 MAR 2022| inArchitecture,Artificial Intelligence,AWS Well-Architected,Sustainability|Permalink|ShareMore complexity often means using more energy, and machine learning (ML) models are becoming bigger and more complex. And though ML hardware is getting more efficient, the energy required to train these ML models isincreasing sharply.In this series, we’re following the phases of theWell-Architected machine learning lifecycle(Figure 1) to optimize your artificial intelligence (AI)/ML workloads. In Part 2, we examine the model development phase and show you how to train, tune, and evaluate your ML model to help you reduce your carbon footprint.If you missed the first part of this series, we showed you how to examine your workload to help you 1) evaluate the impact of your workload, 2) identify alternatives to training your own model, and 3) optimize data processing.Figure 1. ML lifecycleModel buildingDefine acceptable performance criteriaWhen you build an ML model, you’ll likely need to make trade-offs between your model’s accuracy and its carbon footprint. When we focus only on the model’s accuracy, we “ignore the economic, environmental, or social cost of reaching the reported accuracy.” Because therelationship between model accuracy and complexity is at best logarithmic, training a model longer or looking for better hyperparameters only leads to asmall increase in performance.Establish performance criteria that support your sustainability goals while meeting your business requirements, not exceeding them.Select energy-efficient algorithmsBegin with a simple algorithm to establish a baseline. Then,test different algorithms with increasing complexityto observe whether performance has improved. If so, compare the performance gain against the difference in resources required.Try to find simplified versions of algorithms. This will help you use less resources to achieve a similar outcome. For example,DistilBERT, a distilled version ofBERT, has 40% fewer parameters, runs 60% faster, and preserves 97% of BERT’s performance.Use pre-trained or partially pre-trained modelsConsider techniques to avoid training a model from scratch:Transfer Learning:Use a pre-trained source model and reuse it as the starting point for a second task. For example, a model trained onImageNet(14 million images) can generalize with other datasets.Incremental Training:Use artifacts from an existing model on an expanded dataset to train a new model.Optimize your deep learning models to accelerate trainingCompile your DL models from their high-level language representation to hardware-optimized instructions to reduce training time. You can achieve this with open-source compilers orAmazon SageMakerTraining Compiler, which can speed up training of DL models by up to 50% by more efficiently using SageMaker GPU instances.Start with small experiments, datasets, and compute resourcesExperiment with smaller datasets in your development notebook. This allows you to iterate quickly with limited carbon emission.Automate the ML environmentWhen building your model, useLifecycle Configuration Scriptstoautomatically stopidle SageMaker Notebook instances. If you are usingSageMaker Studio, install theauto-shutdown Jupyter extensionto detect and stop idle resources.Use thefully managed training processprovided by SageMaker to automatically launch training instances and shut them down as soon as the training job is complete. This minimizes idle compute resources and thus limits the environmental impact of your training job.Adopt a serverless architecture for yourMLOpspipelines. For example, orchestration tools likeAWS Step FunctionsorSageMaker Pipelinesonly provision resources when work needs to be done. This way, you’re not maintaining compute infrastructure 24/7.Model trainingSelect sustainable AWS RegionsAs mentioned inPart 1, select an AWS Region with sustainable energy sources. When regulations and legal aspects allow, choose Regionsnear Amazon renewable energy projectsand Regions where the grid has low published carbon intensity to train your model.Use a debuggerA debugger likeSageMaker Debuggercan identify training problems like system bottlenecks, overfitting, saturated activation functions, andunder-utilization of system resources. It also providesbuilt-in ruleslikeLowGPUUtilizationorOverfit. These rules monitor your workload and will automatically stop a training job as soon as it detects a bug (Figure 2), which helps you avoid unnecessary carbon emissions.Figure 2. Automatically stop buggy training jobs with SageMaker DebuggerOptimize the resources of your training environmentReference the recommended instance types for the algorithm you’ve selected in the SageMaker documentation. For example,for DeepAR, you should start with a single CPU instance and only switch to GPU and multiple instanceswhen necessary.Right size your training jobs withAmazon CloudWatchmetrics that monitor the utilization of resources like CPU, GPU, memory, and disk utilization.ConsiderManaged Spot Training, which takes advantage of unusedAmazon Elastic Compute Cloud (Amazon EC2)capacity and can save you up to 90% in cost compared to On-Demand instances. By shaping your demand for the existing supply of EC2 instance capacity, you will improve your overall resource efficiency andreduce idle capacity of the overall AWS Cloud.Use efficient siliconUseAWS Trainiumfor optimized for DL training workloads. It is expected to beour most energy efficient processorfor this purpose.Archive or delete unnecessary training artifactsOrganize your ML experiments withSageMaker Experimentstoclean up training resourcesyou no longer need.Reduce the volume of logs you keep. By default, CloudWatch retains logs indefinitely. Bysetting limited retention timefor your notebooks and training logs, you’ll avoid the carbon footprint of unnecessary log storage.Model tuning and evaluationUse efficient cross-validation techniques for hyperparameter optimizationPrefer Bayesian search over random search(andavoid grid search). Bayesian search makes intelligent guesses about the next set of parameters to pick based on the prior set of trials. It typically requires10 times fewer jobsthan random search, and thus 10 times less compute resources, to find the best hyperparameters.Limit the maximum number of concurrent training jobs. Running hyperparameter tuning jobs concurrently gets more work done quickly. However, a tuning job improves only through successive rounds of experiments. Typically, running one training job at a time achieves the best results with the least amount of compute resources.Carefully choose the number of hyperparameters and their ranges. You get better results and use less compute resources by limiting your search to a few parameters and small ranges of values. If you know that a hyperparameter is log-scaled, convert it to further improve the optimization.Use warm-start hyperparameter tuningUsewarm-startto leverage the learning gathered in previous tuning jobs to inform which combinations of hyperparameters to search over in the new tuning job. This technique avoids restarting hyperparameter optimization jobs from scratch and thus reduces the compute resources needed.Measure results and improveTo monitor and quantify improvements of your training jobs, track the following metrics:Resources provisioned for your training jobs(InstanceCount,InstanceType, andVolumeSizeInGB)Efficient use of these resources(CPUUtilization,GPUUtilization,GPUMemoryUtilization,MemoryUtilization, andDiskUtilization) in theSageMaker Console, theCloudWatch Consoleor yourSageMaker Debugger Profiling ReportFor storage:The total size of yourAmazon Simple Storage Service (Amazon S3)buckets and storage class distribution, usingAmazon S3 Storage LensThesize of your CloudWatch log groupsConclusionIn this blog post, we discussed techniques and best practices to reduce the energy required to build, train, and evaluate your ML models.We also provided recommendations for the tuning process as it makes up a large part of the carbon impact of building an ML model. During hyperparameter and neural design search, hundreds of versions of a given model are created, trained, and evaluated before identifying an optimal design.In the next post, we’ll continue our sustainability journey through the ML lifecycle and discuss the best practices you can follow when deploying and monitoring your model in production.Want to learn more?Check out theArchitecting for sustainability session at re:Invent 2021, and other blog posts onarchitecting for sustainability.These practices are part of theSustainability Pillarof theAWS Well-Architected Framework, which helps you build secure, high-performing, resilient, and efficient infrastructure for your applications and workloads. Use theAWS Well-Architected Toolto address important design considerations and ensure that your workloads follow the best practices and guidance of the Well-Architected Framework. For follow-up questions or comments, join our growing community onAWS re:Post.Other posts in this seriesOptimize AI/ML workloads for sustainability: Part 1, identify business goals, validate ML use, and process dataOptimize AI/ML workloads for sustainability: Part 3, deployment and monitoringLooking for more architecture content?AWS Architecture Centerprovides reference architecture diagrams, vetted architecture solutions, AWSWell-Architectedbest practices, patterns, icons, and more!TAGS:AWS Well-Architected Framework,Optimize AI/ML workloads for sustainability,SustainabilityBenoit de ChateauvieuxBenoit de Chateauvieux is a Startup Solutions Architect at AWS, based in Montreal, Canada. As a former CTO, he enjoys helping startups build great and sustainable products using the cloud. Outside of work, you’ll find Benoit in canoe-camping expeditions, paddling across Canadian rivers.Eddie PickEddie Pick is a Senior Startup Solutions Architect at AWS based in Montréal, Canada. As an ex co-founder and former CTO, his goal is to help startups build great products faster on the Cloud, particularly using machine learning.Dan FergusonDan Ferguson is a Solutions Architect at AWS, based in New York, USA. As a machine learning services expert, Dan works to support customers on their journey to integrating ML workflows efficiently, effectively, and sustainably.Brendan SissonBrendan Sisson is a Principal Sustainability Solutions Architect at AWS based in London, UK. As a contributor to the Sustainability Pillar of the AWS Well-Architected Framework, he supports customers on how they can optimize their workloads running in the AWS Cloud and how they can use the AWS Cloud to help solve their wider sustainability challenges.
More complexity often means using more energy, and machine learning (ML) models are becoming bigger and more complex. And though ML hardware is getting more efficient, the energy required to train these ML models is increasing sharply. In this series, we’re following the phases of the Well-Architected machine learning lifecycle (Figure 1) to optimize your […]
Benoit de Chateauvieux
Architecture Artificial Intelligence AWS Well-Architected Sustainability