Detecting data drift using Amazon SageMaker
AWS Architecture BlogDetecting data drift using Amazon SageMakerby Shibu Nair, Zorina Alliata, Péter Molnár, and Joe Pazak | on03 MAY 2022| inAmazon Athena,Amazon CloudWatch,Amazon QuickSight,Amazon SageMaker,Amazon SageMaker Data Wrangler,Amazon Simple Storage Service (S3),Architecture|Permalink|ShareAs companies continue to embrace the cloud and digital transformation, they use historical data in order to identify trends and insights. This data is foundational to power tools, such as data analytics and machine learning (ML), in order to achieve high quality results.This is a time where major disruptions are not only lasting longer, but also happening more frequently, as discussed ina McKinsey article on risk and resilience. Any disruption—a pandemic, hurricane, or even blocked sailing routes—has a major impact on the patterns of data and can create anomalous behavior.ML models are dependent on data insights to help plan and support production-ready applications. With any disruptions, data drift can occur. Data drift is unexpected and undocumented changes to data structure, semantics, and/or infrastructure. If there is data drift, the model performance will degrade and no longer provide an accurate guidance. To mitigate the effects of the disruption, data drift needs to be detected and the ML models quickly trained and adjusted accordingly.This blog post explains how to approach changing data patterns in the age of disruption and how to mitigate its effects on ML models. We also discuss the steps of building a feedback loop to capture the request data in the production environment and create a data pipeline to store the data for profiling and baselining. Then, we explain howAmazon SageMaker Clarifycan help detect data drift.How to detect data driftThere are three stages to detecting data drift:data quality monitoring,model quality monitoring, anddrift evaluation(see Figure 1).Figure 1. Stages in detecting data driftData quality monitoringestablishes a profile of the input data during model training, and then continuously compares incoming data with the profile. Deviations in the data profile signal a drift in the input data.You can also detect drift throughmodel quality monitoring, which requires capturing actual values that can be compared with the predictions. For example, using weekly demand forecasting, you can compare the forecast quantities one week later with the actual demand. Some use cases can require extra steps to collect actual values. For example, product recommendations may require you to ask a selected group of consumers for their feedback to the recommendation.SageMaker Clarify provides insights into your trained models, including importance of model features and any biases towards certain segments of the input data. Changes of these attributes between re-trained models also signal drift.Drift evaluationconstitutes the monitoring data and mechanisms to detect changes and triggering consequent actions. WithAmazon CloudWatch, you can define rules and thresholds that prompt drift notifications.Figure 2 illustrates a basic architecture with the data sources for training and production (on the left) and the observed data concerning drift (on the right). You can useAmazon SageMaker Data Wrangler, a visual data preparation tool, to clean and normalize your input data for your ML task. You can store the features that you defined for your models in theAmazon SageMaker Feature Store, a fully managed, purpose-built repository to store, update, retrieve, and share ML features.The white, rectangular boxes in the architecture diagram represent the tasks for detecting data and model drift. You can integrate those tasks into your ML workflow withAmazon SageMaker Pipelines.Figure 2. Basic architecture on how data drift is detected using Amazon SageMakerThe drift observation data can be captured in tabular format, such as comma-separated values orParquet, onAmazon Simple Storage Service (S3)and analyzed withAmazon AthenaandAmazon QuickSight.How to build a feedback loopThe baselining task establishes a data profile from training data. It usesAmazon SageMaker Model Monitorand runs before training or re-training the model. The baseline profile is stored on Amazon S3 to be referenced by the data drift monitoring job.Thedata drift monitoringtask continuously profiles the input data, compares it with baseline, and the results are captured in CloudWatch. This tasks runs on its own computation resources usingDeequ, which checks that the monitoring job does not slow down your ML inference flow and scales with the data. The frequency of running this task can be adjusted to control cost, which can depend on how rapidly you anticipate that the data may change.Themodel quality monitoringtask computes model performance metrics from actuals and predicted values. The origin of these data points depends on the use case. Demand forecasting use cases naturally capture actuals that can be used to validate past predictions. Other use cases can require extra steps to acquire ground-truth data.CloudWatch is a monitoring and observability service with which you can define rules to act on deviation in model performance or data drift. With CloudWatch, you can setup alerts to users via e-mail or SMS, and it can automatically start the ML model re-training process.Run thebaselinetask on your updated data set before re-training your model. Use theSageMaker model registryto catalog your ML models for production, manage model versions, and control the associate training metrics.Gaining insight into data and modelsSageMaker Clarify provides greater visibility into your training data and models, helping identify and limit bias and explain predictions. For example, the trained models may consider some features more strongly than others when generating predictions. Compare the feature importance and bias between model-provided versions for a better understanding of the changes.ConclusionAs companies continue to use data analytics and ML to inform daily activity, data drift may become a more common occurrence. Recognizing that drift can have a direct impact on models and production-ready applications, it is important to architect to identify potential data drift and avoid downgrading the models and negatively impacting results. Failure to capture changes in data can result in loss of process confidence, downgraded model accuracy, or a bottom-line impact to the business.Shibu NairShibu Nair is an Principal Solutions Architect helping customers in their cloud transformation, AI/ML and Data Analytics. He is a technical leader, advising executives and engineers on cloud strategies designed to innovate using data-driven methodology and artificial intelligence/machine learning (AI/ML).Zorina AlliataZorina Alliata is a Sr. Machine Learning Strategist, working with global customers to find solutions that speed up operations and enhance processes using Artificial Intelligence and Machine Learning. Zorina helps companies across several industries identify strategies and tactical execution plans for their ML use cases, platforms, and ML at scale implementations.Péter MolnárPéter Molnár is a Data Scientist, working with global customers to develop solutions in Artificial Intelligence and Machine Learning. Péter helps companies across several industries building scalable AI and ML applications.Joe PazakJoe Pazak is a Global Advisory Practice Manager for Supply Chain and Logistics, covering end-to-end supply chain and helping customers with digital transformation. Joe brings deep supply chain expertise from multiple industries and major transformation projects, covering demand planning, supply planning, SIOP, logistics, and procurement.
As companies continue to embrace the cloud and digital transformation, they use historical data in order to identify trends and insights. This data is foundational to power tools, such as data analytics and machine learning (ML), in order to achieve high quality results. This is a time where major disruptions are not only lasting longer, […]
Shibu Nair
Amazon Athena Amazon CloudWatch Amazon QuickSight Amazon SageMaker Amazon SageMaker Data Wrangler Amazon Simple Storage Service (S3) Architecture