Improving Retail Forecast Accuracy with Machine Learning
AWS Architecture BlogImproving Retail Forecast Accuracy with Machine Learningby Soonam Jose, Johny Duval, and Paul Miller | on26 JUL 2021| inAmazon Athena,Amazon Forecast,Amazon Machine Learning,Amazon QuickSight,Amazon SageMaker,Architecture,AWS Glue,AWS Solutions Implementations,Intermediate (200)|Permalink|ShareTheglobal retail market continues to grow largerand the influx of consumer data increases daily. The rise in volume, variety, and velocity of data poses challenges with demand forecasting and inventory planning. Outdated systems generate inaccurate demand forecasts. This results in multiple challenges for retailers. They are faced with over-stocking and lost sales, and often have to rely on increased levels of safety stock to avoid losing sales.A recentMcKinsey study indicatesthat AI-based forecasting improves forecasting accuracy by 10–20 percent. This translates to revenue increases of 2–3 percent. An accurate forecasting system can also help determine ideal inventory levels and better predict the impact of sales promotions. It provides a single view of demand across all channels and a better customer experience overall.In this blog post, we will show you how to build a reliable retail forecasting system. We will useAmazon Forecast, and an AWS-vetted solution calledImproving Forecast Accuracy with Machine Learning. This is an AWS Solutions Implementation that automatically produces forecasts and generates visualization dashboards. This solution can be extended to use cases across a variety of industries.Improving Forecast Accuracy solution architectureThis post will illustrate a retail environment that has an SAP S/4 HANA system for overall enterprise resource planning (ERP). We will show a forecasting solution based on Amazon Forecast to predict demand across product categories. The environment also has a unified platform for customer experience provided bySAP Customer Activity Repository (CAR). Replenishment processes are driven bySAP Forecasting and Replenishment (F&R), andSAP Fioriapps are used to manage forecasts.The solution is divided into four parts: Data extraction and preparation, Forecasting and monitoring, Data visualization, and Forecast import and utilization in SAP.Figure 1. Notional architecture for improving forecasting accuracy solution and SAP integration­­Data extraction and preparationHistorical demand data such as sales, web traffic, inventory numbers, and resource demand are extracted from SAP and uploaded toAmazon Simple Storage Service (S3). There aremultiple ways to extract data from an SAP system into AWS. As part of this architecture, we will use operational data provisioning (ODP) extraction. ODP acts as a data source for OData services, enabling REST-based integrations with external applications. TheODP-Based Data Extraction via OData documentdetails this approach. The steps involved are:Create a data source using transaction RSO2, allowChange Data Capturefor specific data to be extractedCreate an OData service using transaction SEGWCreate a Data model for ODP extraction, which refers to the defined data source, then register the serviceInitiate the service from SAP gateway clientIn theAWS Management Console, create an AWS Lambda function to extract data and upload to S3. Check out thesample extractor code using Python, referenced in the blogBuilding data lakes with SAP on AWSRelated data that can potentially affect demand levels can be uploaded to Amazon S3. These could include seasonal events, promotions, and item price. Additional item metadata, such as product descriptions, color, brand, size may also be uploaded. Amazon Forecast provides built-in related time series data forholidaysandweather. These three components together form theforecast inputs.Forecasting and monitoringAn S3 event notification will be initiated when new datasets are uploaded to the input bucket. This in turn, starts anAWS Step Functionsstate machine. The state machine combines a series ofAWS Lambdafunctions that build, train, and deploy machine learning models inAmazon Forecast. All AWS Step Functions logs are sent toAmazon CloudWatch. Administrators will be notified with the results of the AWS Step Functions throughAmazon Simple Notification Service (SNS).AnAWS Gluejob combines raw forecast input data, metadata, predictor backtest exports, and forecast exports. These all go into an aggregated view of forecasts in an S3 bucket. It is then translated to the format expected by theExternal Forecastimport interface.Amazon Athenacan be used to query forecast output in S3 using standard SQL queries.Data visualizationAmazon QuickSightanalyses can be created on a per-forecast basis. This provides users with forecast output visualization across hierarchies and categories of forecasted items. It also displays item-level accuracy metrics. Dashboards can be created from these analyses and shared within the organization. Additionally, data scientists and developers can prepare and process data, and evaluate Forecast outputs using anAmazon SageMakerNotebook Instance.Forecast import and utilization in SAPAmazon Forecast outputs located in Amazon S3 will beimported into the Unified Demand Forecast (UDF)module within the SAP Customer Activity Repository (CAR). You can read here about how toimport external forecasts. An AWS Lambda function will be initiated when aggregated forecasts are uploaded to the S3 bucket. The Lambda function performs a remote function call (RFC) to the SAP system through the officialSAP JCo Library. The SAP RFC credentials and connection information may be stored securely insideAWS Secrets Managerand read on demand to establish connectivity.Once imported, forecast values from the solution can beretrieved bySAP Forecasting and Replenishment (F&R). They will be consumed as an input to replenishment processes, which consist of requirements calculation and­­­­­ requirement quantity optimization. SAP F&R calculates requirements based on the forecast, the current stock, and the open purchase orders. The requirement quantity then may be improved in accordance with optimization settings defined in SAP F&R.­­­Additionally, you have the flexibly toadjust the system forecastas required by the demand situation oranalyze forecastsvia respectiveSAP Fiori Apps.Sample use case: AnyCompany Stores, Inc.To illustrate how beneficial this solution can be for retail organizations, let’s consider AnyCompany Stores, Inc. This is a hypothetical customer and leader in the retailer industry with 985 stores across the United States. They struggle with issues stemming from their existing forecasting implementation. That implementation only understands certain categories and does not factor in the entire product portfolio. Additionally, it is limited to available demand history and does not consider related information that may affect forecasts. AnyCompany Stores is looking to improve their demand forecasting system.UsingImproving Forecast Accuracy with Machine Learning, AnyCompany Stores can easily generate AI-based forecasts at appropriate quantiles to address sensitivities associated with respective product categories. This mitigates inconsistent inventory buys, overstocks, out-of-stocks, and margin erosion. The solution also considers all relevant related data in addition to the historical demand data. This ensures that generated forecasts are accurate for each product category.The generated forecasts may be used to complement existing forecasting and replenishment processes. With an improved forecasting solution, AnyCompany Stores will be able to meet demand, while holding less inventory and improving customer experience. This also helps ensure that potential demand spikes are accurately captured, so staples will always be in stock. Additionally, the company will not overstock expensive items with short shelf lives that are likely to spoil.ConclusionIn this post, we explored how to implement an accurate retail forecasting solution using a ready-to-deploy AWS Solution. We use generated forecasts to drive inventory replenishment optimization and improve customer experience. The solution can be extended to inventory, workforce, capacity, and financial planning.We showcase one of the ways in whichImproving Forecast Accuracy with Machine Learningmay be extended for a use case in the retail industry. If your organization would like to improve business outcomes with the power of forecasting, explore customizing this solution to fit your unique needs.Further reading:From forecasting demand to ordering – An automated machine learning approach with Amazon Forecast to decrease stockouts, excess inventory, and costsTechnical Guide: Time Series Forecasting Principles with Amazon ForecastBig Data Analytics – ForecastingTAGS:aws solutionsSoonam JoseSoonam Jose is a Senior Solutions Architect at Amazon Web Services, working with a leading Global System Integrator (GSI). She helps organizations migrate and run their businesses in the cloud. She is focused on data analytics, artificial intelligence, and machine learning. Soonam provides architectural guidance and supports the GSI partner in building strategic industry solutions on AWS.Johny DuvalJohny Duval is the AI/ML Product Manager for the AWS Solutions team. He leads a team of Solution Builders in bringing to market Well-Architected end-to-end products leveraging AI & ML services and technologies. He has worked many years leading startup teams of various sizes through different stages, integrating mobile technology to higher education institutions, launching analytics products to enterprise clients, and operationalizing AI technology to Fortune 500 companies.Paul MillerAs an experienced builder and solutions architect, Paul Miller specializes in cloud architecture and DevOps Automation. With experience designing secure, well-architected solutions, Paul now focuses on AI/ML solutions and enablement as a builder and SDE on the AWS Solutions team. Paul is the author of the "Improving Forecast Accuracy with Machine Learning" solution.
The global retail market continues to grow larger and the influx of consumer data increases daily. The rise in volume, variety, and velocity of data poses challenges with demand forecasting and inventory planning. Outdated systems generate inaccurate demand forecasts. This results in multiple challenges for retailers. They are faced with over-stocking and lost sales, and […]
Soonam Jose
Amazon Athena Amazon Forecast Amazon Machine Learning Amazon QuickSight Amazon SageMaker Architecture AWS Glue AWS Solutions Implementations Intermediate (200)