Automating Anomaly Detection in Ecommerce Traffic Patterns
AWS Architecture BlogAutomating Anomaly Detection in Ecommerce Traffic Patternsby Aditya Pendyala and Senthil Kumar | on07 FEB 2022| inAmazon Kinesis,Amazon Lookout for Metrics,Amazon Simple Notification Service (SNS),Amazon Simple Storage Service (S3),AWS Lambda,Kinesis Data Firehose,Kinesis Data Streams|Permalink|ShareMany organizations with large ecommerce presences have procedures to detect major anomalies in their user traffic. Often, these processes use static alerts or manual monitoring. However, the ability to detect minor anomalies in traffic patterns near real-time can be challenging. Early detection of these minor anomalies in ecommerce traffic (such as website page visits and order completions) helps organizations take corrective actions to address issues. This decreases negative impacts to business key performance indicators (KPIs).In this blog post, we will demonstrate an artificial intelligence/machine learning (AI/ML) solution using AWS services. We’ll show howAmazon KinesisandAmazon Lookout for Metricscan be used to detect major and minor anomalies near-real time, based on historical and current traffic trends.The inconsistency of ecommerce trafficThe ecommerce traffic (and number of orders placed) varies based on season, month, date, and time of day. For example, ecommerce websites experience high traffic during weekday evening hours, compared to morning hours. Similarly, there is a spike in web traffic on weekends, compared to weekdays. However, the ecommerce traffic on holiday events (for example, Black Friday, Cyber Monday) does not follow this trend. Due to such dynamic and varying patterns, detecting minor anomalies in user traffic near-real time becomes difficult.We need a smart solution that can detect the smallest deviation in user traffic based on historical data (date and time). As you can imagine, programming these trends based on static rules is time-intensive. In the next section, we discuss a solution that can help organizations automate and detect minor (and major) anomalies while still accounting for varying traffic trends.The components of our anomaly detection solutionThe architecture consists of three functional components:The ecommerce application that customers use for interactionThe data ingesting, transforming, and storage platformAnomaly detection and notificationThis solution automates data ingestion and anomaly detection, and provides a graphical user interface to interact, tweak, and filter anomalies based on severity.Figure 1 illustrates the architecture of this solution:Figure 1. Architecture diagram of an anomaly detection solution for ecommerce trafficLet’s look at the individual components of this architecture before reviewing the overall solution.The ecommerce application that customers use for interactionA customer’s journey of purchasing a product online involves user actions that include:Searching for and viewing the product on the “Product Display Page” (PDP)Adding to the “cart”Completing the purchase on the “checkout“ pageThe traffic on these pages is broken down into chunks based on time intervals. These serve as the data points that we can use to understand traffic patterns.The data ingesting, transforming, and storage platformEcommerce applications generate data in multiple formats and in different volumes. This data must be fed into a streaming platform that can ingest and collect data continuously. Typically, the data must be transformed and stored for analysis and machine learning purposes. To satisfy these requirements, we will useAmazon Kinesis Data Streamsas a streaming platform for data ingestion.Amazon Kinesis Data FirehosewithAWS Lambdacan transform the data. And we’ll store the data inAmazon Simple Storage Service (S3).Anomaly detection and notification in near-real timeOnce our data is ready, we must analyze it near-real time to identify anomalies. We must notify the concerned team about this anomaly so that they can take necessary corrective actions, if needed. We will use Lookout for Metrics andAmazon Simple Notification Service (SNS)to satisfy these requirements.Lookout for Metrics can detect and diagnose anomalies in traffic patterns using ML. Amazon Lookout for Metrics accepts feedback on detected anomalies and tunes the results to improve accuracy over time. Lookout for Metrics is also capable of integrating with Amazon SNS, which can send notifications via SMS, mobile push, and emails.Monitoring ecommerce traffic with Lookout for MetricsAs shown in Figure 1, data from user traffic and user interactions with the ecommerce application is captured as a function of time, and ingested into Kinesis Data Streams. Using Kinesis Data Firehose and Lambda, data is transformed and stored in an S3 bucket. We then create adetectorin Lookout for Metrics and use the S3 bucket as the data source. Because of seamless integration between S3 and Lookout for Metrics, data from S3 bucket is automatically ingested into the detector we created.Once the detector is activated, Lookout for Metrics will start monitoring the data for anomalies, and start identifying the anomalies near-real time. Lookout for Metrics also provides a mechanism to adjust severity threshold on a scale of 0-100, which will help decrease false positives as much as desired. In addition, it integrates with SNS, and can publish notifications to an SNS Topic. An email/ SMS or mobile push subscription can be created on this topic, which will notify users about any current anomalies.ConclusionIn this post, we discussed how minor anomalies are hard to detect near-real time in ecommerce traffic of organizations. We also discussed the services that can be used to monitor these anomalies, such as Lookout for Metrics. Use this architecture to help you monitor, detect anomalies in near-real time, and reduce any negative impact to your business KPIs.For further reading:Amazon Lookout for Metrics Developer guideAmazon Kinesis Data Streams Developer guideAmazon Kinesis Data Firehose Developer guideAditya PendyalaAditya is a Senior Solutions Architect at AWS based out of NYC. He has extensive experience in architecting cloud based applications. He is currently working with large enterprises to help them craft highly scalable, flexible, and resilient cloud architectures, and guides them on all things cloud. He has a Master of Science degree in Computer Science from Shippensburg University and believes in the quote “When you cease to learn, you cease to grow”.Senthil KumarSenthil Kumar is an AWS Solutions Architect based out of New York, NY. He is passionate about enabling enterprise customers on their digital transformation journey in the cloud and helps architecting and building cloud native solutions.
Many organizations with large ecommerce presences have procedures to detect major anomalies in their user traffic. Often, these processes use static alerts or manual monitoring. However, the ability to detect minor anomalies in traffic patterns near real-time can be challenging. Early detection of these minor anomalies in ecommerce traffic (such as website page visits and […]
Aditya Pendyala
Amazon Kinesis Amazon Lookout for Metrics Amazon Simple Notification Service (SNS) Amazon Simple Storage Service (S3) AWS Lambda Kinesis Data Firehose Kinesis Data Streams