Automating Recommendation Engine Training with Amazon Personalize and AWS Glue
AWS Architecture BlogAutomating Recommendation Engine Training with Amazon Personalize and AWS Glueby Alexander Spivak | on13 JAN 2021| inAmazon Machine Learning,Amazon Personalize,Architecture,AWS Glue|Permalink|ShareCustomers from startups to enterprises observe increased revenue when personalizing customer interactions. Still, many companies are not yet leveraging the power of personalization, or, are relying solely on rule-based strategies. Those strategies are effort-intensive to maintain and not effective. Common reasons for not launching machine learning (ML) based personalization projects include: the complexity of aggregating and preparing the datasets, gaps in data science expertise and the lack of trust regarding the quality of ML recommendations.This blog post demonstrates an approach for product recommendations to mitigate those concerns using historical datasets. To get started with your personalization journey, you don’t need ML expertise or a data lake. The following serverless end-to-end architecture involves aggregating and transforming the required data, as well as automatically training an ML-based recommendation engine.I will outline the architectural production-ready setup for personalized product recommendations based on historical datasets. This is of interest to data analysts who look for ways to bring an existing recommendation engine to production, as well as solutions architects new to ML.Solution OverviewThe two core elements to create a proof-of-concept for ML-based product recommendations are:the recommendation engine and,the data set to train the recommendation engine.Let’s start with the recommendation engine first, and work backwards to the corresponding data needs.Product recommendation engineTo create the product recommendation engine, we useAmazon Personalize. Amazon Personalize differentiates three types of input data:user events calledinteractions(user events like views, signups or likes),item metadata(description of your items: category, genre or availability), anduser metadata(age, gender, or loyalty membership).An interactions dataset is typically the minimal requirement to build a recommendation system. Providing user and item metadata datasets improves recommendation accuracy, and enables cold starts, item discovery and dynamic recommendation filtering.Most companies already have existing historical datasets to populate all three input types. In the case of retail companies, the product order history is a good fit for interactions. In the case of the media and entertainment industry, the customer’s consumption history maps to the interaction dataset. The product and media catalogs map to the items dataset and the customer profiles to the user dataset.Amazon Personalize: from datasets to a recommendation APITheAmazon Personalize Deep Dive Seriesprovides a great introduction into the service and explores the topics of training, inference and operations. There are also multiple blog posts available explaining how tocreate a recommendation engine with Amazon Personalizeand how toselect the right metadata for the engine training. Additionally, the Amazon Personalize samples repository in GitHub showcases a variety of topics: fromgetting started with Amazon Personalize, up to performing aPOC in a Boxusing existing datasets, and, finally,automating the recommendation engine with MLOps. In this post, we focus on getting the data from the historical data sources into the structure required by Amazon Personalize.Creating the datasetWhile manual data exports are a quick way to get started with one-time datasets for experiments, we useAWS Glueto automate this process. The automated approach with AWS Glue speeds up the proof of concept (POC) phase and simplifies the process to production by:easily reproducing dataset exports from various data sources. This are used to iterate with other feature sets for recommendation engine training.adding additional data sources and using those to enrich existing datasetsefficiently performingtransformation logiclikecolumn renamingandfuzzy matchingout of the box with code generation support.AWS Glue is a serverless data integration service that is scalable and simple to use. It provides all of the capabilities needed for data integration and supports a widevariety of data sources:Amazon S3buckets, JDBC connectors, MongoDB databases, Kafka, andAmazon Redshift, the AWS data warehouse. You can even make use of data sources living outside of your AWS environment, e.g.on-premises data centersor other services outside of your VPC. This enables you to perform a data-driven POC even when the data is not yet in AWS.Modern application environments usually combine multiple heterogeneous database systems, like operational relational and NoSQL databases, in addition to, the BI-powering data warehouses. With AWS Glue, we orchestrate theETL (extract, transform, and load) jobsto fetch the relevant data from the corresponding data sources. We then bring it intoa format that Amazon Personalize understands: CSV files with pre-defined column names hosted in an Amazon S3 bucket.Each dataset consists of one or multiple CSV files, which can be uniquely identified by an Amazon S3 prefix. Additionally, each dataset must have anassociated schemadescribing the structure of your data. Depending on the dataset type, there arerequired and pre-defined fields:USER_ID (string) and one additional metadata field for the users datasetITEM_ID (string) and one additional metadata field for the items datasetUSER_ID (string), ITEM_ID (string), TIMESTAMP (long; asEpoch time) for the interactions datasetThe following graph presents a high-level architecture for a retail customer, who has a heterogeneous data store landscape.Using AWS Glue to export datasets from heterogeneous data sources to Amazon S3To understand how AWS Glue connects to the variety of data sources and allows transforming the data into the required format, we need to drill down into the AWS Glue concepts and components.One of the key components of AWS Glue is theAWS Glue Data Catalog: a persistent metadata store containing table definitions, connection information, as well as, the ETL job definitions.The tables are metadata definitions representing the structure of the data in the defined data sources. They do not contain any data entries from the sources but solely the structure definition. You can create a table either manually or automatically by usingAWS Glue Crawlers.AWS Glue Crawlers scan the data in the data sources, extract the schema information from it, and store the metadata as tables in the AWS Glue Data Catalog. This is the preferred approach for defining tables. The crawlers useAWS Glue Connectionsto connect to the data sources. Each connection contains the properties that are required to connect to a particular data store. The connections will be also used later by the ETL jobs to fetch the data from the data sources.AWS Glue Crawlers also help to overcome a challenge frequently appearing in microservice environments. Microservice architectures are frequently operated by fully independent and autonomous teams. This means that keeping track of changes to the data source format becomes a challenge. Based on a schedule, the crawlers can be triggered to update the metadata for the relevant data sources in theAWS Glue Data Catalogautomatically. To detect cases when a schema change would break the ETL job logic, you can combine theCloudWatch Events emitted by AWS Glueon updating the Data Catalog tables with anAWS Lambda functionor a notification send via theAmazon Simple Notification Service (SNS).The AWS GlueETL jobsuse the defined connections and the table information from the Data Catalog to extract the data from the described sources, apply the user-defined transformation logic and write the results into a data sink. AWS Glue can automatically generate code for ETL jobs to help perform a variety of usefuldata transformation tasks.AWS Glue Studio makes the ETL development even simplerby providing aneasy-to-use graphical interfacethat accelerates the development and allows designing jobs without writing any code. If required, the generated code can be fully customized.AWS Glue supports Apache Spark jobs, written either inPythonor inScala, and Python Shell jobs. Apache Spark jobs are optimized to run in a highly scalable, distributed way dealing with any amount of data and are a perfect fit for data transformation jobs. The Python Shell jobs provide access to the raw Python execution environment, which is less scalable but provides a cost-optimized option for orchestrating AWS SDK calls.The following diagram visualizes the interaction between the components described.The basic concepts of populating your Data Catalog and processing ETL dataflow in AWS GlueFor each Amazon Personalize dataset type, we create a separate ETL job. Since those jobs are independent, they also can run in parallel. After all jobs have successfully finished, we can start the recommendation engine training.AWS Glue Workflowsallow simplifying data pipelines by visualizing and monitoring complex ETL activities involving multiple crawlers, jobs, and triggers, as a single entity.The following graph visualizes a typical dataset export workflow for training a recommendation engine, which consists of:a workflow trigger being either manual or scheduleda Python Shell job to remove the results of the previous export workflow from S3a trigger firing when the removal job is finished and initiating the parallel execution of the dataset ETL jobsthe three Apache Spark ETL jobs, one per dataset typea trigger firing when all three ETL jobs are finished and initiating the training notification joba Python Shell job to initiate a new dataset import or a full training cycle in Amazon Personalize (e.g. by triggering theMLOps pipelineusing theAWS SDK)AWS Glue workflow for extracting the three datasets and triggering the training workflow of the recommendation engineCombining the data export and the recommendation engineIn the previous sections, we discussed how to create an ML-based recommendation engine and how to create the datasets for the training of the engine. In this section, we combine both parts of the solution leveraging an adjusted version of theMLOps pipeline solution available on GitHubto speed up the iterations on new solution versions by avoiding manual steps. Moreover, automation means new items can be put faster into production.The MLOps pipeline uses a JSON file hosted in an S3 bucket to describe the training parameters for Amazon Personalize. The creation of a new parameter file version triggers a new training workflow orchestrated in a serverless manner usingAWS Step FunctionsandAWS Lambda.To integrate the Glue data export workflow described in the previous section, we also enable the Glue workflow to trigger the training pipeline. Additionally, we manipulate the pipeline to read the parameter file as the first pipeline step. The resulting architecture enables an automated end-to-end set up from dataset export up to the recommendation engine creation.End-to-end architecture combining the data export with AWS Glue, the MLOps training workflow, and Amazon PersonalizeThe architecture for the end-to-end data export and recommendation engine creation solution is completely serverless. This makes it highly scalable, reliable, easy to maintain, and cost-efficient. You pay only for what you consume. For example, in the case of the data export, you pay only for the duration of the AWS Glue crawler executions and ETL jobs. These are only need to run to iterate with a new dataset.The solution is also flexible in terms of the connected data sources. This architecture is also recommended for use cases with a single data source. You can also start with a single data store and enrich the datasets on-demand with additional data sources in future iterations.Testing the quality of the solutionA common approach to validate the quality of the solution is the A/B testing technique, which is widely used tomeasure the efficacy of generated recommendations. Based on the testing results, you can iterate on the recommendation engine by optimizing the underlying datasets and models. The high degree of automation increases the speed of iterations and the resiliency of the end-to-end process.ConclusionIn this post, I presented a typical serverless architecture for a fully automated, end-to-end ML-based recommendation engine leveraging available historical datasets. As you begin to experiment with ML-based personalization, you will unlock value currently hidden in the data. This helps mitigate potential concerns like the lack of trust in machine learning and you can put the resulting engine into production.Start your personalization journey today with theAmazon Personalize code samplesand bring the engine to production with the architecture outlined in this blog. As a next step, you can involverecording real-time eventsto update the generated recommendations automatically based on the event data.TAGS:Amazon Personalize,machine learningAlexander SpivakAlexander Spivak is a Senior Startup Solutions Architect at AWS, focusing on B2B ISV customers across EMEA North. Prior to AWS, Alexander worked as a consultant in financial services engagements, including various roles in software development and architecture. He is passionate about data analytics, serverless architectures, and creating efficient organizations.
Customers from startups to enterprises observe increased revenue when personalizing customer interactions. Still, many companies are not yet leveraging the power of personalization, or, are relying solely on rule-based strategies. Those strategies are effort-intensive to maintain and not effective. Common reasons for not launching machine learning (ML) based personalization projects include: the complexity of aggregating […]
Alexander Spivak
Amazon Machine Learning Amazon Personalize Architecture AWS Glue