Field Notes: Comparing Algorithm Performance Using MLOps and the AWS Cloud Development Kit
AWS Architecture BlogField Notes: Comparing Algorithm Performance Using MLOps and the AWS Cloud Development Kitby Moataz Gaber | on06 JAN 2021| inAmazon Machine Learning,Architecture,AWS Cloud Development Kit,Technical How-to|Permalink|ShareComparing machine learning algorithm performance is fundamental for machine learning practitioners, and data scientists. The goal is to evaluate the appropriate algorithm to implement for a known business problem.Machine learning performance is often correlated to the usefulness of the model deployed. Improving the performance of the model typically results in an increased accuracy of the prediction. Model accuracy is a key performance indicator (KPI) for businesses when evaluating production readiness and identifying the appropriate algorithm to select earlier in model development. Organizations benefit from reduced project expenses, accelerated project timelines and improved customer experience. Nevertheless, some organizations have not introduced a model comparison process into their workflow which negatively impacts cost and productivity.In this blog post, I describe how you can compare machine learning algorithms usingMachine Learning Operations(MLOps). You will learn how to create an MLOps pipeline for comparing machine learning algorithms performance usingAWS Step Functions,AWS Cloud Development Kit(CDK) andAmazon SageMaker.First, I explain the use case that will be addressed through this post. Then, I explain the design considerations for the solution. Finally, I provide access to a GitHub repository which includes all the necessary steps for you to replicate the solution I have described, in your own AWS account.Understanding the Use CaseMachine learning has many potential uses and quite often the same use case is being addressed by different machine learning algorithms. Let’s take Amazon Sagemaker built-in algorithms. As an example, if you are having a “Regression” use case, it can be addressed using (Linear Learner,XGBoostandKNN) algorithms. Another example for a “Classification” use case you can use algorithm such as (XGBoost,KNN,Factorization MachinesandLinear Learner). Similarly for “Anomaly Detection” there are (Random Cut ForestsandIP Insights).In this post, it is a “Regression” use case to identify the age of the abalone which can be calculated based on the number of rings on its shell (age equals to number of rings plus 1.5). Usually the number of rings are counted through microscopes examinations.I use the abalone dataset in libsvm format which contains 9 fields [‘Rings’, ‘Sex’, ‘Length’,’ Diameter’, ‘Height’,’ Whole Weight’,’ Shucked Weight’,’ Viscera Weight’ and ‘Shell Weight’] respectively.The features starting from Sex to Shell Weight are physical measurements that can be measured using the correct tools. Therefore, using the machine learning algorithms (Linear Learner and XGBoost) to address this use case, the complexity of having to examine the abalone under microscopes to understand its age can be improved.Benefits of the AWS Cloud Development Kit (AWS CDK)TheAWS Cloud Development Kit(AWS CDK) is an open source software development framework to define your cloud application resources.The AWS CDK uses thejsiiwhich is an interface developed by AWS that allows code in any language to naturally interact with JavaScript classes. It is the technology that enables the AWS Cloud Development Kit to deliver polyglot libraries from a single codebase.This means that you can use the CDK and define your cloud application resources in typescript language for example. Then by compiling your source module usingjsii, you can package it as modules in one of the supported target languages (e.g: Javascript, python, Java and .Net). So if your developers or customers prefer any of those languages, you can easily package and export the code to their preferred choice.Also, thecdk tfprovides constructs for defining Terraform HCL state files and thecdk8senables you to use constructs for defining kubernetes configuration in TypeScript, Python, and Java. So by using the CDK you have a faster development process and easier cloud onboarding. It makes your cloud resources more flexible for sharing.PrerequisitesAnAWS accountInstall gitInstall python 3.xInstall CDKUnderstanding of the machine learning development CI/CD cycle ofbuilding, training and deploying a Machine Learning Model with Amazon SageMaker.Overview of solutionThis architecture serves as an example of how you can build a MLOps pipeline that orchestrates the comparison of results between the predictions of two algorithms.The solution uses a completely serverless environment so you don’t have to worry about managing the infrastructure. It also deletes resources not needed after collecting the predictions results, so as not to incur any additional costs.Figure 1: Solution ArchitectureWalkthroughIn the preceding diagram, the serverless MLOps pipeline is deployed using AWS Step Functions workflow. The architecture contains the following steps:The dataset is uploaded to the Amazon S3 cloud storage under the/Inputsdirectory (prefix).The uploaded file triggers AWS Lambda using anAmazon S3 notification event.The Lambda function then will initiate the MLOps pipeline built using a Step Functions state machine.The starting lambda will start by collecting the region correspondingtraining images URIsfor both Linear Learner and XGBoost algorithms. These are used in training both algorithms over the dataset. It will also get theAmazon SageMaker Spark Container Imagewhich is used for running the SageMaker processing Job.The dataset is in libsvm format which is accepted by the XGBoost algorithm as per theInput/Output Interface for the XGBoost Algorithm. However, this is not supported by the Linear Learner Algorithm as per Input/Output interface for the linear learner algorithm. So we need to run a processing job usingAmazon SageMaker Data Processing with Apache Spark. The processing job will transform the data from libsvm to csv and will divide the dataset into train, validation and test datasets. The output of the processing job will be stored under/Xgboostand/Lineardirectories (prefixes).Figure 2: Train, validation and test samples extracted from dataset6. Then the workflow of Step Functions will perform the following steps in parallel:Train both algorithms.Create models out of trained algorithms.Create endpoints configurations and deploy predictions endpoints for both models.Invoke lambda function to describe the status of the deployed endpoints and wait until the endpoints become in “InService”.Invoke lambda function to perform 3 live predictions usingboto3and the “test” samples taken from the dataset to calculate the average accuracy of each model.Invoke lambda function to delete deployed endpoints not to incur any additional charges.7. Finally, a Lambda function will be invoked to determine which model has better accuracy in predicting the values.The following shows a diagram of the workflow of the Step Functions:Figure 3: AWS Step Functions workflow graphThe code to provision this solution along with step by step instructions can be found at thisGitHub repo.Results and Next StepsAfter waiting for the complete execution of step functions workflow, the results are depicted in the following diagram:Figure 4: Comparison resultsThis doesn’t necessarily mean that the XGBoost algorithm will always be the better performing algorithm. It just means that the performance was the result of these factors:the hyperparameters configured for each algorithmthe number of epochs performedthe amount of dataset samples used for trainingTo make sure that you are getting better results from the models, you can runhyperparameters tuning jobswhich will run many training jobs on your dataset using the algorithms and ranges of hyperparameters that you specify. This helps you allocate which set of hyperparameters which are giving better results.Finally, you can use this comparison to determine which algorithm is best suited for your production environment. Then you can configure your step functions workflow to update the configuration of the production endpoint with the better performing algorithm.Figure 5: Update production endpoint workflowConclusionThis post showed you how to create a repeatable, automated pipeline to deliver the better performing algorithm to your production predictions endpoint. This helps increase the productivity and reduce the time of manual comparison.  You also learned to provision the solution using AWS CDK and to perform regular cleaning of deployed resources to drive down business costs. If this post helps you or inspires you to solve a problem, share your thoughts and questions in the comments. You can use and extend the code on theGitHub repo.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customersTAGS:Field NotesMoataz GaberMoataz Gaber is a Partner Solutions Architect (SA) at (Accenture and Amazon Web Services Business Group) AABG Launchpad team.
Comparing machine learning algorithm performance is fundamental for machine learning practitioners, and data scientists. The goal is to evaluate the appropriate algorithm to implement for a known business problem. Machine learning performance is often correlated to the usefulness of the model deployed. Improving the performance of the model typically results in an increased accuracy of […]
Moataz Gaber
Amazon Machine Learning Architecture AWS Cloud Development Kit Technical How-to