Field Notes: Develop Data Pre-processing Scripts Using Amazon SageMaker Studio and an AWS Glue Development Endpoint
AWS Architecture BlogField Notes: Develop Data Pre-processing Scripts Using Amazon SageMaker Studio and an AWS Glue Development Endpointby Sam Mokhtari and Julie Tran | on17 JUN 2021| inAmazon SageMaker,Architecture,AWS Glue,Technical How-to|Permalink|ShareThis post was co-written with Marcus Rosen, a Principal  – Machine Learning Operations with Rio Tinto, a global mining company.Data pre-processing is an important step in setting up Machine Learning (ML) projects for success. Many AWS customers use Apache Spark onAWS GlueorAmazon EMRto run data pre-processing scripts while using Amazon SageMaker to build ML models.  To develop spark scripts in AWS Glue, you can create an environment called a Glue Development (Dev) Endpoint that lets you author and test your data pre-processing scripts iteratively. When you’re satisfied with the results of your development, you can create aGlue ETL jobthat runs the final script as part of your automation framework.With the introduction ofAmazon SageMaker Studioin AWS re:Invent 2020, you can now use a single web-based IDE to spin up a notebook and perform all ML development steps. These include data pre-processing, ML model training, ML model deployment and monitoring.This post walks you through how to connect a SageMaker Studio notebook to an AWS Glue Dev Endpoint, so you can use a single tool to iteratively develop both data pre-processing scripts and ML models.Solution OverviewThe following diagram shows the components that are used in this solution.First, we use anAWS CloudFormation templateto set up the required networking components (for example, VPC, subnets).Then, we create an AWS Glue Dev Endpoint and use a security group to allow SageMaker Studio to securely access the endpoint.Finally, we create a studio domain and use a SparkMagic kernel to connect to the AWS Glue Dev Endpoint and run spark scripts.In the Amazon SageMaker Studio notebook, SparkMagic will call a REST API against a Livy server running on the AWS Glue Dev Endpoint. Apache Livy is a service that enables interaction with a remote Spark cluster over a REST API.Set up the VPCYou can use the following CloudFormation template to set up the environment needed for this solution.This template deploys the following resources in your account:A new VPC, with both public and private subnet.VPC endpoints for the following resources:Amazon S3AWS GlueAmazon SageMaker APIAmazon SageMaker RunTimeSecurity groups for SageMaker Studio, Glue endpoint and VPC endpointsSageMaker Service IAM roleAWS Glue Dev Endpoint IAM roleSet up the AWS Glue Dev EndpointSet up AWS Glue Dev EndpointReview thisDeveloper Guide: Adding a Development Endpointfor instructions to create an AWS Glue Dev Endpoint.Note:you must use the AWS Glue Dev Endpoint IAM role provisioned by the CloudFormation template.In the Networking section, selectChoose a VPC, subnet, and security groups.Then choose theVPC glue security group,which you provisioned through the CloudFormation template.The AWS Glue Dev Endpoint needs to be secured with an SSH public key, which should be generated within your local environment. An SSH key pair (public/private) can be generated using the ssh-keygen on Linux or using PuTTYgen on Windows.The final review page looks similar to the following screenshot.Once the AWS Glue Dev Endpoint is inReadystatus, keep note of its private IP address (Glue -> ETL -> Dev Endpoints). You will use this IP for the Livy port forwarding.Set up SageMaker StudioWe recommend launching the SageMaker Studio resource by following the instructions inSecuring Amazon SageMaker Studio connectivity using a private VPC.Follow these steps when you provision the SageMaker Studio resources:SelectStandard setupwith the AWS Identity and Access Management (IAM) authentication method.Attach aSageMaker Service IAM role, created by the CloudFormation template, to SageMaker Studio.UnderNetwork and storage, select the same VPC and private subnet as the AWS Glue endpoint.For theNetwork Access for Studiooption, selectVPC Only — SageMaker Studio will use your VPC. Direct internet access is disabled.Then ensure that the security group with the self-referencing rule is attached. Also, check your other required security groups are attached for SageMaker Studio from the CloudFormation template output.Connect the SageMaker Studio notebook to the AWS Glue Dev EndpointOnce you launch the SageMaker Studio and you add the users. Follow these steps to connect the SageMaker Studio notebook to the AWS Glue Dev Endpoint:Open the Studio and go to thelauncher page(by pressing the “+” icon on the top-left of the page.UnderNotebooks and compute resources, selectSparkMagicin the dropdown menu and selectNotebook.Then open another launcher page, selectSparkMagicin the same dropdown menu and selectImage terminal. One thing to note is that the SparkMagic app will take some time to initialize. Proceed once the apps are inReadystatus (2-3 minutes).4. Upload the private key into SparkMagic Image terminal. In other words, copy the private key to “.ssh” directory and update its permissions using “chmod 400”.Note:the private key is corresponding to the public key used when you create the AWS Glue Dev Endpoint.5. Now, you need to achieve port forwarding of the Livy service in order for SparkMagic kernel to be able to connect to the AWS Glue Dev Endpoint.  You run the following command in the image terminal:/usr/bin/ssh -4 -N -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o StrictHostKeyChecking=no -i /root/.ssh/{PRIVATE_KEY} -L 8998:169.254.76.1:8998 glue@{GLUE_ENDPOINT_PRIVATE_IP_ADDRESS}The command consists of:{PRIVATE_KEY}is the private key file name that you copied into .ssh directory.{GLUE_ENDPOINT_PRIVATE_IP_ADDRESS}is the private IP address of the AWS Glue Dev Endpoint.“8998”is the Livy port we are using for port forwarding.“169.254.76.1”is the remote IP address defined by AWS Glue, this IP address doesnotchange.Note:Keep this terminal open and the SSH command running in order to keep the Livy session active.6. Go to theSparkMagic notebookand restart the kernel, by going to the top menu and selectingKernel > Restart Kernel.7. Once the notebook kernel is restarted, the connection between the Studio Notebook and the AWS Glue Dev Endpoint is ready. To test the integration, you can run the following example command to list the tables in the AWS Glue Data Catalog.spark.sql("show tables").show()Cleaning upTo avoid incurring future charges, delete the resources you created:Shut down the notebooksyou started with Studio.Delete the Studio domain.Delete Amazon EFS storage volume for SageMaker StudioDelete AWS Glue EndpointOn the AWS CloudFormation console, delete the CloudFormation stack.ConclusionOur customers needed a single web-based IDE to spin up a notebook and perform all ML development steps including data pre-processing, ML model training, ML model deployment and monitoring. This blog post demonstrated how you can configure a SageMaker Studio notebook and connect to AWS Glue Dev Endpoint. This provides a framework for you to use  when developing both data preprocessing scripts and ML models.To learn more about how to develop data pre-processing scripts and ML models in Amazon SageMaker, you can check out the examples in thisrepository.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Field NotesSam MokhtariDr. Sam Mokhtari is leading the sustainability pillar of AWS Well-Architected framework. His main area of depth is data and analytics, and he has published more than 30 influential articles in this field.Julie TranJulie Tran is a Cloud Engineer in Big Data with AWS. She provides support to various customers in their own AWS journey. Her specialization is in big data architectures, ETL workflows, ML and data analytics in the cloud. She enjoys gaming and plays MOBA with her friends.
This post was co-written with Marcus Rosen, a Principal  – Machine Learning Operations with Rio Tinto, a global mining company.  Data pre-processing is an important step in setting up Machine Learning (ML) projects for success. Many AWS customers use Apache Spark on AWS Glue or Amazon EMR to run data pre-processing scripts while using Amazon SageMaker […]
Sam Mokhtari
Amazon SageMaker Architecture AWS Glue Technical How-to