Field Notes: How to Build an AWS Glue Workflow using the AWS Cloud Development Kit
AWS Architecture BlogField Notes: How to Build an AWS Glue Workflow using the AWS Cloud Development Kitby Michael Hamilton and Joe Losinski | on07 OCT 2021| inArchitecture,AWS Cloud Development Kit,AWS Glue,Technical How-to|Permalink|ShareMany customers useAWS Glueworkflows to build and orchestrate their ETL (extract-transform-load) pipelines directly in the AWS Glue console using the visual tool to author workflows. This can be time consuming, harder to version control, and error prone due to manual configurations, when compared to managing your workflows as code. To improve your operational excellence, consider deploying the entire AWS Glue ETL pipeline using theAWS Cloud Development Kit(AWS CDK).In this blog post, you will learn how to build an AWS Glue workflow usingAmazon Simple Storage Service(Amazon S3), various components of AWS Glue,AWS Secrets Manager,Amazon Redshift, and the AWS CDK.Architecture overviewIn this architecture, you will use the AWS CDK to deploy your data sources, ETL scripts, AWS Glue workflow components, and an Amazon Redshift cluster for analyzing the transformed data.Figure 1. AWS Glue workflow architectureIt is common for customers to pre-aggregate data before sending it downstream to analytical engines, like Amazon Redshift, because table joins and aggregations are computationally expensive. The AWS Glue workflow will joinCOVID-19 case data, and COVID-19 hiring data together on their date columns in order to run correlation analysis on the final dataset. The datasets may seem arbitrary, but we wanted to offer a way to better understand the impacts COVID-19 had on jobs in the United States. The takeaway here is to use this as a blueprint for automating the deployment of data analytic pipelines for the data of interest to your business.After the AWS CDK application is deployed, it will begin creating all of the resources required to build the complete workflow. When it completes, the components in the architecture will be created, and the AWS Glue workflow will be ready to start. In this blog post, you start workflows manually, but they can be configured to start on a scheduled time or from a workflow trigger.The workflow is programmed to dynamically pull the raw data from theRegistry of Open Data on AWSwhere you can find theCovid-19 case dataand theHiring Datarespectively.PrerequisitesThis blog post uses an AWS CDK stack written in TypeScript and AWS Glue jobs written in Python. Follow the instructions in theAWS CDK Getting Started guideto set up your environment, before you proceed to deployment.In addition to setting up your environment, you need to clone the Git repository, which contains the AWS CDK scripts and Python ETL scripts used by AWS Glue. The ETL scripts will be deployed to Amazon S3 by the AWS CDK stack asassets, and referenced by the AWS Glue jobs as part of the AWS Glue Workflow.You should have the following prerequisites:Create and activate a new AWS accountfor the installation.Use AWS Cloud9, or configure your own computer or Amazon Elastic Compute Cloud (Amazon EC2) instance to deploy the stack. By using AWS Cloud9, you can quickly deploy a preconfigured development environment that already has many of the prerequisites installed.AWS Cloud9Creating an EC2 EnvironmentOpening an environment in AWS Cloud9Your own computer or EC2 instanceInstall theAWS Command Line Interfaceandconfigure it to point to your AWS account.Use a named profileto access your AWS account.Install theAWS CDK.InstallTypeScriptandPython3.Clonethe Git repository –https://github.com/aws-samples/glue-workflow-aws-cdkInstall third-party packagesusing a package manager of your choice.DeploymentAfter you have cloned the repository, navigate to the glue-cdk-blog/lib folder and open theblog-glue-workflow-stack.tsfile. This is the AWS CDK script used to deploy all necessary resources to build your AWS Glue workflow. Theblog-redshift-vpc-stack.tscontains the necessary resources to deploy the Amazon Redshift cluster, connections, and permissions. Theglue-cdk-blog/lib/assetsfolder also contains the AWS Glue job scripts. These files are uploaded to Amazon S3 by the AWS CDK when you bootstrap.You won’t review the individual lines of code in the script in this blog post, but if you are unfamiliar with any of the AWS CDK level 1 or level 2 constructs used in the sample, you can review what each construct does with theAWS CDK documentation. Familiarize yourself with the script you cloned and anticipate what resources will be deployed. Then, deploy both stacks and verify your initial findings.After your environment is configured, and the packages and modules installed, deploy the AWS CDK stack and assets in two commands.Bootstrap the AWS CDK stack to create an S3 bucket in the predefined account that will contain the assets.cdk bootstrapDeploy the AWS CDK stacks.cdk deploy --allVerify that both of these commands have completed successfully, and remediate any failures returned. Upon successful completion, you’re ready to start the AWS Glue workflow that was just created. You can find the AWS CDK commands reference in theAWS CDK Toolkit commandsdocumentation, and help withTroubleshooting common AWS CDK issuesyou may encounter.WalkthroughPrior to initiating the AWS Glue workflow, explore the resources the AWS CDK stacks just deployed to your account.Log in to the AWS Management Console and the AWS CDK account.Navigate toAmazon S3in the AWS console (you should see an S3 bucket with the name prefix ofcdktoolkit-stagingbucket-xxxxxxxxxxxx).Review the objects stored in the bucket in the assets folder. These are the .py files used by your AWS Glue jobs. They were uploaded to the bucket when you issued the AWS CDK bootstrap command, and referenced within the AWS CDK script as the scripts to use for the AWS Glue jobs. When retrieving data from multiple sources, you cannot always control the naming convention of the sourced files. To solve this and create better standardization, you will use a job within the AWS Glue workflow to copy these scripts to another folder and rename them with a more meaningful name.Navigate toAmazon Redshiftin the AWS console and verify your new cluster. You can use the Amazon Redshift Query Editor within the console to connect to the cluster and see that you have an empty database calleddb-covid-hiring. The Amazon Redshift cluster and networking resources were created by theredshift_vpc_stackwhich are listed here:VPC, subnet and security group for Amazon RedshiftSecrets Manager secretAWS Glue connection and S3 endpointAmazon Redshift clusterNavigate toAWS Gluein the AWS console and review the following new resources created by theworkflow_stackCDK stack:Two crawlers to crawl the data in S3Three AWS Glue jobs used within the AWS Glue workflowFive triggers to initiate AWS Glue jobs and crawlersOne AWS Glue workflow to manage the ETL orchestrationAll of these resources could have been deployed within a single stack, but this is intended to be a simple example on how to share resources across multiple stacks. The AWS Identity and Access Management (IAM) role that AWS Glue uses to run the ETL jobs in theworkflow_stack, is also used by Secrets Manager for Amazon Redshift in theredshift_vpc_stack. Inspect the /bin/blog-glue-workflow-stack.ts file to further understandcross stack resource sharing.By performing these steps, you have deployed all of the AWS Glue resources necessary to perform common ETL tasks. You then combined the resources to create an orchestration of tasks using an AWS Glue workflow. All of this was done using IaC with AWS CDK. Your workflow should look like Figure 2.Figure 2. AWS Glue console showing the workflow created by the CDKAs mentioned earlier, you could have started your workflow using a scheduled cron trigger, but you initiated the workflow manually so you had time to review the resources theworkflow_stackCDK deployed, prior to initiation of the workflow. Now that you have reviewed the resources, validate your workflow by initiating it and verifying it runs successfully.From within the AWS Glue console, selectWorkflowsunder ETL.Select the workflow namedglue-workflow, and then selectRunfrom the actions listbox.You can verify the status of the workflow by viewing the run details under the History tab.Your job will take approximately 15 minutes to successfully complete, and your history should look like Figure 3.Figure 3. AWS Glue console showing the workflow as completed after the runThe workflow performs the following tasks:Prepares the ETL scripts by copying the files in the S3 asset bucket to a new folder and renames them with a more relevant name.Initiates a crawler to crawl the raw source data as csv files and adds tables to the Glue Data Catalog.Runs a Python script to perform some ETL tasks on the .csv files and converts them to parquet files.Crawls the parquet files and adds them to the Glue Data Catalog.Loads the parquet files into aDynamicFrameand runs an Amazon Redshift COPY command to load the data into the Amazon Redshift database.After the workflow completes, you can query and perform analytics on the data that was populated in Amazon Redshift. Open the Amazon Redshift Query Editor and run a simple SELECT statement to query thecovid_hiring_tablewhich is the joined Covid-19 case data and hiring data (see Figure 4).Figure 4. Amazon Redshift query editor showing the data that the workflow loaded into the Redshift tablesCleaning upSome resources, like S3 buckets and Amazon DynamoDB tables, must be manually emptied and deleted through the console to be fully removed. To clean up the deployment, delete all objects in the AWS CDK asset bucket in Amazon S3 by using the AWS console to empty the bucket, and then runcdk destroy –allto delete the resources the AWS CDK stacks created in your account. Finally, if you don’t plan on usingAWS CloudFormationassets in this account in the future, you will need to delete the AWS CDK asset stack within the CloudFormation console to remove the AWS CDK asset bucket.ConclusionIn this blog post, you learned how to automate the deployment of AWS Glue workflows using the AWS CDK. This further enhances your continuous integration and delivery (CI/CD) data pipelines by automating the deployment of the ETL jobs and AWS Glue workflow orchestration, providing an efficient, fast, and repeatable way to build and deploy AWS Glue workflows at scale.Although AWS CDK primarily supports level 1 constructs for most AWS Glue resources, new constructs are added continually. See theAWS CDK API Referencefor updates, prior to authoring your stacks, for AWS Glue level 2 construct support. You can find the code used in this blog post in thisGitHub repository, and the AWS CDK in TypeScript reference to theAWS CDK namespace.We hope this blog post helps enrich your work through the skills gained of automating the creation of Glue Workflows, enabling you to quickly build and deploy your own ETL pipelines and run analytical models that power your business.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Field NotesMichael HamiltonMichael is an Analytics Specialist Solutions Architect who enjoys working with customers to solve their complex needs when it comes to data on AWS. He enjoys spending time with his wife and kids outside of work and has recently taken up mountain biking!Joe LosinskiJoe Losinski is a Senior Solutions Architect with AWS. He helps financial, healthcare, and power and utilities companies with business innovation using cloud technologies, and focuses on serverless architectures.
Many customers use AWS Glue workflows to build and orchestrate their ETL (extract-transform-load) pipelines directly in the AWS Glue console using the visual tool to author workflows. This can be time consuming, harder to version control, and error prone due to manual configurations, when compared to managing your workflows as code. To improve your operational […]
Michael Hamilton
Architecture AWS Cloud Development Kit AWS Glue Technical How-to