Genomics workflows, Part 3: automated workflow manager
AWS Architecture BlogGenomics workflows, Part 3: automated workflow managerby Rostislav Markov and Matt Noyce | on21 DEC 2022| inAmazon CloudWatch,Amazon DynamoDB,Amazon Simple Storage Service (S3),Architecture,AWS Batch,AWS Fargate,AWS Lambda|Permalink|ShareGenomics workflowsare high-performance computing workloads. Life-science research teams make use of various genomics workflows. With each invocation, they specify custom sets of data and processing steps, and translate them into commands. Furthermore, team members stay to monitor progress and troubleshoot errors, which can be cumbersome, non-differentiated, administrative work.In Part 3 of this series, we describe the architecture of a workflow manager that simplifies the administration of bioinformatics data pipelines. The workflow manager dynamically generates the launch commands based on user input and keeps track of the workflow status. This workflow manager can be adapted to many scientific workloads—effectively becoming a bring-your-own-workflow-manager for each project.Use caseInPart 1, we demonstrated how life-science research teams can use Amazon Web Services to remove the heavy lifting of conducting genomic studies, and our design pattern was built onAWS Step FunctionswithAWS Batch. We mentioned that we’ve worked with life-science research teams to put failed job logs ontoAmazon DynamoDB. Some teams prefer to use command-line interface tools, such as theAWS Command Line Interface; other interfaces, such asPyBDAwithApache Spark, orCWL experimental grammarin combination with theAmazon Simple Storage Service (Amazon S3)API, are also used when access to theAWS Management Consoleis prohibited. In our use case, scientists used the console to easily update table items, plus initiate retry viaDynamoDB streams.In this blog post, we extend this idea to a new frontend layer in our design pattern. This layer automates command generation and monitors the invocations of a variety of workflows—becoming a workflow manager. Life-science research teams use multiple workflows for different datasets and use cases, each with different syntax and commands. The workflow manager we create removes the administrative burden of formulating workflow-specific commands and tracking their launches.Solution overviewWe allow scientists to upload their requested workflow configuration as objects in Amazon S3. We useS3 Event NotificationsonPUTrequests to invoke anAWS Lambdafunction. The function parses the uploaded S3 object and registers the new launch request as a DynamoDB item using thePutItemoperation. Eachitemcorresponds with a distinct launch request, stored as key-value pair. Item values store the:S3 data path containing genomic datasetsWorkflow endpointPreferred compute service (optional)Another Lambda function monitors for change data captures in the DynamoDB Stream (Figure 1). With eachPutItemoperation, the Lambda function prepares a workflow invocation, which includes translating the user input into the syntax and launch commands of the respective workflow.In the case ofSnakemake(discussed inPart 2), the function creates aSnakefilethat declares processing steps and commands. The function spins up anAWS Fargatetask that builds the computational tasks, distributes them with AWS Batch, and monitors for completion. An AWS Step Functions state machine orchestrates job processing, for example, initiated byTibanna.Amazon CloudWatchprovides a consolidated overview of performance metrics, like time elapsed, failed jobs, and error types. We store log data, including status updates and errors, inAmazon CloudWatch Logs. A third Lambda function parses those logs and updates the status of each workflow launch request in the corresponding DynamoDB item (Figure 1).Figure 1. Workflow manager for genomics workflowsImplementation considerationsIn this section, we describe some of our past implementation considerations.Register new workflow requestsDynamoDB items are key-value pairs. We use launch IDs as key, and the value includes the workflow type, compute engine, S3 data path, the S3 object path to the user-defined configuration file and workflow status. Our Lambda function parses the configuration file and generates all commands plus ancillary artifacts, such as Snakefiles.Launch workflowsLaunch requests are picked by a Lambda function from the DynamoDB stream. The function has the following required parameters:Launch ID: unique identifier of each workflow launch requestConfiguration file: the Amazon S3 path to the configuration sheet with launch details (ins3://bucket/object format)Compute service (optional): our workflow manager allows to select a particular service on which to run computational tasks, such asAmazon Elastic Compute Cloud (Amazon EC2)orAWS ParallelClusterwithSlurm Workload Manager. The default is the pre-defined compute engine.These points assume that the configuration sheet is already uploaded into an accessible location in an S3 bucket. This will issue a new Snakemake Fargate launch task. If either of the parameters is not provided or access fails, the workflow manager returnsMissingRequiredParametersError.Log workflow launchesLogs are written to CloudWatch Logs automatically. We write the location of the CloudWatch log group and log stream into the DynamoDB table. To send logs to Amazon CloudWatch, specify theawslogs driverin the Fargate task definition settings in your provisioning template.Our Lambda function writes Fargate task launchlogsfrom CloudWatch Logs to our DynamoDB table. For example,OutOfMemoryErrorcan occur if the process utilizes more memory than the container is allocated.AWS Batch job state logs are written to the following log group in CloudWatch Logs:/aws/batch/job. Our Lambda function writes status updates to the DynamoDB table. AWS Batch jobs may encounter errors, such as being stuck inRUNNABLEstate.Manage state transitionsWe manage the status of each job in DynamoDB. Whenever a Fargate task changes state, it is picked up by a CloudWatch rule that references the Fargate compute cluster. This CloudWatch rule invokes a notifier Lambda function that updates the workflow status in DynamoDB.ConclusionIn this blog post, we demonstrated how life-science research teams can simplify genomic analysis across an array of workflows. These workflows usually have their own command syntax and workflow management system, such as Snakemake. The presented workflow manager removes the administrative burden of preparing and formulating workflow launches, increasing reliability.The pattern is broadly reusable with any scientific workflow and related high-performance computing systems. The workflow manager provides persistence to enable historical analysis and comparison, which enables us to automatically benchmark workflow launches for cost and performance.Stay tuned for Part 4 of this series, in which we explore how to enable our workflows to process archival data stored inAmazon Simple Storage Service Glacierstorage classes.Related informationGenomics workflows, Part 1: automated launchesGenomics workflows, Part 2: simplify Snakemake launchesGenomics on AWSAmazon OmicsAmazon Genomics CLIRostislav MarkovRostislav Markov is principal architect with AWS Professional Services. As technical leader in Strategic Industries, he works with AWS customers and partners on their cloud transformation programs. Outside of work, he enjoys spending time with his family outdoors and exploring New York City culture.Matt NoyceMatt Noyce is a Senior Application Architect, who works primarily with Healthcare and Life Sciences customers in AWS professional services. He works with customers to build, architect, and design solutions that meet their business needs. In his spare time, Matt likes to run, hike, and explore new cities and locations.
Genomics workflows are high-performance computing workloads. Life-science research teams make use of various genomics workflows. With each invocation, they specify custom sets of data and processing steps, and translate them into commands. Furthermore, team members stay to monitor progress and troubleshoot errors, which can be cumbersome, non-differentiated, administrative work. In Part 3 of this series, […]
Rostislav Markov
Amazon CloudWatch Amazon DynamoDB Amazon Simple Storage Service (S3) Architecture AWS Batch AWS Fargate AWS Lambda