Genomics workflows, Part 1: automated launches
AWS Architecture BlogGenomics workflows, Part 1: automated launchesby Rostislav Markov, Balaji Desikachari, and Matt Noyce | on16 NOV 2022| inAmazon Elastic Container Registry,Architecture,AWS Batch,AWS Lambda,AWS Step Functions|Permalink|ShareGenomics workflowsare high-performance computing workloads. Traditionally, they run on-premises with a collection of scripts. Scientists run and manage these workflows manually, which slows down the product development lifecycle. Scientists spend time to administer workflows and handle errors on a day-to-day basis. They also lack sufficient compute capacity on-premises.In Part 1 of this series, we demonstrate how life sciences companies can use Amazon Web Services (AWS) to remove the traditional heavy lifting associated with genomic studies. We useAWS Step Functionsto orchestrate workflow steps, including error handling. WithAWS Batch, we horizontally scale-out the analytic tasks for optimal performance. This allows genome scientists to focus on scientific discovery while AWS runs their workflows.Use caseWorkflow systems used for genomic analysis includeCromwell,Nextflow, andregenie. These high-performance computing systems share the following requirements:Fast access to datasets at petabyte scaleParallel task distribution, with horizontal compute scale-outData processing in batches following a specific sequence of data analysis steps, which vary by use caseWe explore the use case of regenie. regenie is a common, open-source utility for whole-genome regression modelling of largegenome-wide association studies(GWAS). GWAS compareDNAdatasets of individuals with a specific trait or disease. The intent is to associate the identified trait/disease with DNA variants. Among other positive results, this helps identify at-risk patients, plus testing and prevention opportunities.regenie is a C++ program that runs intwo steps:The first step searches for variants associated with a specific trait in a dataset of individuals with the trait, in order to create a whole-genome regression model that captures the variance.The second step validates for association with the identified variants against a larger dataset, typically in the scale of petabytes, and launches a sequence of tasks run on data batches.Solution overviewThe entire regenie workflow and associated tasks of attaching and deleting file-share access to sample data, as well as spinning up compute instances for parallel computing, can be orchestrated with Step Functions. We useAmazon FSx for Lustreas a high-performance, transient file system providing file access to the datasets stored in anAmazon Simple Storage Service (Amazon S3)bucket. AWS Batch allows us to programmatically spin up multipleAmazon Elastic Compute Cloud (Amazon EC2)instances on which regenie can distribute parallel computing tasks. We do this with anAWS Lambdafunction that calculates the number of required batch jobs based on the requested size of samples per batch.regenie is available as Docker image onGitHub. We push the image toAmazon Elastic Container Registryfrom which AWS Batch can pull it with the creation of new jobs at launch time. The Step Functions state machine is initiated by a Lambda function, with interactive user input. In the past, scientists have also directly interacted with theStep Functions APIvia theAWS Management Consoleor by runningstart-executionin theAWS Command Line Interfaceand passing aJSONfile with the input parameters.Amazon CloudWatchprovides a consolidated overview of performance metrics, including elapsed time, failed jobs, and error types. You can keep logs of your failed jobs inAmazon CloudWatch Logs(Figure 1). You can set upfiltersto match specific error types, plus createsubscriptionsto deliver a real-time stream of your log events toAmazon Kinesisor AWS Lambda for further retry.Figure 1. Solution overview for automating regenie workflows on AWSAlternatively, the Step Functions workflow triggers another Lambda function, which puts failed job logs toAmazon DynamoDB. In the past, we have used this to ease data access and manipulation via the AWS management console. Scientists updated table items andDynamoDB Streamsinitiated the retry.Workflow automationWith each invocation, Step Functions initiates a new instance of the state machine. AWS documentation provides an overview of the APIquotas. Step Functions allows the modeling of the entire workflow, including custom applicationerror handling.Mapstate improves performance by parallelizing workflow branches.The state machine initiates the build of the file system and, once it’s ready, creates a data repository association with the sample data stored on Amazon S3. It waits until thedata repository associationis complete and proceeds with the calculation of batch jobs, based on a user-defined number of samples to be processed per batch job (Figure 2). This is essential to determine the amount of compute instances required for data processing.Figure 2. AWS Step Functions workflow for regenie: initialize file accessNext, the state machine builds the commands to launch the regenie steps, as requested by the user, and submit the jobs for AWS Batch (Figure 3). The workflow checks if a specific version of regenie was requested by the user, otherwise, it defaults to the version of regenie on the container.Then, we build the commands to initiate the two regenie steps. Step 2 may need to run in multiple iterations on different datasets (more often than Step 1). This is also determined with user input at initiation of the workflow. With Step Functions, we create runner logic to build the set of commands dynamically. This pattern is applicable to other scientific workloads, as well.Figure 3. AWS Step Functions workflow for regenie: prepare and submit jobsOnce jobs are submitted, the workflow proceeds (by default) with the initiation of Step 1 of regenie; if requested by the user, the workflow will proceed directly to step 2 (Figure 4).Any errors during batch launch leading to the failure of a job are passed, in this case, to a Lambda function. We configure the Lambda function to write the failed job logs to Amazon DynamoDB or as S3 objects.Figure 4. AWS Step Functions workflow for regenie: launch jobsFinally, the Step Functions workflow checks for pending errors and confirms that all jobs have finished their initiation. Then, it deletes the file system and data repository association and ends the workflow instance (Figure 5).Figure 5. AWS Step Functions workflow for regenie: complete error handling and delete file systemAs demonstrated, we can automate the entire process, from data access to verifying job completion and cleaning-up transient resources. This removes manual error handling and retry, plus reduces the overall cost of running regenie workflows. We also showed in Figure 3 that you can build commands dynamically for different scientific workloads.ConclusionIn this blog post, we addressed a common pain point in the daily work of life sciences research teams. Traditionally, they had to run genomics workflows manually on limited compute capacity. Moving those workflows to AWS eliminates the heavy lifting of running scripts manually and expedites computational cycles. This allows research teams to stay focused on scientific discovery.We recommend a thorough performance testing when setting up your genomics workflows. This includes determining the most suitable EC2 instance size. Some workflows, such as regenie, are single-threaded and benefit from horizontal scale-out of the number of instances but not from vertical scale-out of instance sizes.Related informationGenomics workflows on AWSGenomics on AWSAmazon Genomics CLIRostislav MarkovRostislav Markov is principal architect with AWS Professional Services. As technical leader in Strategic Industries, he works with AWS customers and partners on their cloud transformation programs. Outside of work, he enjoys spending time with his family outdoors and exploring New York City culture.Balaji DesikachariBalaji is a Sr Cloud Infrastructure Architect with AWS Professional Services Team. He enjoys solving complex problems for the customers and building custom solutions tailored to their needs. In his free time, Balaji spends time with friends and family watching movies or traveling.Matt NoyceMatt Noyce is a Senior Application Architect, who works primarily with Healthcare and Life Sciences customers in AWS professional services. He works with customers to build, architect, and design solutions that meet their business needs. In his spare time, Matt likes to run, hike, and explore new cities and locations.
Genomics workflows are high-performance computing workloads. Traditionally, they run on-premises with a collection of scripts. Scientists run and manage these workflows manually, which slows down the product development lifecycle. Scientists spend time to administer workflows and handle errors on a day-to-day basis. They also lack sufficient compute capacity on-premises. In Part 1 of this series, […]
Rostislav Markov
Amazon Elastic Container Registry Architecture AWS Batch AWS Lambda AWS Step Functions