Genomics workflows, Part 2: simplify Snakemake launches
AWS Architecture BlogGenomics workflows, Part 2: simplify Snakemake launchesby Rostislav Markov and Matt Noyce | on09 DEC 2022| inAmazon CloudWatch,Amazon EC2,Amazon Elastic Container Registry,Amazon Simple Storage Service (S3),Architecture,AWS Lambda,AWS Step Functions|Permalink|ShareGenomics workflowsare high-performance computing workloads. InPart 1of this series, we demonstrated how life-science research teams can focus on scientific discovery without the associated heavy lifting. We usedregeniefor large genome-wide association studies. Our design pattern built onAWS Step FunctionswithAWS BatchandAmazon FSx for Lustre.In Part 2, we explore genomics workloads with built-in workflow logic. Historically, running bioinformatics data pipelines was a manual and error-prone task. Over the last years, multiple workflow management systems have emerged. An example of these is theSnakemakeworkflow management system withTibannaorchestration. We discuss the solution design and how you can fully automate the launch with Amazon Web Services (AWS).Use caseWe focus on the use case of Snakemake, an open-source utility for whole genome sequence mapping indirected acyclic graph (DAG)format. Snakemake usesSnakefilesto declare workflow steps and commands. A Snakefile extends Python syntax to declare workflow steps such as mapping data sets to DAG structure and identifying variants. Consult theSnakemake tutorialfor further information on workflow rules.Snakefiles provide an exception from the general design pattern and an alternative to granular modeling workflow logic inAmazon States Language. In our real-life use case, we used Tibanna to orchestrate Snakemake. Tibanna is an open-source, AWS-native software that runs bioinformatics data pipelines. It supports Snakefile syntax, plus other workflow languages, includingCommon Workflow LanguageandWorkflow Description Language (WDL).We recommend usingAmazon Genomics CLI, if Tibanna is not needed for your use case, andAmazon Omics, if your workflow definitions are compliant with the supported WDL andNextflowspecifications.Solution overviewSnakemake is available as Docker image onGitHub. We push the image toAmazon Elastic Container Registry. Tibanna is also available as Docker image onGitHub—it comes with Snakemake. Consult theTibanna installation guidefor more information.We store Snakefiles onAmazon Simple Storage Service (Amazon S3). We configureS3 Event NotificationsonPUTrequest operations. The event notification triggers anAWS Lambdafunction. The Lambda function launches anAWS Fargatetask, which overrides the task definition command with the appropriate Snakemake start command and arguments.The launched AWS Fargate task pulls the Snakefiles at launch time for each job and prepares the Snakemake initiation commands. Once the Snakefiles are downloaded on the Fargate task, the Snakemake head initiation command is invoked to begin launching jobs using Tibanna. Tibanna invokes a Step Functions state machine which orchestrates the launch of Snakemake onAmazon Elastic Compute Cloud (Amazon EC2).Amazon CloudWatchprovides a consolidated overview of performance metrics, including elapsed time, failed jobs, and error types. You can keep logs of your failed jobs inCloudWatch Logs(Figure 1). You can set upfiltersto match specific error types, plus createsubscriptionsto deliver a real-time stream of your log events toAmazon Kinesisor Lambda for further retry.Figure 1. Solution architecture for Snakemake with Tibanna on AWSImplementation considerationsHere, we describe some of the implementation considerations.Creating SnakefilesThe launching point for the initiation depends on a Snakefile. Each Snakefile may contain one or more samples to be launched. The sheet resides in an S3 bucket. This adds flexibility and the ability to purge any sensitive or restrictive information after the job has been processed.Invoking TibannaIn order to launch Snakemake DAGs using Tibanna, we will need to set up a new Tibanna Unicorn. A Tibanna Unicorn is an Step Functions state machine and a corresponding Lambda function for provisioning EC2 instances.The state machine runs the following sequence:Create EC2 instanceCheck EC2 statusExitAfter the Tibanna Unicorn has been created, we can start a Snakemake DAG using the following sample commands inside of the Fargate task.$ export TIBANNA_DEFAULT_STEP_FUNCTION_NAME=YOUR_UNICORN_PROJECT
$ snakemake --tibanna --tibanna-config spot_instance=true --default-remote-prefix=YOUR_S3_BUCKET/BUCKET_PREFIX --retries 3.The Snakemake command is used with the--tibannaflag to send launch requests to the Step Functions state machine in order to provision EC2 instances and run DAG tasks.We recommend deploying the solution withAWS Serverless Application Modelor theAWS Cloud Development Kit, both of which launchAWS CloudFormation.Logging and troubleshootingWith this solution, each launch will automatically capture and retain start logs in a centralized location inAmazon CloudWatch Logsfor tracing and auditing.If there are issues during the launch of the Tibanna Step Function state machine, such as Amazon EC2 capacity limits, logs will be available in the S3 bucket that was specified during the Tibanna Unicorn creation process. There will be a file available in the format of<EXECUTION_ID>.loginside of the S3 bucket. This information is easily accessible via the command line interface. Use the following command to display specific log results or error messages.tibanna log -j <EXECUTION_ID> -TRetries and EC2 Spot InstancesWe advise to useAmazon EC2 Spot Instances, if possible, for additional cost savings. This option is available in the--tibanna-configarguments with the settingspot_instance=true.This is optional, and you need to create retry logic in the event a Spot Instance gets reclaimed. You can include--retries=3in your Tibanna launch command. This would ensure all rules are retried three times. You can also specify the number of retries for individual rules when defining the Snakemake DAG definition; for example:rule a:
    output:
        "test.txt"
    retries: 3
    shell:
        "curl https://some.unreliable.server/test.txt > {output}"If EC2 Spot Instance capacity is hit, you can automatically switch to usingEC2 On-Demand Instancesinstead. Add thebehavior_on_capacity_limitargument and setretry_without_spot=true.Adding servicesThe presented solution can be adapted to use other compute services supported by Snakemake. These includeAmazon Elastic Kubernetes ServiceandAWS ParallelClusterwithSlurm Workload Managerplus Amazon FSx for Lustre volumes attached to the head node and cluster nodes.To initiate jobs on ParallelCluster, install theAWS Systems Manageragent on the head node. This is the launching point into the cluster and used for submitting jobs to the initiation queue. Systems Manager is a secure way to remotely invoke commands on an EC2 instance without the need for SSH access. You can restrict access to your EC2 instance throughIAM policies.ConclusionIn this blog post, we demonstrated how life-science research teams can simplify the launch of Snakemake using AWS. We used Snakefiles and Tibanna to orchestrate workflow steps. Snakefiles provide an exception from the general design pattern and an alternative to Amazon States Language. File uploads to Amazon S3 served as our launching point for workflow initiations.Stay tuned for Part 3 of this series, in which we create a job manager that administrates multiple workflows.Related informationGenomics on AWSAmazon OmicsAmazon Genomics CLIRostislav MarkovRostislav Markov is principal architect with AWS Professional Services. As technical leader in Strategic Industries, he works with AWS customers and partners on their cloud transformation programs. Outside of work, he enjoys spending time with his family outdoors and exploring New York City culture.Matt NoyceMatt Noyce is a Senior Application Architect, who works primarily with Healthcare and Life Sciences customers in AWS professional services. He works with customers to build, architect, and design solutions that meet their business needs. In his spare time, Matt likes to run, hike, and explore new cities and locations.
Genomics workflows are high-performance computing workloads. In Part 1 of this series, we demonstrated how life-science research teams can focus on scientific discovery without the associated heavy lifting. We used regenie for large genome-wide association studies. Our design pattern built on AWS Step Functions with AWS Batch and Amazon FSx for Lustre. In Part 2, […]
Rostislav Markov
Amazon CloudWatch Amazon EC2 Amazon Elastic Container Registry Amazon Simple Storage Service (S3) Architecture AWS Lambda AWS Step Functions