Genomics workflows, Part 4: processing archival data
AWS Architecture BlogGenomics workflows, Part 4: processing archival databy Rostislav Markov and Matt Noyce | on04 JAN 2023| inAmazon DynamoDB,Amazon S3 Glacier,Amazon S3 Glacier Deep Archive,Amazon Simple Queue Service (SQS),Architecture,AWS Fargate,AWS Lambda|Permalink|ShareGenomics workflowsanalyze data at petabyte scale. After processing is complete, data is often archived in cold storage classes. In some cases, like studies on the association ofDNAvariants against larger datasets, archived data is needed for further processing. This means manually initiating the restoration of each archived object and monitoring the progress. Scientists require a reliable process for on-demand archival data restoration so their workflows do not fail.In Part 4 of this series, we look into genomics workloads processing data that is archived withAmazon Simple Storage Service (Amazon S3). We design a reliable data restoration process that informs the workflow when data is available so it can proceed. We build on top of the design pattern laid out in Parts 1-3 of this series. We use event-driven and serverless principles to provide the most cost-effective solution.Use caseOur use case focuses on data inAmazon Simple Storage Service Glacier (Amazon S3 Glacier)storage classes. The S3 Glacier Instant Retrieval storage class provides the lowest-cost storage for long-lived data that is rarely accessed but requires retrieval in milliseconds.TheS3 Glacier Flexible RetrievalandS3 Glacier Deep Archiveprovide further cost savings, with retrieval times ranging from minutes to hours. We focus on the latter in order to provide the most cost-effective solution.You must first restore the objects before accessing them. Our genomics workflow will pause until the data restore completes. The requirements for this workflow are:Reliable launch of the restore so our workflow doesn’t fail (due toS3 Glacier service quotas, or because not all objects were restored)Event-driven design to mirror the event-driven nature of genomics workflows and perform the retrieval upon requestCost-effective and easy-to-manage by using serverless servicesUpfront detection of archived data when formulating the genomics workflow task, avoiding idle computational tasks that incur costScalable and elastic to meet the restore needs of large, archived datasetsSolution overviewGenomics workflows take multiple input parameters to prepare the initiation, such as launch ID, data path, workflow endpoint, and workflow steps. We store this data, including workflow configurations, in an S3 bucket. AnAWS Fargatetask reads from the S3 bucket and prepares the workflow. It detects if the input parameters include S3 Glacier URLs.We useAmazon Simple Queue Service (Amazon SQS)to decouple S3 Glacier index creation from object restore actions (Figure 1). This increases the reliability of our process.Figure 1. Solution architecture for S3 Glacier object restoreAnAWS Lambdafunction creates the index of all objects in the specifiedS3 bucket URLsand submits them as an SQS message.Another Lambda function polls the SQS queue and submits the request(s) to restore the S3 Glacier objects toS3 Standard storage class.The function writes the job ID of each S3 Glacier restore request toAmazon DynamoDB. After the restore is complete, Lambda sets the status of the workflow toREADY. Only then can any computing jobs start, such as withAWS Batch.Implementation considerationsWe consider the use case ofSnakemakewithTibanna, which we detailed inPart 2of this series. This allows us to dive deeper on launch details.Snakemake is an open-source utility for whole-genome-sequence mapping indirected acyclic graphformat. Snakemake usesSnakefilesto declare workflow steps and commands. Tibanna is an open-source, AWS-native software that runs bioinformatics data pipelines. It supports Snakefile syntax, plus other workflow languages, includingCommon Workflow LanguageandWorkflow Description Language (WDL).We recommend usingAmazon Genomics CLIif Tibanna is not needed for your use case, orAmazon Omicsif your workflow definitions are compliant with the supported WDL andNextflowspecifications.Formulate the restore requestThe Snakemake Fargate launch container detects if the S3 objects under the requested S3 bucket URLs are stored in S3 Glacier. The Fargate launch container generates and puts a JSONbinary base call (BCL)configuration file into an S3 bucket and exits successfully. This file includes the launch ID of the workflow, corresponding with the DynamoDB item key, plus the S3 URLs to restore.Query the S3 URLsOnce the JSON BCL configuration file lands in this S3 bucket, theS3 Event NotificationPutObjectevent invokes a Lambda function. This function parses the configuration file and recursively queries for all S3 object URLs to restore.Initiate the restoreThe main Lambda function then submits messages to the SQS queue that contains the full list of S3 URLs that need to be restored. SQS messages also include the launch ID of the workflow. This is to ensure we can bind specific restoration jobs to specific workflow launches. If all S3 Glacier objects belong toFlexible Retrieval storage class, the Lambda function puts the URLs in a single SQS message, enabling restoration withBulk Glacier Job Tier. The Lambda function also sets the status of the workflow to WAITING in the corresponding DynamoDB item. The WAITING state is used to notify the end user that the job is waiting on the data-restoration process and will continue once the data restoration is complete.A secondary Lambda function polls for new messages landing in the SQS queue. This Lambda function submits the restoration request(s)—for example, as a free-of-charge Bulk retrieval—using theRestoreObjectAPI. The function subsequently writes the S3 Glacier Job ID of each request in our DynamoDB table. This allows the main Lambda function to check if all Job IDs associated with a workflow launch ID are complete.Update statusThe status of our workflow launch will remainWAITINGas long as the Glacier object restore is incomplete. The AWS CloudTraillogsof completed S3 Glacier Job IDs invoke our main Lambda function (via anAmazon EventBridge rule) to update the status of the restoration job in our DynamoDB table. With each invocation, the function checks if all Job IDs associated with a workflow launch ID are complete.After all objects have been restored, the function updates the workflow launch with statusREADY. This launches the workflow with the same launch ID prior to the restore.ConclusionIn this blog post, we demonstrated how life-science research teams can make use of their archival data for genomic studies. We designed an event-driven S3 Glacier restore process, which retrieves data upon request. We discussed how to reliably launch the restore so our workflow doesn’t fail. Also, we determined upfront if an S3 Glacier restore is needed and used the WAITING state to prevent our workflow from failing.With this solution, life-science research teams can save money using Amazon S3 Glacier without worrying about their day-to-day work or manually administering S3 Glacier object restores.Related informationGenomics workflows, Part 1: automated launchesGenomics workflows, Part 2: simplify Snakemake launchesGenomics workflows, Part 3: automated workflow managerGenomics workflows, Part 5: automated benchmarkingGenomics on AWSAmazon OmicsAmazon Genomics CLIRostislav MarkovRostislav Markov is principal architect with AWS Professional Services. As technical leader in Strategic Industries, he works with AWS customers and partners on their cloud transformation programs. Outside of work, he enjoys spending time with his family outdoors and exploring New York City culture.Matt NoyceMatt Noyce is a Senior Application Architect, who works primarily with Healthcare and Life Sciences customers in AWS professional services. He works with customers to build, architect, and design solutions that meet their business needs. In his spare time, Matt likes to run, hike, and explore new cities and locations.
Genomics workflows analyze data at petabyte scale. After processing is complete, data is often archived in cold storage classes. In some cases, like studies on the association of DNA variants against larger datasets, archived data is needed for further processing. This means manually initiating the restoration of each archived object and monitoring the progress. Scientists […]
Rostislav Markov
Amazon DynamoDB Amazon S3 Glacier Amazon S3 Glacier Deep Archive Amazon Simple Queue Service (SQS) Architecture AWS Fargate AWS Lambda