Genomics workflows, Part 5: automated benchmarking
AWS Architecture BlogGenomics workflows, Part 5: automated benchmarkingby Rostislav Markov and Matt Noyce | on24 MAR 2023| inAmazon DynamoDB,Amazon EC2,Amazon Elastic Kubernetes Service,Amazon EventBridge,Amazon FSx for Lustre,Architecture,AWS Batch,AWS Lambda,AWS Step Functions|Permalink|ShareLaunching and runninggenomics workflowscan take hours and involves large pools of compute instances that process data at a petabyte scale.Benchmarkinghelps you evaluate workflow performance and discover faster and cheaper ways of running them.In practice, performance evaluations happen irregularly because of the associated heavy lifting. In this blog post, we discuss how life-science research teams can automate evaluations.Business benefitsAn automated benchmarking solution provides:more accurate enterprise resource planning by performing historical analytics,lower cost to the business by comparing performance on different resource types, andcost transparency to the business by quantifying periodical chargeback.We’ve used automated benchmarking to compare processing times on different services such asAmazon Elastic Compute Cloud(Amazon EC2),AWS Batch,AWS ParallelCluster,Amazon Elastic Kubernetes Service(Amazon EKS), and on-premises HPC clusters. Scientists, financiers, technical leaders, and other stakeholders can build reports and dashboards to compare consumption data by consumer, workflow type, and time period.Design patternOur automated benchmarking solution measures performance on two dimensions:Timing: measures the duration of a workflow launch on a specific datasetPricing: measures the associated costThis solution can be extended to other performance metrics such as iterations per second or process/thread distribution across compute nodes.Our requirements include the following:Consistent measurement of timing based on workflow status (such as preparing, waiting, ready, running, failed, complete)Extensible pricing models based on unit prices (the Amazon EC2Spot priceat a specific period of time compared to Amazon EC2On-Demand pricing)Scalable, cost-efficient, and flexible data store enabling historical benchmarking and estimationsMinimal infrastructure management overheadWe choose a serverless design pattern usingAWS Step Functionsorchestration,AWS Lambdafor our application code, andAmazon DynamoDBto track workflow launch IDs and states (as described inPart 3). We assume that the genomics workflows run onAWS Batchwith genomics data onAmazon FSx for Lustre(Part 1). AWS Step Functions allows us to break down processing into smaller steps and avoid monolithic application code. Our evaluation process runs in four steps:Monitor for completed workflow launches in theDynamoDB streamusing anAmazon EventBridge pipewith a Step Functions workflow as target. This event-driven approach is more efficient than periodic polling and avoids custom code for parsing status and cost values in all records of the DynamoDB stream.Collect a list of all compute resources associated with the workflow launch. Design a Lambda function that queries theAWS Batch API(seePart 1) to describe compute environment parameters like the Amazon EC2 instance IDs and their details, such as processing times, instance family/size, and allocation strategy (for example,Spot Instances,Reserved Instances,On-Demand Instances).Calculate the cost of all consumed resources. We achieve this with another Lambda function, which calculates the total price based on unit prices from theAWS Price List Query API.Our state machine updates the total price in the DynamoDB table without the need for additional application code.Figure 1 visualizes these steps.Figure 1. Automated benchmarking of genomics workflowsImplementation considerationsAWS Step Functions orchestrates our benchmarking workflow reliably and makes our application code easy to maintain. Figure 2 summarizes the state machine transitions that we’ll describe.Figure 2. AWS Step Functions state machine for automated benchmarkingGather consumption detailsConfigure the DynamoDB stream view type toNew imageso that the entire item is passed through as it appears after it was changed. We set up anAmazon EventBridge pipewithevent filteringand the DynamoDB stream as asource. Our event filter usesmultiple matchingon records with a status of COMPLETE, but no cost entry in order to avoid an infinite loop. Once our state machine has updated the DynamoDB item with the workflow price, the resulting record in the DynamoDB stream will not pass our event filter.The syntax of our event filter is as follows:{
  "dynamodb": {
    "NewImage": {
      "status": {
        "S": ["COMPLETE"]
      },
      "totalCost": {
        "S": [{
          "exists": false
        }]
      }
    }
  }
}We use aninput transformerto simplify follow-on parsing by removing unnecessary metadata from the event.The consumed resources included in the stream record are the auto-scaling group ID for AWS Batch and the Amazon FSx for Lustre volume ID. We use theDescribeJobs API(describe_jobsin Boto3) to determine which compute resources were used. If the response is a list of EC2 instances, we then look up consumption information including start and end times using theListJobs API(list_jobsin Boto3) for each compute node. We usedescribe_volumeswith filters on the identified EC2 instances to obtain the size and type ofAmazon Elastic Block Store(Amazon EBS) volumes.Calculate pricesAnother Lambda function obtains the associated unit prices of all consumed resources using theGetProductsrequest ofAWS Price List Query API(get_productsin Boto3) and then parsing the pricePerUnit value. For Spot Instances, we usedescribe_spot_price_historyof the EC2 client in Boto3 and specify the time range and instance types for which we want to receive prices.Calculate the price of workflow launches based on the following factors:Number and size of EC2 instances in auto-scaling node groupsSize of EBS volumes andAmazon FSx for LustreProcessing durationOur Python-based Lambda function calculates the total, rounds it, and delivers the price breakdown in the following format:total_cost: str, instance_cost: str, volume_cost: str, filesystem_cost: strLastly, we put the price breakdown to the DynamoDB table using UpdateItem directly from theAmazon States Language.Note that AWS credits and enterprise discounts might not be reflected in the responses of theAWS Price List Query APIunless applied to the particular AWS account. This is often considered best practice in light of least-privilege considerations.In the past, we’ve also usedAWS Cost Explorerinstead of theAWS Price List API. AWS Cost Explorer data is updated at least once every 24 hours. You can denote the pending price status in the DynamoDB table item and use theWaitstate to delay the calculation process.The presented solution can be extended to other compute services such asAmazon Elastic Kubernetes Service(Amazon EKS). For Amazon EKS, events are enriched with the cluster ID from the DynamoDB table and the price calculation should also include control plane costs.ConclusionLife-science research teams usebenchmarkingto compare workflow performance and inform their architectural decisions. Such evaluations are effort-intensive and therefore done irregularly.In this blog post, we showed how life-science research teams can automate benchmarking for their scientific workflows. The insights teams gain from automated benchmarking indicate continuous optimization opportunities, such as by adjusting compute node configuration. The evaluation data is also available on demand for other purposes including chargeback.Stay tuned for our next post in which we show how to use historical benchmarking data for price estimations of future workflow launches.Related informationGenomics workflows, Part 1: automated launchesGenomics workflows, Part 2: simplify Snakemake launchesGenomics workflows, Part 3: automated workflow managerGenomics workflows, Part 4: processing archival dataGenomics workflows on AWSGenomics on AWSAmazon Genomics CLIRostislav MarkovRostislav Markov is principal architect with AWS Professional Services. As technical leader in Strategic Industries, he works with AWS customers and partners on their cloud transformation programs. Outside of work, he enjoys spending time with his family outdoors and exploring New York City culture.Matt NoyceMatt Noyce is a Senior Application Architect, who works primarily with Healthcare and Life Sciences customers in AWS professional services. He works with customers to build, architect, and design solutions that meet their business needs. In his spare time, Matt likes to run, hike, and explore new cities and locations.
Launching and running genomics workflows can take hours and involves large pools of compute instances that process data at a petabyte scale. Benchmarking helps you evaluate workflow performance and discover faster and cheaper ways of running them. In practice, performance evaluations happen irregularly because of the associated heavy lifting. In this blog post, we discuss […]
Rostislav Markov
Amazon DynamoDB Amazon EC2 Amazon Elastic Kubernetes Service Amazon EventBridge Amazon FSx for Lustre Architecture AWS Batch AWS Lambda AWS Step Functions