text: user13937: Coming from devops background, working on automating usual genomics pipelines, I am benchmarking existing pipelines in order to choose cloud resources properly.
  Past week most of work was on testing bwa mem.
  I've noticed that bwa takes about 10 minutes to start processing data. When monitoring bwa resource usage I've noticed that RAM is slowly building up to about 6GB. That is amount of reference indexed data.
  After bwa loads the reference first time, any subsequent run is fast. But why this preloading is happening so slow? My reference data set is already indexed with bwa index.
  When I monitor disk activity with iotop I noticed that disk read is only 5MB/s, when I run bwa mem alignment (only first time), which is very slow. Performance of the drive is lot more than that and a simple copy test confirms that.
  Is it a "limitation" of bwa, is this slow preloading expected to be like that and I should just accept it and move on forward or there is something I don't understand and maybe someone can help me understand what could I do?
  The same "slow preload" happens when I run bwa shm /path/to/ref.
  Then bwa mem starts immediately. But again slow preload is something that is bothering me and I am trying to understand what can I do about it.
  
  user13937: I'll add an answer to my own question, which was asked out of lack of experience with cloud (AWS) EC2 service.
  The way I was testing bwa is from a custom AMI image created for our pipelines.
  The image has "preloaded" reference data that we are using for alignement so that we spare some time downloading reference data from S3 and spare some S3 api calls since we have a lot of files used as reference data.
  Simply our image is single 200GB EBS volume used with AWS ECS AMI where we stored our reference data and created a custom image for Batch out of it. Then we use that AMI for AWS Batch jobs.
  Once I used iotop (after few days of testing) to monitor disk activity I've noticed that bwa loads index into memory very slow, cca 5MB/s. Syncing files from S3 was faster then using this snapshot approach.
  For benchmarking I've used the same AMI manually launched. In the AWS documentation there is part which explains why and how you should optimize EBS when AMI is launched from a snapshot.
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html
  After spining up fresh EC2, syncing reference data and running bwa everything was clear. My EBS volume was slow and had to be initialized first.
  That is all. My lack of EBS storage knowledge was the issue.
  This helped me understand that I should not use custom AMI with ref data stored and will probably use EFS or simply syncing the data from S3 inside Docker entrypoint.
  
original_text: 13937: <p>Coming from devops background, working on automating usual genomics pipelines, I am benchmarking existing pipelines in order to choose cloud resources properly.</p>
  <p>Past week most of work was on testing <code>bwa mem</code>.</p>
  <p>I've noticed that <code>bwa</code> takes about 10 minutes to start processing data. When monitoring <code>bwa</code> resource usage I've noticed that RAM is slowly building up to about 6GB. That is amount of reference indexed data.</p>
  <p>After bwa loads the reference first time, any subsequent run is fast. But why this preloading is happening so slow? My reference data set is already indexed with <code>bwa index</code>.</p>
  <p>When I monitor disk activity with <code>iotop</code> I noticed that <code>disk read</code> is only <code>5MB/s</code>, when I run <code>bwa mem</code> alignment (only first time), which is very slow. Performance of the drive is lot more than that and a simple <code>copy</code> test confirms that.</p>
  <p>Is it a &quot;limitation&quot; of <code>bwa</code>, is this slow preloading expected to be like that and I should just accept it and move on forward or there is something I don't understand and maybe someone can help me understand what could I do?</p>
  <p>The same &quot;slow preload&quot; happens when I run <code>bwa shm /path/to/ref</code>.
  Then <code>bwa mem</code> starts immediately. But again slow preload is something that is bothering me and I am trying to understand what can I do about it.</p>
   13937: <p>I'll add an answer to my own question, which was asked out of lack of experience with cloud (AWS) EC2 service.</p>
  <p>The way I was testing <code>bwa</code> is from a custom AMI image created for our pipelines.</p>
  <p>The image has &quot;preloaded&quot; reference data that we are using for alignement so that we spare some time downloading reference data from <code>S3</code> and spare some <code>S3</code> api calls since we have a lot of files used as reference data.</p>
  <p>Simply our image is single 200GB EBS volume used with <code>AWS ECS AMI</code> where we stored our reference data and created a custom image for Batch out of it. Then we use that AMI for AWS Batch jobs.</p>
  <p>Once I used <code>iotop</code> (after few days of testing) to monitor disk activity I've noticed that <code>bwa</code> loads index into memory very slow, cca <code>5MB/s</code>. Syncing files from S3 was faster then using this <code>snapshot</code> approach.</p>
  <p>For benchmarking I've used the same AMI manually launched. In the AWS documentation there is part which explains why and how you should optimize EBS when AMI is launched from a snapshot.
  <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html</a></p>
  <p>After spining up fresh EC2, syncing reference data and running <code>bwa</code> everything was clear. My EBS volume was slow and had to be initialized first.</p>
  <p>That is all. My lack of EBS storage knowledge was the issue.</p>
  <p>This helped me understand that I should not use custom AMI with ref data stored and will probably use EFS or simply syncing the data from S3 inside Docker entrypoint.</p>
  