original_docstring:  Wrap toil-vg mapeval. 
          
          Evaluates realignments (to the linear reference and to a set of graphs)
          of reads simulated from a single "base" graph. Realignments are
          evaluated based on how close the realignments are to the original
          simulated source position. Simulations are done inside this function.
  
          sim_xg_paths are xg filenames used for simulation. base_xg_path is an xg
          filename used for everything else like annotation and mapping.
          sim_xg_paths can be [base_xg_path]
          
          Simulates the given number of reads (reads), from the given XG files
          (sim_xg_paths), optionally restricted to a set of named embedded paths
          (source_path_namess). Uses the given FASTA (fasta_path) as a BWA
          reference for comparing vg and BWA alignments within mapeval.
          (Basically, BWA against the linear reference functions as a negative
          control "graph" to compare against the real test graphs.)
          
          test_index_bases specifies a list of basenames (without extension) for a
          .xg, .gcsa, and .gcsa.lcp file set, one per of graph that is to be
          compared.
          
          test_names has one entry per graph to be compared, and specifies where
          the realigned read GAM files should be saved.
          
          score_baseline_name, if not None, is a name from test_names to be used
          as a score baseline for comparing all the realignment scores against.
          
          tag is a unique slug for this test/run, which determines the Toil job
          store name to use, and the location where the output files should be
          saved.
          
          
docstring: Wrap toil-vg mapeval.
  Evaluates realignments (to the linear reference and to a set of graphs)
  of reads simulated from a single "base" graph. Realignments are
  evaluated based on how close the realignments are to the original
  simulated source position. Simulations are done inside this function.
  
  sim_xg_paths are xg filenames used for simulation. base_xg_path is an xg
  filename used for everything else like annotation and mapping.
  
  Simulates the given number of reads (reads), from the given XG files
  (sim_xg_paths), optionally restricted to a set of named embedded paths
  (source_path_namess). Uses the given FASTA (fasta_path) as a BWA
  reference for comparing vg and BWA alignments within mapeval.
  (Basically, BWA against the linear reference functions as a negative
  control "graph" to compare against the real test graphs.)
  
  test_index_bases specifies a list of basenames (without extension) for a
  .xg, .gcsa, and .gcsa.lcp file set, one per of graph that is to be
  compared.
  
  test_names has one entry per graph to be compared, and specifies where
  the realigned read GAM files should be saved.
  
  score_baseline_name, if not None, is a name from test_names to be used
  as a score baseline for comparing all the realignment scores against.
  
  tag is a unique slug for this test/run, which determines the Toil job
  store name to use, and the location where the output files should be
  saved.
docstring_tokens: Wrap toil - vg mapeval . Evaluates realignments ( to the linear reference and to a set of graphs ) of reads simulated from a single " base " graph . Realignments are evaluated based on how close the realignments are to the original simulated source position . Simulations are done inside this function . sim_xg_paths are xg filenames used for simulation . base_xg_path is an xg filename used for everything else like annotation and mapping . Simulates the given number of reads ( reads ) from the given XG files ( sim_xg_paths ) optionally restricted to a set of named embedded paths ( source_path_namess ) . Uses the given FASTA ( fasta_path ) as a BWA reference for comparing vg and BWA alignments within mapeval . ( Basically BWA against the linear reference functions as a negative control " graph " to compare against the real test graphs . ) test_index_bases specifies a list of basenames ( without extension ) for a . xg . gcsa and . gcsa . lcp file set one per of graph that is to be compared . test_names has one entry per graph to be compared and specifies where the realigned read GAM files should be saved . score_baseline_name if not None is a name from test_names to be used as a score baseline for comparing all the realignment scores against . tag is a unique slug for this test / run which determines the Toil job store name to use and the location where the output files should be saved .
code: def _mapeval_vg_run(self, reads, base_xg_path, sim_xg_paths,
                          source_path_names, fasta_path, test_index_bases,
                          test_names, score_baseline_name,  mappers,
                          paired_only, sim_opts, sim_fastq, more_mpmap_opts, tag):
          """ Wrap toil-vg mapeval. 
          
          Evaluates realignments (to the linear reference and to a set of graphs)
          of reads simulated from a single "base" graph. Realignments are
          evaluated based on how close the realignments are to the original
          simulated source position. Simulations are done inside this function.
  
          sim_xg_paths are xg filenames used for simulation. base_xg_path is an xg
          filename used for everything else like annotation and mapping.
          sim_xg_paths can be [base_xg_path]
          
          Simulates the given number of reads (reads), from the given XG files
          (sim_xg_paths), optionally restricted to a set of named embedded paths
          (source_path_namess). Uses the given FASTA (fasta_path) as a BWA
          reference for comparing vg and BWA alignments within mapeval.
          (Basically, BWA against the linear reference functions as a negative
          control "graph" to compare against the real test graphs.)
          
          test_index_bases specifies a list of basenames (without extension) for a
          .xg, .gcsa, and .gcsa.lcp file set, one per of graph that is to be
          compared.
          
          test_names has one entry per graph to be compared, and specifies where
          the realigned read GAM files should be saved.
          
          score_baseline_name, if not None, is a name from test_names to be used
          as a score baseline for comparing all the realignment scores against.
          
          tag is a unique slug for this test/run, which determines the Toil job
          store name to use, and the location where the output files should be
          saved.
          
          """
  
          job_store = self._jobstore(tag)
          out_store = self._outstore(tag)
  
          # start by simulating some reads
          # TODO: why are we using strings here when we could use much safer lists???
          opts = ' '.join(self._toil_vg_io_opts()) + ' '
          if self.vg_docker:
              opts += '--vg_docker {} '.format(self.vg_docker)
          if self.container:
              opts += '--container {} '.format(self.container)
          # note, using the same seed only means something if using same
          # number of chunks.  we make that explicit here
          sim_chunks = int(reads / self.sim_chunk_size)
          if reads % self.sim_chunk_size:
              sim_chunks += 1
          opts += '--maxCores {} --sim_chunks {} --seed {} '.format(self.cores, sim_chunks, 8)
          if sim_opts:
              # Make sure to prefix the sim options string with a space so that
              # Toil's parser doesn't see a leading dash and decide it's an
              # option and not an argument.
              opts += '--sim_opts \' {}\' '.format(sim_opts)
          if sim_fastq:
              opts += '--fastq {} '.format(sim_fastq)
          opts += '--annotate_xg {} '.format(base_xg_path)
          for source_path_name in source_path_names:
              opts += '--path {} '.format(source_path_name)
          cmd = 'toil-vg sim {} {} {} {} --gam {} --fastq_out'.format(
              job_store, ' '.join(sim_xg_paths), int(reads / 2), out_store, opts)
          subprocess.check_call(cmd, shell=True)
  
          # then run mapeval
          
          # What do we want to override from the default toil-vg config?
          overrides = argparse.Namespace(
              # toil-vg options
              vg_docker = self.vg_docker,
              container = self.container,
              alignment_cores = self.cores,
              # Toil options
              realTimeLogging = True,
              realTimeStderr = True,
              logLevel = "INFO",
              maxCores = self.cores,
              # toil-vg map options
              # don't waste time sharding reads since we only run on one node
              single_reads_chunk = True,
              mpmap_opts = ['-B', '-F', 'GAM', '-n', 'DNA'],
              more_mpmap_opts = more_mpmap_opts,
              force_outstore = self.force_outstore
          )
          
          # Make the context
          context = Context(out_store, overrides)
  
          # And what options to configure the mapeval run do we want? These have
          # to get turned into a plan in order to import all the files with names
          # derived algorithmically from the names given here. TODO: move
          # positional/required arguments out of this somehow? So we can just use
          # this to get the default optional settings and fill in the required
          # things as file IDs?
          mapeval_options = get_default_mapeval_options()
          mapeval_options.truth = make_url(os.path.join(out_store, 'true.pos'))
          mapeval_options.bwa = True
          mapeval_options.paired_only = paired_only        
          mapeval_options.fasta = make_url(fasta_path)
          mapeval_options.index_bases = [make_url(x) for x in test_index_bases]
          mapeval_options.gam_names = test_names
          mapeval_options.fastq = [make_url(os.path.join(out_store, 'sim.fq.gz'))]
          # We have 150 bp reads reduced to a point position, at a resolution of
          # only the nearest 100 bp (on the primary graph). How close do two such
          # point positions need to be to say the read is in the right place?
          mapeval_options.mapeval_threshold = 200
          if score_baseline_name is not None:
              mapeval_options.compare_gam_scores = score_baseline_name
  
          if 'mpmap' in mappers and more_mpmap_opts:
              # If we're doing more than one mpmap test, disable vg map
              # Just checking here to make sure old logic preserved across interface change
              assert mappers == ['mpmap']
          mapeval_options.mappers = mappers
          
          mapeval_options.ignore_quals = 'mpmap' in mappers and not sim_fastq
          
          # Make Toil
          with context.get_toil(job_store) as toil:
              
              # Make a plan by importing those files specified in the mapeval
              # options
              plan = make_mapeval_plan(toil, mapeval_options)
              
              # Make a job to run the mapeval workflow, using all these various imported files.
              main_job = Job.wrapJobFn(run_mapeval,
                                       context, 
                                       mapeval_options, 
                                       plan.xg_file_ids,
                                       [],
                                       plan.gcsa_file_ids,
                                       plan.gbwt_file_ids,
                                       [],
                                       [],
                                       [],
                                       plan.id_range_file_ids,
                                       plan.snarl_file_ids,
                                       plan.vg_file_ids, 
                                       plan.gam_file_ids, 
                                       plan.reads_gam_file_id,
                                       None,
                                       None,
                                       plan.reads_fastq_file_ids,
                                       plan.fasta_file_id, 
                                       plan.bwa_index_ids,
                                       None,
                                       plan.bam_file_ids,
                                       plan.pe_bam_file_ids, 
                                       plan.true_read_stats_file_id)
                  
              # Output files all live in the out_store, but if we wanted to we could export them also/instead.
              
              # Run the root job
              returned = toil.start(main_job)
              
              # TODO: I want to do the evaluation here, working with file IDs, but
              # since we put the results in the out store maybe it really does
              # make sense to just go through the files in the out store.
code_tokens: def _mapeval_vg_run ( self , reads , base_xg_path , sim_xg_paths , source_path_names , fasta_path , test_index_bases , test_names , score_baseline_name , mappers , paired_only , sim_opts , sim_fastq , more_mpmap_opts , tag ) : job_store = self . _jobstore ( tag ) out_store = self . _outstore ( tag ) opts = ' ' . join ( self . _toil_vg_io_opts ( ) ) + ' ' if self . vg_docker : opts += '--vg_docker {} ' . format ( self . vg_docker ) if self . container : opts += '--container {} ' . format ( self . container ) sim_chunks = int ( reads / self . sim_chunk_size ) if reads % self . sim_chunk_size : sim_chunks += 1 opts += '--maxCores {} --sim_chunks {} --seed {} ' . format ( self . cores , sim_chunks , 8 ) if sim_opts : opts += '--sim_opts \' {}\' ' . format ( sim_opts ) if sim_fastq : opts += '--fastq {} ' . format ( sim_fastq ) opts += '--annotate_xg {} ' . format ( base_xg_path ) for source_path_name in source_path_names : opts += '--path {} ' . format ( source_path_name ) cmd = 'toil-vg sim {} {} {} {} --gam {} --fastq_out' . format ( job_store , ' ' . join ( sim_xg_paths ) , int ( reads / 2 ) , out_store , opts ) subprocess . check_call ( cmd , shell = True ) overrides = argparse . Namespace ( vg_docker = self . vg_docker , container = self . container , alignment_cores = self . cores , realTimeLogging = True , realTimeStderr = True , logLevel = "INFO" , maxCores = self . cores , single_reads_chunk = True , mpmap_opts = [ '-B' , '-F' , 'GAM' , '-n' , 'DNA' ] , more_mpmap_opts = more_mpmap_opts , force_outstore = self . force_outstore ) context = Context ( out_store , overrides ) mapeval_options = get_default_mapeval_options ( ) mapeval_options . truth = make_url ( os . path . join ( out_store , 'true.pos' ) ) mapeval_options . bwa = True mapeval_options . paired_only = paired_only mapeval_options . fasta = make_url ( fasta_path ) mapeval_options . index_bases = [ make_url ( x ) for x in test_index_bases ] mapeval_options . gam_names = test_names mapeval_options . fastq = [ make_url ( os . path . join ( out_store , 'sim.fq.gz' ) ) ] mapeval_options . mapeval_threshold = 200 if score_baseline_name is not None : mapeval_options . compare_gam_scores = score_baseline_name if 'mpmap' in mappers and more_mpmap_opts : assert mappers == [ 'mpmap' ] mapeval_options . mappers = mappers mapeval_options . ignore_quals = 'mpmap' in mappers and not sim_fastq with context . get_toil ( job_store ) as toil : plan = make_mapeval_plan ( toil , mapeval_options ) main_job = Job . wrapJobFn ( run_mapeval , context , mapeval_options , plan . xg_file_ids , [ ] , plan . gcsa_file_ids , plan . gbwt_file_ids , [ ] , [ ] , [ ] , plan . id_range_file_ids , plan . snarl_file_ids , plan . vg_file_ids , plan . gam_file_ids , plan . reads_gam_file_id , None , None , plan . reads_fastq_file_ids , plan . fasta_file_id , plan . bwa_index_ids , None , plan . bam_file_ids , plan . pe_bam_file_ids , plan . true_read_stats_file_id ) returned = toil . start ( main_job )
short_docstring: Wrap toil-vg mapeval.
short_docstring_tokens: Wrap toil - vg mapeval .
comment: """ Wrap toil-vg mapeval. 
          
          Evaluates realignments (to the linear reference and to a set of graphs)
          of reads simulated from a single "base" graph. Realignments are
          evaluated based on how close the realignments are to the original
          simulated source position. Simulations are done inside this function.
  
          sim_xg_paths are xg filenames used for simulation. base_xg_path is an xg
          filename used for everything else like annotation and mapping.
          sim_xg_paths can be [base_xg_path]
          
          Simulates the given number of reads (reads), from the given XG files
          (sim_xg_paths), optionally restricted to a set of named embedded paths
          (source_path_namess). Uses the given FASTA (fasta_path) as a BWA
          reference for comparing vg and BWA alignments within mapeval.
          (Basically, BWA against the linear reference functions as a negative
          control "graph" to compare against the real test graphs.)
          
          test_index_bases specifies a list of basenames (without extension) for a
          .xg, .gcsa, and .gcsa.lcp file set, one per of graph that is to be
          compared.
          
          test_names has one entry per graph to be compared, and specifies where
          the realigned read GAM files should be saved.
          
          score_baseline_name, if not None, is a name from test_names to be used
          as a score baseline for comparing all the realignment scores against.
          
          tag is a unique slug for this test/run, which determines the Toil job
          store name to use, and the location where the output files should be
          saved.
          
          """$SEP$# start by simulating some reads$SEP$# TODO: why are we using strings here when we could use much safer lists???$SEP$# note, using the same seed only means something if using same$SEP$# number of chunks.  we make that explicit here$SEP$# Make sure to prefix the sim options string with a space so that$SEP$# Toil's parser doesn't see a leading dash and decide it's an$SEP$# option and not an argument.$SEP$# then run mapeval$SEP$# What do we want to override from the default toil-vg config?$SEP$# toil-vg options$SEP$# Toil options$SEP$# toil-vg map options$SEP$# don't waste time sharding reads since we only run on one node$SEP$# Make the context$SEP$# And what options to configure the mapeval run do we want? These have$SEP$# to get turned into a plan in order to import all the files with names$SEP$# derived algorithmically from the names given here. TODO: move$SEP$# positional/required arguments out of this somehow? So we can just use$SEP$# this to get the default optional settings and fill in the required$SEP$# things as file IDs?$SEP$# We have 150 bp reads reduced to a point position, at a resolution of$SEP$# only the nearest 100 bp (on the primary graph). How close do two such$SEP$# point positions need to be to say the read is in the right place?$SEP$# If we're doing more than one mpmap test, disable vg map$SEP$# Just checking here to make sure old logic preserved across interface change$SEP$# Make Toil$SEP$# Make a plan by importing those files specified in the mapeval$SEP$# options$SEP$# Make a job to run the mapeval workflow, using all these various imported files.$SEP$# Output files all live in the out_store, but if we wanted to we could export them also/instead.$SEP$# Run the root job$SEP$# TODO: I want to do the evaluation here, working with file IDs, but$SEP$# since we put the results in the out store maybe it really does$SEP$# make sense to just go through the files in the out store.