original_docstring: 
          Check the simulated mapping evaluation results.
          
          read_source_graph is the name of the graph that the reads were generated
          from; we'll compare the scores realigned to that graph against the
          scores that the generated reads had.
          
          score_baseline_name is the name of the graph we compared scores against;
          we will chack that reads increase in score in the other graphs against
          that graph and complain if they don't. It may be None, in which case
          scores are only compared against the scores the reads got when
          simulated.
          
          
docstring: Check the simulated mapping evaluation results.
  read_source_graph is the name of the graph that the reads were generated
  from; we'll compare the scores realigned to that graph against the
  scores that the generated reads had.
  
  score_baseline_name is the name of the graph we compared scores against;
  we will chack that reads increase in score in the other graphs against
  that graph and complain if they don't. It may be None, in which case
  scores are only compared against the scores the reads got when
  simulated.
docstring_tokens: Check the simulated mapping evaluation results . read_source_graph is the name of the graph that the reads were generated from ; we ' ll compare the scores realigned to that graph against the scores that the generated reads had . score_baseline_name is the name of the graph we compared scores against ; we will chack that reads increase in score in the other graphs against that graph and complain if they don ' t . It may be None in which case scores are only compared against the scores the reads got when simulated .
code: def _verify_mapeval(self, reads, read_source_graph, score_baseline_name,
                          positive_control, negative_control, tag, acc_threshold,
                          auc_threshold):
          """
          Check the simulated mapping evaluation results.
          
          read_source_graph is the name of the graph that the reads were generated
          from; we'll compare the scores realigned to that graph against the
          scores that the generated reads had.
          
          score_baseline_name is the name of the graph we compared scores against;
          we will chack that reads increase in score in the other graphs against
          that graph and complain if they don't. It may be None, in which case
          scores are only compared against the scores the reads got when
          simulated.
          
          """
  
          # Make some plots in the outstore
          self._mapeval_r_plots(tag, positive_control, negative_control)
  
          stats_path = os.path.join(self._outstore(tag), 'stats.tsv')
          with io.open(stats_path, 'r', encoding='utf8') as stats:
              stats_tsv = stats.read()
          baseline_tsv = self._read_baseline_file(tag, 'stats.tsv')
  
          # Dict from aligner to a list of float stat values, in order
          stats_dict = self._tsv_to_dict(stats_tsv)
          # Dict from aligner to a list of float stat values, in order
          baseline_dict = self._tsv_to_dict(baseline_tsv)
  
          # print out a table of mapeval results
          table_name = 'map eval results'
          if positive_control:
              table_name += ' (*: positive control)'
          if negative_control:
              table_name += ' (**: negative control)'
          self._begin_message(table_name, is_tsv=True)
          
          # How many different columns do we want to see in the stats files?
          # We need to pad shorter rows with 0s
          stats_columns = 5 # read count, accuracy, AUC, QQ-plot r value, max F1
          
          print('\t'.join(['Method', 'Acc.', 'Baseline Acc.', 'AUC', 'Baseline AUC', 'Max F1', 'Baseline F1']))
          for key in sorted(set(list(baseline_dict.keys()) + list(stats_dict.keys()))):
              # What values do we have for the graph this run?
              sval = list(stats_dict.get(key, []))
              while len(sval) < stats_columns:
                  sval.append('DNE')
              # And what baseline values do we have stored?
              bval = list(baseline_dict.get(key, []))
              while len(bval) < stats_columns:
                  bval.append('DNE')
              
              method = key            
              if not key.endswith('-pe'):
                  # to be consistent with plots
                  method += '-se'
              if positive_control and key in [positive_control, positive_control + '-pe']:
                  method += '*'
              if negative_control and key in [negative_control, negative_control + '-pe']:
                  method += '**'
              def r4(s):
                  return round(s, 5) if isinstance(s, float) else s 
                  
              row = [method]
              for metric_index in [1, 2, 4]:
                  # For each metric, compare stat to baseline
                  stat_val = str(r4(sval[metric_index]))
                  baseline_val = str(r4(bval[metric_index]))
                  if stat_val != 'DNE' and baseline_val != 'DNE':
                      if sval[metric_index] < bval[metric_index]:
                          # Stat got worse
                          stat_val = '↓ {}'.format(stat_val)
                      elif sval[metric_index] > bval[metric_index]:
                          # Stat got better
                          stat_val = '↑ {}'.format(stat_val)
                  row.append(stat_val)
                  row.append(baseline_val)
  
              print('\t'.join(row))
          self._end_message()
  
          # test the mapeval results, only looking at baseline keys
          for key, val in list(baseline_dict.items()):
              if key in stats_dict:
                  # For each graph we have a baseline and stats for, compare the
                  # columns we actually have in both.
                  if len(stats_dict[key]) > 0:
                      self.assertEqual(stats_dict[key][0], reads)
                  if len(stats_dict[key]) > 1 and len(val) > 1:
                      # Compare accuracy stats
                      self.assertGreaterEqual(stats_dict[key][1], val[1] - acc_threshold)
                  if len(stats_dict[key]) > 2 and len(val) > 2:
                      # Compare AUC stats. Make sure to patch up 0 AUCs from perfect classification.
                      new_auc = stats_dict[key][2] if stats_dict[key][2] != 0 else 1
                      old_auc = val[2] if val[2] != 0 else 1
                      self.assertGreaterEqual(new_auc, old_auc - auc_threshold)
                  if len(stats_dict[key]) > 4 and len(val) > 4:
                      self.assertGreaterEqual(stats_dict[key][4], val[4] - acc_threshold)
                  if len(stats_dict[key]) != len(val):
                      log.warning('Key {} has {} baseline entries and {} stats'.format(key, len(val), len(stats_dict[key])))
              else:
                  log.warning('Key {} from baseline not found in stats'.format(key))
              
          # This holds the condition names we want a better score than
          score_baselines = ['input']
          if score_baseline_name is not None:
              score_baselines.append(score_baseline_name)
              
          for compare_against in score_baselines:
              # For each graph/condition we want to compare scores against
          
              # Now look at the stats for comparing scores on all graphs vs. scores on this particular graph.
              score_stats_path = os.path.join(self._outstore(tag), 'score.stats.{}.tsv'.format(compare_against))
              if os.path.exists(score_stats_path):
                  # If the score comparison was run, make sure not too many reads
                  # get worse moving from simulated to realigned scores, or moving
                  # from the baseline graph to the other (more inclusive) graphs.
                  
                  try:
                      # Parse out the baseline stat values (not for the baseline
                      # graph; we shouldn't have called these both "baseline")
                      baseline_tsv = self._read_baseline_file(tag, 'score.stats.{}.tsv'.format(compare_against))
                      baseline_dict = self._tsv_to_dict(baseline_tsv)
                  except:
                      # Maybe there's no baseline file saved yet
                      # Synthesize one of the right shape
                      baseline_dict = collections.defaultdict(lambda: [0, 0])
                      
                  # Parse out the real stat values
                  score_stats_dict = self._tsv_to_dict(io.open(score_stats_path, 'r', encoding='utf8').read())
                      
                  for key in list(score_stats_dict.keys()):
                      # For every kind of graph
                      
                      if compare_against == 'input' and (key != read_source_graph and
                          key != read_source_graph + '-pe'):
                          # Only compare simulated read scores to the scores the
                          # reads get when aligned against the graph they were
                          # simulated from.
                          continue
                      
                      # Guess where the file for individual read score differences for this graph is
                      # TODO: get this file's name/ID from the actual Toil code
                      read_comparison_path = os.path.join(self._outstore(tag), '{}.compare.{}.scores'.format(key, compare_against))
                      for line in io.open(read_comparison_path, 'r', encoding='utf8'):
                          if line.strip() == '':
                              continue
                          # Break each line of the CSV
                          parts = line.split(', ')
                          # Fields are read name, score difference, aligned score, baseline score
                          read_name = parts[0]
                          score_diff = int(parts[1])
                          
                          if score_diff < 0:
                              # Complain about anyone who goes below 0.
                              log.warning('Read {} has a negative score increase of {} on graph {} vs. {}'.format(
                                  read_name, score_diff, key, compare_against))
                  
                      if key not in baseline_dict:
                          # We might get new graphs that aren't in the baseline file.
                          log.warning('Key {} missing from score baseline dict for {}. Inserting...'.format(key, compare_against))
                          # Store 0 for the read count, and 1 for the portion that got worse.
                          # We need a conservative default baseline so new tests will pass.
                          baseline_dict[key] = [0, 1]
                      
                      # Report on its stats after dumping reads, so that if there are
                      # too many bad reads and the stats are terrible we still can see
                      # the reads.
                      print('{} vs. {} Worse: {} Baseline: {}  Threshold: {}'.format(
                          key, compare_against, score_stats_dict[key][1], baseline_dict[key][1], self.worse_threshold))
                      # Make sure all the reads came through
                      self.assertEqual(score_stats_dict[key][0], reads)
                      
                      if not key.endswith('-pe'):
                          # Skip paired-end cases because their pair partners can
                          # pull them around. Also they are currently subject to
                          # substantial nondeterministic alignment differences
                          # based on the assignment of reads to threads.
                      
                          # Make sure not too many got worse
                          self.assertLessEqual(score_stats_dict[key][1], baseline_dict[key][1] + self.worse_threshold)
code_tokens: def _verify_mapeval ( self , reads , read_source_graph , score_baseline_name , positive_control , negative_control , tag , acc_threshold , auc_threshold ) : self . _mapeval_r_plots ( tag , positive_control , negative_control ) stats_path = os . path . join ( self . _outstore ( tag ) , 'stats.tsv' ) with io . open ( stats_path , 'r' , encoding = 'utf8' ) as stats : stats_tsv = stats . read ( ) baseline_tsv = self . _read_baseline_file ( tag , 'stats.tsv' ) stats_dict = self . _tsv_to_dict ( stats_tsv ) baseline_dict = self . _tsv_to_dict ( baseline_tsv ) table_name = 'map eval results' if positive_control : table_name += ' (*: positive control)' if negative_control : table_name += ' (**: negative control)' self . _begin_message ( table_name , is_tsv = True ) stats_columns = 5 print ( '\t' . join ( [ 'Method' , 'Acc.' , 'Baseline Acc.' , 'AUC' , 'Baseline AUC' , 'Max F1' , 'Baseline F1' ] ) ) for key in sorted ( set ( list ( baseline_dict . keys ( ) ) + list ( stats_dict . keys ( ) ) ) ) : sval = list ( stats_dict . get ( key , [ ] ) ) while len ( sval ) < stats_columns : sval . append ( 'DNE' ) bval = list ( baseline_dict . get ( key , [ ] ) ) while len ( bval ) < stats_columns : bval . append ( 'DNE' ) method = key if not key . endswith ( '-pe' ) : method += '-se' if positive_control and key in [ positive_control , positive_control + '-pe' ] : method += '*' if negative_control and key in [ negative_control , negative_control + '-pe' ] : method += '**' def r4 ( s ) : return round ( s , 5 ) if isinstance ( s , float ) else s row = [ method ] for metric_index in [ 1 , 2 , 4 ] : stat_val = str ( r4 ( sval [ metric_index ] ) ) baseline_val = str ( r4 ( bval [ metric_index ] ) ) if stat_val != 'DNE' and baseline_val != 'DNE' : if sval [ metric_index ] < bval [ metric_index ] : stat_val = '↓ {}'.f o rmat(s t at_val)  elif sval [ metric_index ] > bval [ metric_index ] : stat_val = '↑ {}'.f o rmat(s t at_val)  row . append ( stat_val ) row . append ( baseline_val ) print ( '\t' . join ( row ) ) self . _end_message ( ) for key , val in list ( baseline_dict . items ( ) ) : if key in stats_dict : if len ( stats_dict [ key ] ) > 0 : self . assertEqual ( stats_dict [ key ] [ 0 ] , reads ) if len ( stats_dict [ key ] ) > 1 and len ( val ) > 1 : self . assertGreaterEqual ( stats_dict [ key ] [ 1 ] , val [ 1 ] - acc_threshold ) if len ( stats_dict [ key ] ) > 2 and len ( val ) > 2 : new_auc = stats_dict [ key ] [ 2 ] if stats_dict [ key ] [ 2 ] != 0 else 1 old_auc = val [ 2 ] if val [ 2 ] != 0 else 1 self . assertGreaterEqual ( new_auc , old_auc - auc_threshold ) if len ( stats_dict [ key ] ) > 4 and len ( val ) > 4 : self . assertGreaterEqual ( stats_dict [ key ] [ 4 ] , val [ 4 ] - acc_threshold ) if len ( stats_dict [ key ] ) != len ( val ) : log . warning ( 'Key {} has {} baseline entries and {} stats' . format ( key , len ( val ) , len ( stats_dict [ key ] ) ) ) else : log . warning ( 'Key {} from baseline not found in stats' . format ( key ) ) score_baselines = [ 'input' ] if score_baseline_name is not None : score_baselines . append ( score_baseline_name ) for compare_against in score_baselines : score_stats_path = os . path . join ( self . _outstore ( tag ) , 'score.stats.{}.tsv' . format ( compare_against ) ) if os . path . exists ( score_stats_path ) : try : baseline_tsv = self . _read_baseline_file ( tag , 'score.stats.{}.tsv' . format ( compare_against ) ) baseline_dict = self . _tsv_to_dict ( baseline_tsv ) except : baseline_dict = collections . defaultdict ( lambda : [ 0 , 0 ] ) score_stats_dict = self . _tsv_to_dict ( io . open ( score_stats_path , 'r' , encoding = 'utf8' ) . read ( ) ) for key in list ( score_stats_dict . keys ( ) ) : if compare_against == 'input' and ( key != read_source_graph and key != read_source_graph + '-pe' ) : continue read_comparison_path = os . path . join ( self . _outstore ( tag ) , '{}.compare.{}.scores' . format ( key , compare_against ) ) for line in io . open ( read_comparison_path , 'r' , encoding = 'utf8' ) : if line . strip ( ) == '' : continue parts = line . split ( ', ' ) read_name = parts [ 0 ] score_diff = int ( parts [ 1 ] ) if score_diff < 0 : log . warning ( 'Read {} has a negative score increase of {} on graph {} vs. {}' . format ( read_name , score_diff , key , compare_against ) ) if key not in baseline_dict : log . warning ( 'Key {} missing from score baseline dict for {}. Inserting...' . format ( key , compare_against ) ) baseline_dict [ key ] = [ 0 , 1 ] print ( '{} vs. {} Worse: {} Baseline: {}  Threshold: {}' . format ( key , compare_against , score_stats_dict [ key ] [ 1 ] , baseline_dict [ key ] [ 1 ] , self . worse_threshold ) ) self . assertEqual ( score_stats_dict [ key ] [ 0 ] , reads ) if not key . endswith ( '-pe' ) : self . assertLessEqual ( score_stats_dict [ key ] [ 1 ] , baseline_dict [ key ] [ 1 ] + self . worse_threshold )
short_docstring: Check the simulated mapping evaluation results.
short_docstring_tokens: Check the simulated mapping evaluation results .
comment: """
          Check the simulated mapping evaluation results.
          
          read_source_graph is the name of the graph that the reads were generated
          from; we'll compare the scores realigned to that graph against the
          scores that the generated reads had.
          
          score_baseline_name is the name of the graph we compared scores against;
          we will chack that reads increase in score in the other graphs against
          that graph and complain if they don't. It may be None, in which case
          scores are only compared against the scores the reads got when
          simulated.
          
          """$SEP$# Make some plots in the outstore$SEP$# Dict from aligner to a list of float stat values, in order$SEP$# Dict from aligner to a list of float stat values, in order$SEP$# print out a table of mapeval results$SEP$# How many different columns do we want to see in the stats files?$SEP$# We need to pad shorter rows with 0s$SEP$# read count, accuracy, AUC, QQ-plot r value, max F1$SEP$# What values do we have for the graph this run?$SEP$# And what baseline values do we have stored?$SEP$# to be consistent with plots$SEP$# For each metric, compare stat to baseline$SEP$# Stat got worse$SEP$# Stat got better$SEP$# test the mapeval results, only looking at baseline keys$SEP$# For each graph we have a baseline and stats for, compare the$SEP$# columns we actually have in both.$SEP$# Compare accuracy stats$SEP$# Compare AUC stats. Make sure to patch up 0 AUCs from perfect classification.$SEP$# This holds the condition names we want a better score than$SEP$# For each graph/condition we want to compare scores against$SEP$# Now look at the stats for comparing scores on all graphs vs. scores on this particular graph.$SEP$# If the score comparison was run, make sure not too many reads$SEP$# get worse moving from simulated to realigned scores, or moving$SEP$# from the baseline graph to the other (more inclusive) graphs.$SEP$# Parse out the baseline stat values (not for the baseline$SEP$# graph; we shouldn't have called these both "baseline")$SEP$# Maybe there's no baseline file saved yet$SEP$# Synthesize one of the right shape$SEP$# Parse out the real stat values$SEP$# For every kind of graph$SEP$# Only compare simulated read scores to the scores the$SEP$# reads get when aligned against the graph they were$SEP$# simulated from.$SEP$# Guess where the file for individual read score differences for this graph is$SEP$# TODO: get this file's name/ID from the actual Toil code$SEP$# Break each line of the CSV$SEP$# Fields are read name, score difference, aligned score, baseline score$SEP$# Complain about anyone who goes below 0.$SEP$# We might get new graphs that aren't in the baseline file.$SEP$# Store 0 for the read count, and 1 for the portion that got worse.$SEP$# We need a conservative default baseline so new tests will pass.$SEP$# Report on its stats after dumping reads, so that if there are$SEP$# too many bad reads and the stats are terrible we still can see$SEP$# the reads.$SEP$# Make sure all the reads came through$SEP$# Skip paired-end cases because their pair partners can$SEP$# pull them around. Also they are currently subject to$SEP$# substantial nondeterministic alignment differences$SEP$# based on the assignment of reads to threads.$SEP$# Make sure not too many got worse