peptide 
 mass spectrometry provides a powerful method to analyze 
 proteome expression in cell lysates . at the core of this method , experimental 
 mass spectra of fragmented peptides
the accuracy of peptide matching 
 depends critically on the effectiveness of algorithms that match the 
 theoretical and experimental spectra .
we present here a method to evaluate 
 algorithms to obtain high - confidence interrogation of proteomes . in typical experiments , proteins in cell lysates
are digested with 
 an endoprotease , typically trypsin , to obtain peptides in size ranges 
 that can be successfully analyzed by tandem mass spectrometry ( ms / ms ) . 

trypsin fragments with pronounced peaks in the first ms are selected 
 for collision induced dissociation ( cid ) in the second ms .
cid causes 
 fragmentation of the trypsin peptides , typically at amide bonds , producing 
 n - terminal b ions and c - terminal y ions that give rise to a set of 
 detected mass to charge ( m / z ) peaks . 

peptide - searching algorithms compare experimental m / z spectra with theoretical ion ladders derived 
 from tryptic fragments of an input sequence database 
 of all proteins in the proteome .
each spectrum - matching algorithm , 
 however , is different in its design and structure .
the best peptide - spectrum 
 matches are determined by techniques such as cross - correlation ( e.g. , 
 sequest ) or by model - based approaches 
 using statistical significance ( omssa , mascot ) .
each algorithm has multiple 
 parameter settings , including mass 
 tolerances between theoretical and observed trypsin fragments ( precursor 
 mass tolerance ) or theoretical and cid fragments ( fragment mass tolerance ) ; 
 which and how many optional mass modifications to allow per peptide ; 
 and which cid ion series ( e.g. , a , b , y ) to assess .
it is important 
 to choose appropriate parameter settings of algorithms for accurate 
 peptide identifications . given the many combinations of choices , what 
 are good strategies to determine parameter settings ?
do different 
 parameter settings of algorithms applied to a given data set provide 
 different samplings of the proteome , and are these samplings of high 
 quality ?
the effectiveness of an algorithm is typically assessed 
 through 
 decoy analysis . for each forward protein sequence 
 in the sequence  database  , the algorithm is also presented 
 with the reverse of that sequence . the forward and reverse sequences 

are processed blindly together , including a theoretical trypsin digestion , 
 and the frequency that decoy reverse peptides are chosen by the algorithm 
 indicates the false discovery rate ( fdr ) for that particular choice of parameter settings .
given that the 
 algorithms output quality scores for each peptide - spectrum match ( psm ) , 
 scoring thresholds can be computed and used to filter the output data 
 to ensure an fdr below a desired level ( e.g. , 1% or 5% ) .
decoy analysis 
 is an excellent strategy to assess algorithm parameter settings , especially 
 since the approach is inherently independent of any particular algorithm .
we wished to develop additional approaches to assess algorithm 
 performance : methods to be used alongside decoy analysis in order 
 to build confidence in peptide matches by different standardized parameter 
 settings of algorithms .
expanding on a protocol suggested by park 
 et al . , we present a strategy that evaluates 
 peptide - spectrum matches by assessing the masses of parent proteins 
 prior to trypsin digestion , an approach that can be applied to any 
 algorithm using the spectra sets presented here ( supporting information ) or new data sets if desired .
this 
 parent - protein profiling approach uses a gel slice strategy to partition 
 cell lysates according to parent - protein mass .
application of this 
 approach suggests that different algorithms can provide different , 
 yet valid , samplings of the proteome , and that it can also be extremely 
 productive to run algorithms multiple times with different parameter 
 settings , an approach that is becoming increasingly possible given 
 the availability of increased computational power .
we first focus 
 our discussion on the sequest and omssa algorithms and then present 
 equivalent analysis of the mascot algorithm , which revealed similar 
 results .
conformance 
 to parent proteins before digestion with trypsin was used to assess 
 algorithm matches of spectra to tryptic peptides .
yeast cell lysates 
 were partitioned into gel slices of known molecular weight size ranges 
 ( 2537 , 3750 , and 5075 kda ) .
although the algorithms 
 had no knowledge of the parent - protein sizes before trypsin digestion , 
 peptide matches would be expected to conform to the correct parent - protein 
 size ranges if the algorithm was matching successfully .
this has been 
 shown previously in ms / ms - based gel - band analysis of the proteome 
 of pseudomonas putida bacteria .
protein samples were prepared as follows : 100 ml 
 of ysh474 cells were grown to mid - log phase in ypd and lysed with 
 ripa buffer ( 150 mm nacl , 1% igepal , 0.1% sds , 50 mm tris ph 8.0 ) , 
 and acid - washed glass beads . to prevent degradation , protease inhibitors 
 ( roche ) and pmsf were added , and samples were chilled on ice during 
 lysis .
the lysate was spun at 5,000 rpm , and samples of 500 or 1,000 
 g were run alongside protein standard markers ( bio - rad ) on 
 420% sds - page gels ( bio - rad ) .
protein standard bands served 
 as a guide for the excision of gel slices of various molecular weight 
 size ranges ( 2537 , 3750 , and 5075 kda ; figure 1 ) .
samples were subjected to reduction and alkylation 
 followed by overnight in - gel trypsin digestion .
extracted peptides were resuspended in 0.1% tfa , loaded 
 onto a c18 packed ( michrom ) nanospray column ( polymicro ) , and run 
 with a 180-min gradient on a lcq deca xp ( thermo - scientific ) coupled 
 to a high - performance liquid chromatography ( hplc ) system ( agilent 
 1100 series ) and a nanoelectrospray ( nano - esi ) ion source .
preliminary 
 tests indicated that gels with visible degradation had limited conformance 
 to parent - protein masses .
hence , gels were discarded and not analyzed 
 if their appearance suggested visible degradation .
peptide matches were identified using the sequest algorithm ( proteome 
 discoverer v.1.2 ) run on a dell alienware aurora r4 server , the open 
 mass spectrometry search algorithm ( omssa ) run on a 90-node cluster , 
 and the mascot algorithm run on a dell xps 8300 server .
algorithm 
 parameters were set up to search for either the standard b and y ions 
 following cid or with the addition of a ions .
optional mass increases 
 to peptides included dynamic modifications of + 42 da for acetylation 
 of any n - terminal amino acid residue and + 16 da for oxidation of methionine 
 residues , and static modifications included + 57 da for carbamidomethyl 
 modification of cysteine residues .
the sequest algorithm parameter 
 file included a precursor mass tolerance of 3.0 da and a fragment 
 mass tolerance of 1.0 da , while the omssa algorithm was run using 
 five different standard sets of parameters ( table 1 ) and mascot was run using four parameter sets ( figure 6a ) ; the sequest parameter set is similar to that 
 reported for peptideatlas yeast data ( http://www.peptideatlas.org/ ) ; the omssa and mascot parameter sets are similar to the algorithm 
 default parameters .
precursor peptides for liquid chromatography ms / ms 
 ( lc  ms / ms ) analysis were prepared by trypsin digestion , which 
 cuts after arginine or lysine , except when flanked by proline .
for 
 the sequest , omssa , and mascot analysis , we required trypsin - cleavage 
 sites at both ends of the precursor peptides ( or one end if a terminal 
 peptide ) .
a sequence  database  file containing protein 
 translations of annotated and downstream open reading frames ( dnorfs ) 
 in fasta format was constructed as described previously .
output data from the 
 three algorithms were uploaded to a relational 
 database and analyzed with stored procedures written in ms - sql to 
 compute decoy false discovery rates ( fdrs ) and parent - protein conformance 
 scores .
reverse - sequence decoy analysis was performed as described 
 previously , and peptide matches were filtered 
 to give a target fdr of 5% . before computing decoy score thresholds , 
 for each lc  ms / ms run , we excluded matches with internal trypsin 
 sites and matches with an initial ranking ( rank ) > 1 if a sequest 
 or mascot matched peptide .
the decoy score thresholds were then applied 
 to the output data after first excluding omssa matches ( which are 
 not ranked ) where multiple nondecoy peptides matched to the same spectrum . 
 as discussed in section 3.2 below ,
the peptide 
 matching by omssa is stringent , and the false detection rate was typically 
 below 5% after application of these filters ( 1.65.1% depending 
 on parameter settings and cid ions assessed ) .
for all three algorithms , 
 we also excluded peptides that mapped to multiple parent proteins ; 
 although these were likely correct identifications , these peptides 
 were excluded because they could not be assigned to unique parent 
 proteins .
parent - protein 
 conformance scores were computed from forward peptide matches classified 
 as conforming or nonconforming peptides according to the known molecular 
 weight size range of the gel slice ( 2537 , 3750 , and 
 5075 kda ) .
ms / ms run , 
 even if detected by multiple spectra . to account for aberrant protein 
 travel through the gel or possible post - translational modifications , 
 mass tolerances of 10% of the molecular weight size range
for example , peptide matches from the 2537 kda gel 
 slice range were categorized as conforming if the parent proteins 
 were between 22.5 and 40.7 kda .
conformance scores for individual 
 lc  ms / ms runs were computed as follows : an overall 
 conformance score ( for a single set of parameter settings 
 for the algorithm ) was computed by summing the number of conforming 
 peptides and nonconforming peptides across all 22 lc 
ms / ms 
 runs counting each matched peptide a maximum of once per run :
peptide - spectrum matching algorithms score individual 
 peptide matches according to how well the masses of expected cid fragments 
 of a tryptic peptide match the m / z peaks in a detected spectrum .
the approach is based on the 
 fact that algorithms have no knowledge of the masses of parent proteins 
 prior to trypsin digestion . by partitioning parent proteins into known 
 molecular weight size
ranges ( 2537 , 3750 , and 5075 
 kda ) using sds - page and processing individual gel slices for assessment 
 through lc 
ms / ms ( figure 1 ) , we investigated 
 whether peptide matches by algorithms conformed to the expected parent - protein 
 size range . allowing a mass tolerance of 10% to compensate for experimental 
 variations inherent in the gel slice approach
, we computed conformance 
 scores indicative of the effectiveness of an algorithm parameter set 
 ( table 2a . ) .
fdr threshold 
 values were ( a ) sequest : 0.081 ; omssa parameter sets 0 to 4 : 1.0 . 

( b ) sequest : 0.164025 ; omssa parameter sets 0 , 1 , 3 , 4 : 1.0 ; omssa 
 parameter set 2 : 0.9 .
a total of 2,842 distinct peptides were detected 
 when counted only once regardless of which algorithm , parameter set , 
 ion screen , or gel - slice range . of these , 10.5% ( 298 distinct peptides ) 
 were detected in more than one gel - slice range , and 5.9% ( 168 distinct 
 peptides ) were scored as both conforming and nonconforming depending 
 on size range .
we ran individual 
 lc  ms / ms runs for each of the 22 gel slices 
 and assessed spectra from the runs using a standard parameter set 
 for the sequest algorithm and five different parameter sets for the 
 omssa algorithm ( table 1 ) . after employing 
 decoy analysis to ensure false discovery rates ( fdrs ) below 5%
, we used the convention that even if a peptide were detected with 
 multiple spectra , each peptide was counted only once per lc 
( the same peptide was counted more than once if detected in multiple 
 runs , which in some cases were from gel slices with different size 
 ranges . ) in a standard b / y ion screen , 84.4% of the 4,480 peptides 
 detected by sequest had parent proteins conforming to the expected 
 size range . using the same spectra , omssa detected between 2,134 and 
 3,644 peptides with conformance scores ranging from 87.6% to 88.8% 
 depending on the particular parameter set ( table 2a ) .
the conformance scores provide a relative measure 
 of the peptide 
 matching accuracy of each set of parameter settings of an algorithm . 

for example , the standardized omssa parameter sets have somewhat higher 
 conformance scores than sequest .
however , although correlated , the 
 conformance score is not numerically equal to the matching efficiency 
 of the algorithm due to several factors :  parent proteins 
 may run aberrantly during gel electrophoresis .
 post - translational modifications of parent proteins , 
 such 
 as glycosylation or proteolytic cleavage , may substantially change their molecular weights .  parent proteins of peptides randomly matched by the algorithm 
 may be randomly assigned to the correct size range ; for example , 25% 
 of annotated yeast proteins have masses between 22.5 and 40.7 kda , 
 so 25% of random matches would conform to this size range .
indeed , 
 the overall conformance scores for the decoy reverse peptides are 
 20.6% ( table 2a ) .
because these contributing 
 factors likely apply equivalently across 
 all parameter settings of algorithms analyzing the same spectrum data 
 sets , differences in conformance scores nevertheless provide an excellent 
 assessment of the relative accuracies of the different algorithms 
 and can be used to assess new algorithms or new parameter settings 
 of current algorithms .
the 
 five parameter sets of omssa all showed higher parent - protein conformance 
 scores compared with sequest .
indeed , the conformance score distributions 
 from bootstrap analysis indicated that these differences are significant 
 ( supplementary figure 2 ) . given that the 
 data sets were filtered to give a 5% fdr , the difference in conformance 
 scores indicates that the detection of reverse - sequence decoys by 
 the two algorithms was not equivalent .
indeed , even before applying 
 a 5% fdr scoring filter , the standard implementation of omssa returned 
 decoys at rates of 1.63.9% depending on the parameter set , 
 indicating that the omssa algorithm is quite stringent in its interpretation 
 of acceptable psms .
moreover , unlike sequest , omssa does not standardly 
 output a ranking of psms ; if only the best - scoring psms for each spectrum 
 are considered , then the decoy rates are even lower for the omssa 
 algorithm ( 0.61.7% ) . for comparison
, we reassessed the sequest 
 output using a 0.6% fdr threshold instead of 5% .
this resulted in 
 fewer matches ( 3,398 instead of 4,480 ) and a significantly higher 
 parent - protein conformance rate ( 88.2% instead of 84.4% ) comparable 
 to those of omssa ( supplementary figure 2 ) .
however , for the analysis that follows we used standard implementations 
 of both algorithms with 5% fdr thresholds and filters as described 
 in methods . counting peptides once even if seen with 
 multiple omssa parameter sets , we classified peptides according to 
 the number of standardized parameter sets ( i.e. , 15 ) for which 
 a peptide was detected .
although conformance scores for the individual 
 parameter sets indicated high confidence in peptide matches , only 
 42.9% of the 3,922 peptide matches were detected by all five omssa 
 standardized parameter sets ( figure 2a . ) .
the 
 different samplings of peptides from different parameter settings 
 suggest that using multiple standardized settings of omssa can increase 
 the yield of high - confidence detected peptides .
indeed , when probing 
 the same spectrum data set , the union of the outputs from five omssa 
 parameter sets gave 3,922 detected peptides at an overall conformance 
 score of 86.2% .
furthermore , we found that peptides detected by only 
 one of the five standardized parameter sets of omssa had a considerably 
 lower conformance score ( 61.1% ) and accounted for only 5.0% of the 
 detected peptides ( figure 2a ) .
this suggests 
 that when using multiple settings of omssa it may be appropriate to 
 exclude any peptides detected by only one of the standardized parameter 
 sets given that this class of orphan peptides is found to be of lower 
 confidence based on the parent - protein profiling approach .
union of outputs 
 from multiple omssa parameter sets increases the 
 yield of detected peptides .
conformance scores calculated from 
 distinct peptides detected in the b / y ( a ) or a / b / y ( b ) ion screens 
 in omssa ; peptides were counted once even if seen with multiple standardized 
 parameter sets .
collision induced dissociation 
 ( cid ) most commonly cleaves peptides 
 at the amide bonds ( between the c and n atoms ) to give b and y ions . 

these are the two ion types typically assessed by the matching algorithms 
 ( b / y ion screen ) .
however , a ions can also be produced if the cleavage 
 position is shifted by one carbon , and algorithms can be configured 
 to screen for a ions in addition to the b and y ions ( a / b / y ion screen ) . 
 since assessment of a ions
is sometimes included in specialized screens 
 ( e.g. , of glutaraldehyde modified peptides ) , we tested whether inclusion of the a ions might increase peptide 
 detections .
we performed an a / b / y ion screen on the same spectra data 
 sets from the parent - protein profiling experiments above and examined 
 the conformance scores ( table 2b ) . using 
 sequest , we detected 5,016 peptides , counting a peptide once per lc 
ms / ms 
 run whether seen in one or both of the b / y and a / b / y ion screens ( figure 3b ) .
this corresponds to 2,261 unique peptides , of 
 which 1,752 ( 77.5% ) peptides were detected by both the b / y and a / b / y 
 ion screens .
bootstrap analysis ( figure 3c , p < 0.001 ) indicated that peptides detected by sequest 
 in both the b / y and a / b / y ion screens had significantly higher conformance 
 scores compared to peptides detected by either the b / y or a / b / y ion 
 screens alone ( figure 3a ) .
this suggests that 
 confidence in sequest peptide matches can be increased by performing 
 both b / y and a / b / y ion screens and retaining only the peptide matches 
 detected by both screens ( the intersection of the outputs ) .
conformance 
 scores for a / b / y and b / y ion screens .
( a ) conformance 
 scores were computed on the basis of distinct peptides , counted once 
 per lc 
ms / ms run , detected by sequest in the a / b / y ion screen 
 alone , the b / y ion screen alone , or both a / b / y and b / y ion screens . 

( b ) sequest detected 5,016 peptides , counting each peptide once per 
 lc 
4,221 peptides were detected 
 by both ion screens , while 536 and 259 peptides were detected by a / b / y 
 and b / y ion screens alone , respectively .
( c ) bootstrap analysis shows 
 significantly depressed conformance scores for b / y - alone and a / b / y - alone 
 sequest matches .
dotted lines and percentages represent the conformance 
 score based on distinct peptides detected by b / y or a / b / y ion screens 
 alone . dashed lines and percentages
left panel : distribution of 1,000 conformance scores calculated after 
 random sampling with replacement of 536 samples from a full set of 
 4,757 distinct peptides per lc 
ms / ms run detected in the a / b / y 
 ion screen .
right panel : distribution of 1,000 conformance scores 
 calculated after random sampling with replacement of 259 samples from 
 a full set of 4,480 distinct peptides per lc  ms / ms run detected 
 in the b / y ion screen .
( e ) percentages 
 of distinct peptides detected in either the a / b / y ion screen alone , 
 the b / y ion screen alone , or both the a / b / y and b / y ion screens .
in contrast , the equivalent analysis 
 with omssa did not give similar 
 results .
higher percentages of the total number of distinct peptides 
 ( counted once per lc  ms / ms run ) were detected by omssa in either 
 the b / y or a / b / y ion screen alone as compared to the sequest counterpart 
 ( figure 3d , e ) .
additionally , the high conformance 
 scores ( figure 3d ) of peptides detected by 
 omssa in either the b / y or a / b / y ion screen alone suggest that in 
 order to increase significantly the numbers of detected peptides in 
 omssa studies , it may be beneficial to perform both b / y and a / b / y 
 ion screens and to take the union of the output results ( rather than 
 the intersection as with sequest ) .
we note , however , that this would 
 approximately double the required computational time .
these 
 results indicate that algorithms can have qualitatively different 
 behaviors , emphasizing the importance of having evaluation tools to 
 assess different standardized parameter settings of the algorithms .
we investigated 
 whether the striking differences in performance between sequest and 
 omssa in the b / y and a / b / y ion screens might be related to parent - protein 
 expression levels of the detected peptides . using data from genomic - scale 
 western analyses in yeast
, we found that 
 parent - protein expression values for peptides detected by sequest 
 in the b / y or a / b / y ion screens alone had significantly lower protein 
 expression distributions compared to the distributions for peptides 
 detected by both b / y and a / b / y ion screens ( figure 4a ; p < 1.09  10 for b / y only , p < 1.12  10 for a / b / y only ) .
however , this was not the case for the corresponding 
 omssa analysis ( figure 4b , supplementary figure 3 ) , potentially accounting for the high 
 confidence in the b / y - only and a / b / y - only matches . parent protein expression 
 of distinct peptide matches ( per lc 
ms / ms 
 run ) detected in the a / b / y ion screen alone , the b / y ion screen alone , 
 or both a / b / y and b / y ion screens .
protein expression values ( pe ; 
 estimated molecules per cell ) were obtained from genomic - scale western 
 analyses in yeast .
proteins of unknown 
 pe values or values of zero ( undetected ) are not included .
distribution 
 lines represent parent - protein pe values of peptides detected by the 
 sequest ( a ) and omssa parameter set 0 ( b ) screens .
the significantly lower parent - protein expression levels , 
 in combination 
 with the lower conformance scores of peptide matches detected by the 
 b / y or a / b / y ion screens alone , suggest that , compared to omssa , the 
 sequest algorithm can detect peptides of lower abundance , but that 
 confidence in these lower - abundance peptides is decreased .
the qualitative 
 difference in the results from sequest and omssa raise the possibility 
 that the two algorithms provide different samplings of the same cell 
 lysate ms / ms data . counting each peptide once regardless of how many 
 ms / ms experiments , which standardized parameter sets of the algorithm , 
 or which cid ion screens revealed the peptide
sequest detected 
 2,261 distinct peptides from the union of both the a / b / y and b / y ion 
 screens , while omssa detected 2,097 distinct peptides from the union 
 of the five standardized parameter settings of both the a / b / y and 
 b / y ion screens .
a total of 1,516 peptides were detected by both algorithms 
 ( figure 5a ) and had significantly higher parent 
 conformance than peptides detected by either algorithm alone ( 89.0% 
 of 3,382 peptides , counting peptides once per lc 
ms / ms run ; 
 bootstrap p < 0.001 ; figure 5b , supplementary figure 4 )
. however , conformance 
 rates are higher ( yet still depressed ) for sequest - alone peptides 
 if detected by both b / y and a / b / y screens ( figure 5c ) ; similarly , omssa - alone peptides have higher ( yet still 
 depressed ) conformance rates if detected by more than one parameter 
 set ( figure 5c ) .
conformance of distinct 
 peptides detected by omssa and sequest 
 alone or by both algorithms .
( a ) counting each peptide once , regardless 
 of which parameter set of the algorithm , how many ms / ms experiments , 
 or which cid ion screens revealed the peptide , we find that 53.34% 
 of the peptides are detected by both algorithms . of the 2,261 distinct 
 peptides detected by sequest and the 2,097 distinct peptides detected 
 by omssa
, 1,516 peptides are found in both sequest and omssa , while 
 745 peptides are unique to sequest and 581 peptides are unique to 
 omssa .
ms / ms run , even if seen in both ion screens or in multiple 
 parameter sets , for peptides detected by both algorithms , sequest 
 alone , or omssa alone .
( c ) conformance scores are computed after applying 
 two filters limiting distinct peptides ( i ) from omssa to those that 
 are detected in > 1 parameter set and ( ii ) from sequest to those 
 that 
 are detected in both b / y and a / b / y ion screens .
the above results suggest that low protein expression 
 is associated with poor conformance to parent proteins .
indeed , subsets 
 of sequest spectrum matches with progressively lower protein expression 
 reveal that parent - protein conformance ranges from 59.4% ( protein 
 expression < 10 molecules per cell ) to 86.8% ( protein 
 expression > 10 molecules per cell ) ( table 3a ) .
interestingly , 
 proteins in the  undetected  set ( protein expression 
 value = 0 ) have high conformance rates , 
 suggesting that they were undetected for experimental reasons ( e.g. , 
 inaccessible epitope tag ) rather than low protein expression . protein expression ( pe ; estimated 
 protein molecules per cell ) based on large scale western analysis .
ms / ms 
 run even if detected by both the b / y and a / b / y ion screens and , in 
 the case of omssa , even if detected by multiple parameter sets . scoring confidence d = log10(psm_score / fdr_threshold ) . for omssa , 

implemented in proteome discoverer 1.2 , we used the probability 
 outputs to compute a psm score = 10 . conformance scores are 
 significantly 
 depressed for lower abundance proteins ( chi - squared : sequest , p < 6.18  10 ; omssa , p < 3.46  10 ) and for lower d scores ( sequest , p < 7.23  10 ; omssa , p < 6.25  10 ) this 
 result indicates that parent - protein conformance is not a 
 uniform property within data sets but instead varies with protein 
 expression levels : the algorithms are less effective at matching spectra 
 for lower abundance proteins as might be expected .
we examined the relationship between protein expression 
 and the algorithm peptide - spectrum match ( psm ) scores ( sequest probability 
 score , omssa e - value score ) and found that higher - confidence matching 
 scores tend to be associated with higher expression proteins , whereas 
 lower - confidence matching scores are common for both high and low 
 expression proteins ( supplementary figure 5 ) . for this analysis we used a distance measure : which 
 measures the relative distance between 
 an algorithm psm score and the 95% confidence threshold for each experimental 
 series . for example a d - score of 2 implies that the 
 spectrum match is 10 fold better than the fdr threshold , 
 which is the least stringent acceptable score for inclusion in the 
 data set based on decoy analysis .
not surprisingly , given this 
 relationship between protein expression 
 and algorithm psm scores , we found that conformance to parent proteins 
 tends to be higher for subsets of matches with higher - confidence algorithm 
 psm scores .
we examined parent - protein conformance for subsets of 
 spectrum matches with progressively better score ranges ( table 3b ) .
this revealed that for both algorithms , matches 
 with scores within 10-fold of the fdr threshold have parent conformance 
 scores of only 6164% , whereas score bins with greater distances 
 from the fdr threshold have conformance scores that increase progressively 
 up to 9193% .
this analysis suggests that a given parent - protein 
 conformance 
 rate for a data set represents an average rate for 
 the set of spectrum matches , and that the conformance rate is much 
 lower for matches close to the fdr threshold and correspondingly higher 
 for those further from the threshold .
similarly , as discussed previously , fdrs measure the average values for a data set 
 and subsets of data with algorithm scores closer to the fdr threshold 
 have elevated rates of false identification and hence are of lower 
 confidence compared to matches with scores further from the fdr threshold .
we note that although decoy analysis can be employed 
 to assess subsets of poor - scoring psms as discussed above , it would 
 not be practical to use decoy analysis to assess algorithm performance 
 with the other special subsets of detected peptides presented above , 
 such as the outputs from different ion screens or different algorithms , 
 due to difficulties in determining appropriate subsets of decoys for 
 computing fdrs . the parent - protein
profiling data set analyzed 
 in this study provides 
 a useful resource for assessing spectrum - peptide matching algorithms . 

ms / ms runs from the parent - protein profiling approach 
 may be used as a benchmark data set for future yeast proteomic studies 
 that are based on cid fragmentation and a mass spectrometer of similar 
 resolution and sensitivity to the lcq deca xp ( supplementary materials ) . for example , the spectra from the 
 22 gel slice experiments were used to evaluate several parameter settings 
 of the mascot algorithm , which is commonly 
 used by many groups ( figure 6a ) .
peptide - spectrum matches identified by mascot ( figure 6b , c ) showed similar parent - protein conformance 
 rates as sequest and omssa and a similar graded dependence of conformance 
 on protein expression ( figure 6f ) and scoring 
 confidence ( figure 6 g ) .
like omssa , peptides 
 detected with only one set of parameter settings of mascot had poorer 
 conformance compared to those detected by multiple parameter sets 
 ( figure 6e ) .
however , the behavior of the b / y 
 and a / b / y ion screens was somewhat different for mascot in that all 
 matches detected by the a / b / y ion screen were also detected by the 
 b / y screen , and the few matches detected by the b / y ion screen alone 
 had poor parent - protein conformance ( figure 6d ) .
( a ) the mascot 
 algorithm was run using similar parameter settings to those used with 
 omssa .
( c ) 
 parent protein conformance scores for a / b / y ion screen .
( d ) few peptides 
 were detected by the b / y ion screen alone , and these had relatively 
 lower conformance .
( e ) peptides detected by only one of the mascot 
 standardized parameter sets have lower conformance compared to those 
 detected with multiple parameter sets .
( f ) conformance scores are 
 depressed for detected proteins with low expression ( p < 1.05  10 ; chi - squared ) ; for example , 
 the set of proteins where protein molecules per cell < 1000 ( i.e. , 
 log(pe ) < 3 ) have a conformance level of 53.9% .
( g ) similarly , 
 conformance scores are depressed for peptide matches that score close 
 to the decoy fdr threshold .
this is the case for the subsets of psms 
 with scores below 77.6% , the 1% score threshold from bootstrap analysis . 
 in this assessment , the scoring confidence , d = log10(psm_prob_score / fdr_threshold ) ,
is computed using psm probabilities 
 from the mascot output : score = 10log10(psm_prob_score ) . with the ongoing improvements 
 in the design of currently available peptide - spectrum matching algorithms , 
 as well as the development of new algorithms , our parent - protein profiling 

approach provides an unbiased and valid evaluation for assessing different 
 algorithms and choosing effective parameter settings to obtain high 
 confidence peptide matches . in particular , conformance rates provide 
 a relative measure for comparing different algorithms and different 
 standardized parameter settings .
ms / ms runs also calls for the use of multiple 
 algorithms and parameter settings to increase the yield of identified 
 peptides . in the case of sequest ,
taking the intersection of the output 
 from the b / y and a / b / y ion screens may increase confidence in peptide 
 matches . in the case of omssa , taking the union of the outputs from 
 multiple parameter sets and excluding psms detected by a single parameter 
 set may increase confidence in peptide matches . in the case of mascot , 
 combining both of these filters ( taking the intersection of b / y and 
 a / b / y matches and excluding psms detected by only one parameter set ) 
 may increase confidence . as expected ,
assessment of psms in the context of known protein expression levels 
 of the detected parent proteins indicates that confidence in spectrum 
 matching by an algorithm varies within a data set and is lower for 
 matches to low abundance proteins and matches with low - confidence 
 algorithm psm scores .
 peptide 
 mass spectrometry relies crucially on algorithms that match 
 peptides to spectra . 
 we describe a method to evaluate the accuracy 
 of these algorithms based on the masses of parent proteins before 
 trypsin endoprotease digestion . 
 measurement of conformance to parent 
 proteins provides a score for comparison of the performances of different 
 algorithms as well as alternative parameter settings for a given algorithm . 
 
 tracking of conformance scores for spectrum matches to proteins with 
 progressively lower expression levels revealed that conformance scores 
 are not uniform within data sets but are significantly lower for less 
 abundant proteins . similarly peptides with lower algorithm peptide - spectrum 
 match scores have lower conformance . 
 although peptide mass spectrometry 
 data is typically filtered through decoy analysis to ensure a low 
 false discovery rate , this analysis confirms that the filtered data 
 should not be considered as having a uniform confidence . 
 the analysis 
 suggests that use of different algorithms and multiple standardized 
 parameter settings of these algorithms can increase significantly 
 the numbers of peptides identified . 
 this data set can be used as a 
 resource for future algorithm assessment . 
Introduction
Methods
Results and Discussion