in the early days of biological research , mutations that caused discernable phenotypes were the primary tool for understanding how a biological system worked  in the absence of a mutation a gene was invisible . today , biologists are armed with a whole arsenal of tools to regulate gene , mrna , and protein abundance and activity , thereby promoting the discovery of mechanisms and how a system gone awry can lead to disease ( 1 ) . among these are tools for suppressing the activity of a gene or gene product ( e.g. site - directed mutagenesis , rna interference , small molecule inhibitors ) or enhancing activity ( e.g. activating mutations or receptor agonist ) .
for instance , interfering with protein activity using small - molecule inhibitors should have a phenotype similar to reducing the abundance of the corresponding mrna with anti - sense oligonucleotides ( 2 ) .
likewise , similar responses are expected whether increases in intracellular protein concentration are achieved via an inducible promoter or by addition of recombinant protein ( 3 ) . as such
, perturbations form the core of understanding how biological systems work , how diseases arise , and how they can be treated .
any serious attempt to analyze a biological process starts by identification and characterization of perturbations that have been used in prior work .
this task requires a framework that can be systematically applied and that is amenable to both manual and automatic means .
currently , there is no established categorization that sufficiently represents the range of described experimental manipulations beyond high - level semantic and grammatical classifications ( 4,5 ) or description of techniques ( 6 ) .
for example , the closest concept we have found is  altered expression, defined as  altered expression level of a gene / protein ( 7 ) .
we believe that this concept is overly specific and fails to cover important phenomena , among others , changes in protein activity or gene mutations .
we propose , instead , taking the existing concept of  perturbation and broadening it to comprise the range of terms used in text to indicate changes in the abundance or activity of dna , rna and proteins .
perturbations , in this new formulation , would refer to a collection of phenomena in a manner analogous to the way protein 
protein interactions refer to biological phenomena of different type ( e.g. bind , activate , inhibit ) .
since this proposition , like any other , needs to be tested for validity and utility , we have applied it to a case study involving gene 
phenotype associations in disease and have developed a mining algorithm that detects the diverse forms in which perturbations appear in text .
therefore , we are introducing in this work both a new way to understand a crucial part of biology and a new text - mining method tailored to its extraction .
design ,  test and  analysis. as initial step , we created the design corpus to develop an analytical framework for annotation .
the purpose of this corpus was to identify challenges in the annotation process and to refine guidelines that would help the annotators in choosing their evaluations .
annotating perturbations requires at times thorough knowledge of experimental biology , which can only be captured and organized within a solid framework .
therefore we sought to perform a preliminary analysis on a test corpus to improve on subsequent annotations .
this corpus was limited to sentences that included disease - related gene  phenotype relationships . using the semantic relationship nomenclature of tsai et al .
( 8) , we selected reports in which the  agent that deliberately performs an action is represented by a gene or protein , and the  patient that is the recipient of the action corresponds to disease phenotypes .
the information we sought stands in contrast to associative relationships , such as elevated protein levels correlating with disease activity . to create the design corpus , our initial query matched medline sentences containing ordered triplets of a gene or protein name , a causative verb and a phenotype related to cancer or diabetes .
each member of the triplet was separated within the sentence by a maximum of three words .
this software package has the ability to retrieve sentences from text that include word patterns established by user queries .
semantically , term classes can be defined combining external ontologie and adding term morphological variations and regular expression patterns .
syntactically , it recognizes part - of - speech and shallow parsing constructs such as noun phrases , verbs and prepositions . in terms of scope
, queries can be confined to different document or text sections , including abstracts , titles or sentences .
for example , a sentence - level query may comprise a term class protein , a list of verbs ( i2e can automatically generate morphological variants ) and a list of phenotype objects .
the gene / protein thesaurus was internally developed and based on biothesaurus ( 9 ) .
forty verbs that signal causality ( e.g. inhibition , stimulation ) were compiled manually , as was a set of disease - related phenotypes relevant to cancer ( e.g. tumorigenesis , vascularization ) and diabetes ( e.g. serum glucose levels , weight gain ) .
a set of 100 retrieved sentences and relationships were annotated by three phd - level evaluators for relevance and direction of perturbation ( see table 1 for examples ) . 
relevance annotation was used to mark relationships that should be eliminated from the corpus for being irrelevant to the intent of the retrieval query , e.g. if the gene was not acting as an agent . 
direction indicated the type of perturbation ( increased , decreased or unknown ) relative to the starting state .
for example , if a gene is added back to cells that carry a deletion in that gene to restore the wild - type state , it is noted as an increase , because the abundance of the gene was increased relative to the starting state .
unknown if there was no stated perturbation , or direction could not be inferred at the sentence level .
it is worth noting that although an unknown direction could be frequently resolved by reviewing the abstract or the full text of the article , we strictly limited our scope to evidence found at the sentence level . 

table 1.examples of relevance and direction annotationrelevancedirectionsentenceexplanationpmidirrelevantdifferent members of bcl-2 family may promote or inhibit apoptosis by synthesizing anti- and proapoptotic proteinsmembers of a gene family , not a gene , are causing apoptosis phenotype17404013irrelevantthe atf6 and ire1 pathways cooperatively caused apoptosis via induction of chop , activation of xbp1 and phosphorylation of jnk , and the perk - eif2 pathway counteracted the proapoptotic processespathways , not genes , are causing apoptosis phenotype17464326irrelevantthis result suggests that toxic products such as reactive oxygen species and aldehydes liberated by the action of polyamine oxidase on the acetylated polyamines formed by ssat may enhance tumor developmentmetabolites , not proteins that generate them , are causing tumor development phenotype17675337relevantunknownthese results indicate that survivin is important for optimal development of bovine blastocysts and confirm that survivin expression suppresses apoptosis of pre - implantation embryossurvivin expression does not indicate if there is a state change in amount17075833relevantincreasefinally , we showed that after 79 days of incubation , mch also inhibits proliferation of non - stimulated pbmcincubation indicates mch was added exogenously17537530relevantincreasetaken together , ldm induces apoptosis in a p53-dependent manner when given at low doses , but in a p53-independent manner when given at high dosesgiven at low doses indicates ldm was added exogenously17534142relevantdecreasesoluble flt-1 abolished hypoxia / vegf - induced hyperpermeabilitysoluble forms of receptors create inactive complexes17311300 examples of relevance and direction annotation while most sentences contained straightforward descriptions of perturbations ( e.g.  mutations in gene a or  protein b inhibition ) , the broad range of perturbations in the literature , combined with the complex grammatical structures found in biomedical text , made some sentence  relationship pairs difficult to annotate consistently , mainly due to differences in knowledge of experimental descriptions and settings by the evaluators .
when inter - annotator agreement proves to be challenging , other groups have adopted strategies to further improve the final gold standard annotation set ( 10 ) . for this work , sentences deemed difficult to evaluate were set aside for later annotation by discussion and group consensus .
therefore , the gold standard was comprised of annotations with three - way agreement from the individual assessment , plus the consensus annotations . when each evaluator 's set of  straightforward independent annotations ( n = 237 out of 300 annotations , 79% ) was compared to the gold standard , agreement averaged 92.9% . sentence
 relationships for which there was no agreement or only two - way agreement were discussed and annotated by consensus ( 14% , n = 14 ) . while overall inter - annotator agreement can not be measured with the annotation procedure devised , the agreement metrics described are helpful as proxies to the level of ambiguity of the task .
pairs that are only evaluated by consensus differ from the rest largely in the knowledge required from the evaluator to elucidate the annotation , rather than in the factual ambiguity of the sentence .
hence , discussion and consensus assessment of pairs yields a better annotation set than individual annotation .
only 7% ( n = 7 ) of sentence  relationship pairs were marked irrelevant .
following the same guidelines , we created the test corpus . each of the three evaluators assessed different sets of 500 sentence 
sentences deemed not straightforward were left without annotation ( n = 126 , 8.4% ) and later annotated by consensus .
n = 95 ) were considered irrelevant , demonstrating the high precision of the retrieval query .
overall , the procedure used to construct the corpus assured high quality to the 1405 sentence  relationship annotation .
we performed detection of increased , decreased or unknown perturbations using machine learning techniques . for that purpose
, we constructed a vector of features for each relationship  sentence pair using the test corpus above .
one of them represented token presence as a binary vector of token weights wi , d = { w1,  ,w|t| } , where t is the set of sentence tokens : a weight has value 1 if the token is present in the sentence and value 0 otherwise , an approach called set of words ( sow ) ( 11 ) .
hyphenated names were considered both as single and separate tokens in the sow in order to capture affixes like 
since proximity to the gene name can be important in determining a token 's role in describing a perturbation , the sentence was further divided in several sections relative to the gene name 's position .
the sections considered were : n tokens before the gene name , where n = { 5 , 10 , all } , and n tokens after the gene name , where n = { 10 , all}. we noted that many perturbation descriptions were adjacent or very close to the gene name ( e.g.  overexpression of p53 ) .
hence , we included a feature with the distance between the gene name and the beginning of the sentence ( e.g. distance of 0 or 1 may indicate that the perturbation is unknown , e.g. 
we also created a set of features using a small perturbation ontology developed independently over a disease - agnostic set of retrieved sentences .
if a member of the ontology was present in the sentence a feature was added with value 1 , otherwise with value 0 .
all the feature sets described were integrated in a single feature vector for each sentence .
an algorithm based on the principles of maximum entropy ( 12 , http://homepages.inf.ed.ac.uk/s0450736/maxent_toolkit.html ) was trained using 80% of the test corpus , randomly sampled from the cancer and diabetes sets , and tested over the remaining 20% ( results were averaged over 10 runs ) .
the training allowed the algorithm to predict the presence of a perturbation and its direction .
performance measures were evaluated and compared favorably to an svm algorithm ( 13 ) . to create the analysis corpus
, we extended the scope of our methodology used to retrieve the test corpus by eliminating the disease - specific phenotype constraints in the retrieval query ( hence , phenotypes were not included in the query ) .
we applied the query to medline diabetes abstracts after 1996 and retrieved 359 385 relationships related to different conditions and phenotypes .
this output was run through the machine - learning algorithm to create a wide - scope , disease - agnostic set of 191 240 perturbation predictions . to create a diabetes - specific subset ,
only sentences from medline abstracts containing the word diabetes in their mesh descriptors were included .
there were significant differences both in technique and direction between the cancer and diabetes perturbations in the test corpus , with decreased perturbations more prevalent in cancer .
detection of increased and decreased perturbations had a lower f - measure , 72.9% and 71.2% , respectively .
when we excluded the perturbation ontology in feature generation , results were only slightly lower , whereas when we exclusively used the perturbation ontology , results were much lower , notably due to reduced recall . 
straightforward relationships ( 91.6% of the total ) were those that evaluators annotated without consultation with other annotators .
these relationships were less challenging for humans , and the algorithm had better performance over this subset than overall .
due to absence of previous work in perturbation detection , these results can not be compared , but they fall in ranges typical of other successful biomedical text mining tools . 

table 2.perturbation extraction performancecountprecision ( % ) recall ( % ) f - measure ( % ) perturbation69079.279.879.4  no ontology76.479.677.6  only ontology77.259.567.0  straightforward83.378.981.0increased43675.371.172.9  no ontology76.669.972.9  only ontology65.750.956.8  straightforward77.768.872.7decreased25479.065.171.2  no ontology78.563.670.0  only ontology79.858.466.4  straightforward80.467.272.9precision , recall and macro - averaged f - measure were calculated using four different combinations of feature sets : 
full ,  no ontology ,  only ontology and  straightforward. baseline frequencies were 32.8% for increased perturbations and 18.1% for decreased perturbations .
perturbation extraction performance precision , recall and macro - averaged f - measure were calculated using four different combinations of feature sets : 
full ,  no ontology ,  only ontology and  straightforward. baseline frequencies were 32.8% for increased perturbations and
18.1% for decreased perturbations . to determine whether the disease impacted the frequency of perturbation types , we compared cancer and diabetes literature .
the diabetes literature was significantly enriched in increased perturbations , whereas the cancer literature showed an even distribution between increased and decreased perturbations . in cancer ,
more perturbations were performed with antisense oligonucleotide ( n = 9 ) , antibody ( n = 16 ) and rna interference ( n = 26 ) than in diabetes ( antisense , n = 3 ; antibody , n = 2 ; rnai , n = 0 ) .
perturbations in diabetes were more frequently described as injections ( n = 132 ) and/or administered by dose ( n = 57 ) than in cancer ( dose , n = 3 ; injection , n = 6 ) . perturbations in cancer were also more frequently described as being in vivo ( n = 12 ) or in vitro ( n = 14 ) than in diabetes ( in vivo , n = 5 ; in vitro , n = 0 ) .
we examined the sentences for frequently occurring terms , grouped by their level of regulation ( i.e. protein , mrna or dna ) , if known .
many of the terms related to dosing and routes of administration ( e.g. administration , intracerebroventricular , injection ) show a strong dominance in the diabetes literature compared to the cancer literature . although these usually indicate an increasing perturbation , there can be exceptions , such as systemic delivery of an inhibitor . 

table 3.ontology occurrence count in sentences from the test corpus , separated by diabetes and cancer phenotypesgene and protein perturbationscancerdiabetescancerdiabetestotal increasing modifications266586total decreasing modifications267110general increasing modificationsgeneral decreasing modifications  activation6612  down - regulated , -ion120  administered , -ion7120  inactivation50  dose , -age , -dependent1186  inhibition307  ectopic70  repression10  enhanced
 expression100  suppression103  exogenous623dna decreasing modifications  i.c.v.011  deficiency , -ent1512  i.p.07  deletion32  increased2618  dominant - negative30  induction82  knockout12  infused , -ion035  loss1213  injected , -ion766  mutant00  intracerebroventricular051  mutated , -ion03 
intraperitoneal116mrna decreasing modifications  mg / kg,/kg08  anti(-)sense93  oral64  interference51  overexpressed , -ration380  interfering rna20  peripherally07  knockdown120  recombinant710  rna interference40  restoration20  rnai20  subcutaneous02  short hairpin rna10  systemic19  silenced10  treatment2422  sirna150  up - regulation80protein decreasing modificationsdna increasing modifications  antagonist728  adenoviral21  anti-254  adenovirus00  antibody , -ies113  gain - of - function20  blockade , -ing128  gene delivery13  deactivation10  transgenic32  decoy20mrna increasing modifications  fc-00  inducible20  inhibitor5315protein increasing modifications  inverse agonist01  activator100  mabs10  agonist954  neutralization10  analog / analogue27  reduced 
soluble40  targeting60note that total increased and decreased perturbations exceeds values in table 2 since multiple perturbation terms may appear in a single sentence .
ontology occurrence count in sentences from the test corpus , separated by diabetes and cancer phenotypes note that total increased and decreased perturbations exceeds values in table 2 since multiple perturbation terms may appear in a single sentence .
the phenotypes in this study were selected based on prevalence in the literature without regard to their use in vivo or in vitro .
for instance , in diabetes a change in blood glucose levels signals a change in disease state  clearly a phenotype monitored following perturbation in vivo .
likewise , in cancer a change in the number of metastases is exclusively described in in vivo systems . in contrast , insulin sensitivity can be used to describe both in vivo and in vitro systems . for cancer , cell proliferation and apoptosis rates
if our hypothesis that perturbed genes and proteins form the underpinnings of disease mechanisms , these entities should be well represented in pathway diagrams and among drug candidates .
we applied our trained algorithm to a set of 14 345 sentence  relationship facts for genes from our analysis corpus belonging to a diabetes pathway ( figure 1 ) .
predictably , our algorithm found a strong correlation between the number of medline abstracts in which a gene is mentioned and the number of times it is described as perturbed ( r = 0.70 ) .
the results in figure 1 show the difference in intensity and modality in which each gene is described or pursued experimentally .
some genes were typically more increased or decreased , often reflecting their roles in therapeutics .
examples of significantly ( p < 10 ) increased were insulin , interleukin 2 or parathyroid hormone . among the decreased
were such genes as epidermal growth factor receptor ; caspase 8 and plasminogen activator , urokinase receptor . 

figure 1.perturbations extracted from genes involved in the diabetes mellitus type ii pathway in the kyoto encyclopedia of genes and genomes ( 21 ) .
differences in perturbation direction are large for some members of the pathway , note that the scale bar is logarithmic .
perturbations extracted from genes involved in the diabetes mellitus type ii pathway in the kyoto encyclopedia of genes and genomes ( 21 ) .
differences in perturbation direction are large for some members of the pathway , note that the scale bar is logarithmic .
disease associations included in the omim morbid map ( omim online reference , http://www.ncbi.nlm.nih.gov/omim/ ) .
the average abstract mention and gene perturbation counts were higher for genes with morbidmap associations ( t - test , p < 10 ) .
this was consistent with our expectation that genes associated to disease would be the subject of deeper study .
however , genes that had been perturbed numerous times were not necessarily linked to disease .
for example , out of the 100 most perturbed genes , 54 were not linked to disease in omim morbidmap , including at the top such genes as jun oncogene , interleukin 4 , fibroblast growth factor 2 ( basic ) , colony stimulating factor 2 ( granulocyte  macrophage ) , colony stimulating factor 3 ( granulocyte ) and mitogen - activated protein kinase 3 .
perturbations are relevant for areas like the study of gene  disease associations , protein  protein interactions ( ppi ) or gene regulatory networks . gene 
disease association extraction studies have largely focused on simply detecting associations rather than characterizing them ( 7,14 ) .
ppi and gene regulation extraction studies have side - stepped perturbation types , without considering anything further than the experimental technique ( 15,16 ) .
we note that ppi relationships are frequently devoid of perturbations and the focus is on how a ppi was detected , which does not necessarily involve a causative relationship .
the proteomics standards initiative  molecular interactions ( psi - mi ) ontology ( 6 ) is a comprehensive effort to describe molecular interactions .
this ontology , while including a detailed experimental preparation section , lacks expressivity in describing perturbations generically . similarly to bundschus et al .
( 7 ) , it includes a section on expression level with entries  under expressed ,  over expressed and  physiological. the lack of a general framework for recognizing , characterizing and classifying perturbations is surprising when one considers their importance in phenomena encountered experimentally .
researchers with interest in characterizing previous and current perturbation work on a biological system face the challenge of a naturalist trying to deal with animal species without a linnaean taxonomy .
this is reflected in the methodological landscape that was set in the early literature in the fields of biomedical ontologies and text mining .
for example , the comprehensive text mining tool medlee ( 17 ) only considered the 
state of a gene or protein , where the state has an adjectival role such as  mutated in the phrase  mutated x. perturbation descriptions , however , should be considered carefully .
observe the differences between the sentences ( i )  x activates y. and ( ii )  inhibition of x activates y. from the point of view of classic ppi extraction both relationships are equivalent and can be represented with the triplet x activate y. the perturbation in sentence ( ii ) , however , signals that it is likely that protein x is inhibiting protein y instead .
we have called this phenomenon  reversal. given the results of the present assessment , a review of the relationship data available should be considered under this model .
phenotype associations in disease but the principles shown are applicable to other well - known areas , such as ppi , as well as less explored ones such as identification of biomarkers or cellular processes .
a perturbation taxonomy , like the one described in table 4 , could capture the different dimensions that may be of interest to the inquiring scientist .
unknown direction annotation is intended for perturbations whose direction can not be inferred at the sentence level .
molecule annotation characterizes the type of molecule being primarily affected by the perturbation : gene , rna or protein .
expression annotation is used for a change of expression level without clarifying whether the change is in rna or protein .
the following examples illustrate these annotations : a gene mutation is a decrease in gene activity , where the function / activity of a gene is specifically understood as making a wild - type transcript .
exogenous addition of a gene via viral transfection , plasmid transformation , etc . , is an increase in gene abundance .
a gene duplication is an increase in gene abundance while a knockout is a decrease in gene abundance .
dominant negative is a mutation in a gene , which indicates that , compared to wild - type , it has defective function . silencing , knock - down and antisense all apply to a decrease in rna abundance . an antibody blocking a protein decreases the protein activity or function . interfering with a protein binding
treatment , incubation , recombination , or synthetic refer to exogenous addition of protein . 

table 4.proposed annotation guidelinesrelevancedirectionmoleculeeffectrelevantincreasegeneactivityirrelevantdecreasernaabundanceunknownprotein expression proposed annotation guidelines perturbations evolve , notably as new techniques are developed and targets are identified . we expect perturbations to be subject to trends and popularity variations similar to those in other aspects of biomedicine ( 1820 ) .
 molecular perturbations provide a powerful toolset for biomedical researchers to scrutinize the contributions of individual molecules in biological systems . 
 perturbations qualify the context of experimental results and , despite their diversity , share properties in different dimensions in ways that can be formalized . 
 we propose a formal framework to describe and classify perturbations that allows accumulation of knowledge in order to inform the process of biomedical scientific experimentation and target analysis . 
 we apply this framework to develop a novel algorithm for automatic detection and characterization of perturbations in text and show its relevance in the study of gene  phenotype associations and protein  protein interactions in diabetes and cancer . 
 analyzing perturbations introduces a novel view of the multivariate landscape of biological systems . 
INTRODUCTION
MATERIALS AND METHODS
RESULTS
DISCUSSION
FUNDING