celiac disease ( cd ) is a common autoimmune disorder estimated to affect approximately 1% of the us population .
many studies have evaluated methods for identifying high - risk populations of patients in need of proactive screening for case identification . as with many chronic diseases ,
quality improvement approaches can be applied to populations of patients with cd to standardize care and track outcomes . effectively identifying and tracking outcomes of these cohorts of patients
however , the development of cd patient registries often requires significant resources including practitioners , informatics teams , and administrative support .
our center has developed and maintained a patient registry including all newly diagnosed patients since early 2012 .
prior studies have evaluated the reliability of international classification of diseases-9 ( icd-9 ) searches to identify patients with cd with only moderate success yielding only 17% of sensitivity .
contrast experiments have been done using manually defined keywords to classify high - risk cd cases generating much more accurate results at 73% of sensitivity .
few efforts combining different aspects of available information to further improve classification accuracy including the use of data from visits , laboratory tests and pathology reports have been described .
pathology reports have been previously used in building natural language processing ( nlp ) systems for automated chart review of patients . here , we define nlp broadly as methods to process textual data .
one study found pathology reports to be insufficient for detecting breast cancer when using nlp approach compared with manual approaches while others found the use of nlp for analyzing pathology reports to be effective on detection of colon cancer and manifestation of prostatectomy details .
currently , a lack of research exists on investigating the usefulness of pathology reports on automated cd detection . based on a subset of patients prescreened using icd-9 code 579.0
, we evaluated the effectiveness of using natural language features from pathology reports to improve the identification of cd patients .
this paper was among the first to adopt a machine learning approach to further increase the accuracy of cd patient identification based on a combination of pathological and clinical metrics .
in contrast to prior research , we conducted contrast experiments using both natural language features and clinical features under different machine learning configurations for celiac identification .
usa ) clinical system on patients who visited our hospital from 2012 to 2015 and had icd-9 code of 579.0 assigned indicating concern for cd .
our subsequent analysis was , therefore , based on patients with concerns rather than general patient population .
a laboratory test result was marked with a nominal value of either high ( 20 eu / ml ) or normal ( < 20 eu / ml ) .
an upper endoscopy ( egd ) may also be performed then resulted in a text - based pathology report , which could be used for nlp analysis .
the registry that we developed are all confirmed diagnosis of cd based on confirmation with both abnormal laboratory / serology testing and biopsy confirmation after detailed chart review . for each of our patients in the registry , cd
most confirmed cd patients ( confirmed by our experts for having cd ) only had one pathology report while false - positive cd patients ( confirmed by our experts for not having cd ) had either one or zero pathology report .
twenty - nine patients who had two or more pathology reports were excluded from our experiment as they presented a challenge to our analysis if inconsistent information about cd diagnoses existed at different periods of time . in patients with multiple ttg laboratories ,
the most elevated laboratory result obtained within the past 6 months of the egd was chosen to represent the laboratory status of that cd patient .
variables used for celiac classification are listed in table 1 . for the report category ,
the total number of pathology reports was used with one indicating patients having egd done and 0 otherwise ( v1 ) . both user - define phrases ( high - risk phrases , or v2 ) and
automatically extracted phrases ( n - gram phrases generated from our machine learning program , or v3 ) were chosen to evaluate the effectiveness of utilizing pathology reports .
for the laboratory category , both the laboratory results ( v4 ) and total number of labs done were used ( v5 ) . for the icd-9 category ,
the total number of cd icd-9 codes assigned ( v6 ) was used for classification .
variables for celiac disease classification to evaluate the effectiveness of using different combinations of variables and feature selection methods , we further developed two additional set of variables based on the aforementioned six sets of variables above ( v1v6 ) . combining all first six categories of variables created the variable set 7 ( v7 ) . automatically selecting features out of v6 using
a feature selection algorithm resulted in the variable set 8 ( v8 ) [ table 1 ] . for expert - knowledge - driven feature selection
, we identified a list of nine key phrases described by clinicians as commonly used in pathology reports as indicators of high - risk cd .
these nine key phrases , which were all converted to lower cases , included brunner 's gland hyperplasia , flattening villi , intraepithelial lymphocytes , marsh gland stage , marsh lesion , marsh s3 lesion , shortened villi , villous blunting , and villous atrophy .
each key phrase was first converted to a binary variable with the value of one indicating the phrase existed in the pathology report and 0 otherwise .
automatically extracted n - gram features were phrases generated using weka , an open source java - based machine - learning program , developed by the university of waikato , new zealand .
the maximum number of words included in n - gram feature was chosen to be three resulting in a large set of text features consist of uni- , bi- and tri - grams .
we only collected phrases up to trigrams because the length of the longest user - provided high - risk phrase was also three .
we kept the first 4000 text features with all stop words ( i.e. , a list of common words such as  the , 
skipped . during the process of converting documents into n - gram features , term frequency - inverse document frequency ( tf - idf ) was calculated on each n - gram feature .
tf - idf is a data transformation method commonly used in text classification to show the importance of a word to a document in a collection of documents .
it has been found useful in several recent classification and information retrieval studies using medical documents .
finally , all n - grams were converted to lowercase to avoid duplications during tf - idf transformation .
therefore , we only chose the three most representative classifiers from four major categories to experiment with in this paper .
the four classifier categories included bayes , function - based , lazy model and tree classifiers . in each category
previous studies on classifying medical documents showed that the performance variation among different models could be relatively small within the category compared with across the categories .
this was another main reason we only chose to experiment with three classifiers rather than all available classifiers from each category .
selected classification models for experiments there are different ways to configure a classifier . in most cases of our experiments , the default configuration was chosen as long as the setting was compatible with the type of data we classify .
for example , for libsvm the standard algorithm of regularized c - support vector classification was selected as well as the default radial kernel type [ table 2 ] .
for each one of the twelve classification models [ table 2 ] , we also experimented with the use of the eight different variable configurations ( v1v8 ) [ table 1 ] to find the best variable configuration to produce the best possible classifier .
each variable configuration was used as the inputs to each one of the twelve classification models .
resulting in twelve experiments for each variable configuration . the classification model with the best performance
overview of the architectural design of experiments feature selection ( or variable selection ) is critical for optimizing a classification model .
appropriately selected features could minimize the chance of overfitting as well as reduce feature redundancy and training time in the face of high - dimensional variables .
a typical feature selection process included two separate steps available in the weka tool we used .
the first step is to define a feature evaluation method using criteria based on which features are selected . the second step is to define a search method where features will be searched and selected into the final feature set .
there is not a single rule of thumb for choosing the best possible feature selection method . after performing several experiments
, we found the default feature selection method in weka worked the best for us .
the feature selection model we used implemented a correlation - based feature evaluation model called cfssubseteval .
the cfssubseteval feature evaluation model ranks the features based on their correlation with the class while minimizing the redundancy of variables .
the underlying feature search model was the best first feature search model which searches the attribute space by allowing backtracking .
detailed discussion on cfssubseteval and best first is beyond the scope of this paper and we recommend researchers to refer to the original work from the original authors .
each classifier was validated using a 10-fold cross - validation method on our entire dataset . confirmed cases and controls
the best model with the highest area under the curve ( auc ) for each variable configuration was reported .
auc was a commonly used metric for machine learning - based evolution as a chance corrected measure and independent of class distributions .
chi - square test was conducted to test the statistical association between each of the variables and celiac status .
in addition to auc , we also reported precision , recall , f1 , and kappa .
precision , recall , and f1 were classic machine learning measures for evaluating overall accuracy with 1 being the best and 0 being the worst for predicting positive cases .
the kappa score measured the level of agreement between the user and the classifier with 1 being complete agreement and 0 being complete disagreement .
for expert - knowledge - driven feature selection , we identified a list of nine key phrases described by clinicians as commonly used in pathology reports as indicators of high - risk cd .
these nine key phrases , which were all converted to lower cases , included brunner 's gland hyperplasia , flattening villi , intraepithelial lymphocytes , marsh gland stage , marsh lesion , marsh s3 lesion , shortened villi , villous blunting , and villous atrophy .
each key phrase was first converted to a binary variable with the value of one indicating the phrase existed in the pathology report and 0 otherwise .
automatically extracted n - gram features were phrases generated using weka , an open source java - based machine - learning program , developed by the university of waikato , new zealand .
the maximum number of words included in n - gram feature was chosen to be three resulting in a large set of text features consist of uni- , bi- and tri - grams .
we only collected phrases up to trigrams because the length of the longest user - provided high - risk phrase was also three .
we kept the first 4000 text features with all stop words ( i.e. , a list of common words such as  the ,   of ,   is ,  etc . ) skipped . during the process of converting documents into n - gram features , term frequency - inverse document frequency ( tf - idf )
tf - idf is a data transformation method commonly used in text classification to show the importance of a word to a document in a collection of documents .
it has been found useful in several recent classification and information retrieval studies using medical documents . finally , all n - grams were converted to lowercase to avoid duplications during tf - idf transformation .
therefore , we only chose the three most representative classifiers from four major categories to experiment with in this paper .
the four classifier categories included bayes , function - based , lazy model and tree classifiers . in each category
previous studies on classifying medical documents showed that the performance variation among different models could be relatively small within the category compared with across the categories .
this was another main reason we only chose to experiment with three classifiers rather than all available classifiers from each category .
selected classification models for experiments there are different ways to configure a classifier . in most cases of our experiments ,
the default configuration was chosen as long as the setting was compatible with the type of data we classify .
for example , for libsvm the standard algorithm of regularized c - support vector classification was selected as well as the default radial kernel type [ table 2 ] .
for each one of the twelve classification models [ table 2 ] , we also experimented with the use of the eight different variable configurations ( v1v8 ) [ table 1 ] to find the best variable configuration to produce the best possible classifier .
each variable configuration was used as the inputs to each one of the twelve classification models . resulting in twelve experiments for each variable configuration .
the classification model with the best performance was chosen as the best model for the underlying variable configuration used .
overview of the architectural design of experiments feature selection ( or variable selection ) is critical for optimizing a classification model .
appropriately selected features could minimize the chance of overfitting as well as reduce feature redundancy and training time in the face of high - dimensional variables .
a typical feature selection process included two separate steps available in the weka tool we used .
the first step is to define a feature evaluation method using criteria based on which features are selected . the second step is to define a search method where features will be searched and selected into the final feature set .
there is not a single rule of thumb for choosing the best possible feature selection method . after performing several experiments
, we found the default feature selection method in weka worked the best for us .
the feature selection model we used implemented a correlation - based feature evaluation model called cfssubseteval .
the cfssubseteval feature evaluation model ranks the features based on their correlation with the class while minimizing the redundancy of variables .
the underlying feature search model was the best first feature search model which searches the attribute space by allowing backtracking .
detailed discussion on cfssubseteval and best first is beyond the scope of this paper and we recommend researchers to refer to the original work from the original authors .
therefore , we only chose the three most representative classifiers from four major categories to experiment with in this paper .
the four classifier categories included bayes , function - based , lazy model and tree classifiers . in each category
previous studies on classifying medical documents showed that the performance variation among different models could be relatively small within the category compared with across the categories .
this was another main reason we only chose to experiment with three classifiers rather than all available classifiers from each category .
selected classification models for experiments there are different ways to configure a classifier . in most cases of our experiments ,
the default configuration was chosen as long as the setting was compatible with the type of data we classify .
for example , for libsvm the standard algorithm of regularized c - support vector classification was selected as well as the default radial kernel type [ table 2 ] .
for each one of the twelve classification models [ table 2 ] , we also experimented with the use of the eight different variable configurations ( v1v8 ) [ table 1 ] to find the best variable configuration to produce the best possible classifier .
each variable configuration was used as the inputs to each one of the twelve classification models . resulting in twelve experiments for each variable configuration .
the classification model with the best performance was chosen as the best model for the underlying variable configuration used .
appropriately selected features could minimize the chance of overfitting as well as reduce feature redundancy and training time in the face of high - dimensional variables .
a typical feature selection process included two separate steps available in the weka tool we used .
the first step is to define a feature evaluation method using criteria based on which features are selected . the second step is to define a search method where features will be searched and selected into the final feature set .
there is not a single rule of thumb for choosing the best possible feature selection method . after performing several experiments
, we found the default feature selection method in weka worked the best for us .
the feature selection model we used implemented a correlation - based feature evaluation model called cfssubseteval .
the cfssubseteval feature evaluation model ranks the features based on their correlation with the class while minimizing the redundancy of variables .
the underlying feature search model was the best first feature search model which searches the attribute space by allowing backtracking .
detailed discussion on cfssubseteval and best first is beyond the scope of this paper and we recommend researchers to refer to the original work from the original authors .
each classifier was validated using a 10-fold cross - validation method on our entire dataset . confirmed cases and controls
the best model with the highest area under the curve ( auc ) for each variable configuration was reported .
auc was a commonly used metric for machine learning - based evolution as a chance corrected measure and independent of class distributions .
chi - square test was conducted to test the statistical association between each of the variables and celiac status .
in addition to auc , we also reported precision , recall , f1 , and kappa .
precision , recall , and f1 were classic machine learning measures for evaluating overall accuracy with 1 being the best and 0 being the worst for predicting positive cases .
the kappa score measured the level of agreement between the user and the classifier with 1 being complete agreement and 0 being complete disagreement .
our data extraction resulted in identification of 1498 unique patients with icd-9 codes suggesting possible cd .
we found 363 confirmed cases ( positive celiac cases [ table 3 ] ) matching the previously establish celiac registry and 1135 false positive cases ( negative celiac cases [ table 3 ] ) . demographic data are shown in table 3 .
chi - square test found statistically significant associations between age , gender , race , and celiac status .
classification variables such as those about visit , laboratory and pathology report counts and laboratory results were all found to be significant for differentiating cd patients . as this study mainly focused on using clinical variables for celiac classification we did not include any demographic variables in building classifiers [ table 3 ] .
patient characteristics and their association with celiac disease table 4 listed the results summarizing the best classification model out of each variable configuration .
models were ranked based on auc score from the highest to the lowest . among all twelve classification models , we experimented with the libsvm
was found to be the top performer in five experiments ( experiments 3 , 4 , 5 , 6 , 8) , followed by logistic model in two experiments ( experiments 1 , 7 ) and nave bayes in one experiment ( experiment 2 ) .
the best performing model was the logistic model using all available features filtered by an automatic feature selection process .
it achieved auc of 0.94 , kappa of 0.78 , and f1 of 0.92 [ table 4 ] .
statistics of the best classification models for each variable configuration among experiments using single category of variables ( experiment 2 and 48 ) , nave bayes model based on automatically selected n - gram features generated the best results : auc of 0.92 , kappa of 0.73 , and f1 of 0.90 , which was also the overall second best performing model . here , we also conducted feature selection on executing the nave bayes model as we aimed to find the best possible performance of using text features . despite the slightly lower performance ,
the nave bayes model was much faster than other models in terms of model building and model evaluation time . in our case
, it only took nave bayes a few seconds to build and evaluate the model in 10-fold cross - validation while it took a few minutes for libsvm and logistic model to finish in the case of using all 4000 features . using only high - risk phrases , number of laboratories and number of icd-9 ( experiment 6 , 7
, 8) resulted in the lowest model accuracy . using laboratory resulted in higher accuracy than the previous three .
it was also intriguing that using the number of pathology reports alone only slightly underperformed using report text features alone .
however , additional features resulted in higher accuracy given the results based on a combination of different features .
table 5 listed some of the automatically selected n - gram phrases in contrast to user - provided high - risk phrases .
it was found that automatically created n - gram features not only covered the majority of user - provided high - risk phrases but also had the potential of being more accurate and specific in finding relevant information .
for example , automatically generated n - grams such as celiac and abdominal pain were directly relevant features to cd but were not initially on our clinicians list .
comparison of user - defined high - risk phrases and automated key phrases some of the provided high - risk features were not included in our automatically selected feature list such as the phrase of marsh lesion ( converted to lower case for processing ) .
we found this may be due to the extreme low frequency of the usage of such phrases in our pathology reports ( only 1 out of all 363 confirmed cases had this exact phrase ) . in table 5 , the italicized phrases highlighted some of the overlaps between the two lists : user - provided and automatically extracted .
the automatically extracted list only represented a subset of most relevant n - gram features [ table 5 ] .
our machine learning approach could quickly and effectively choose the most relevant variables from various clinical sources for automated cd classification .
the level of accuracy we achieved was higher compared with previous studies on the same topic .
n - gram features performed well in our cd detection experiment which agreed with and further improved the results from previous studies using text features .
in contrast to previous studies , our automated feature generation and selection approaches were more efficient in finding relevant classification features in a short period of time and more effective than user - provided high - risk phrases .
this result showed the automated list was more effective in providing a comprehensive list of relevant keywords than manual methods .
the number of laboratories done was as effective as the actual lab results reported in our case for cd classification ( auc of 0.71 vs. 0.83 ) .
this could be because an increased number of labs done or the fact that the laboratory was ordered may suggest a higher likelihood of abnormal laboratory result .
our experiments also agreed with previous studies on the fact that icd-9 code alone produced very poor results ( kappa of 0.28 , f1 of 0.75 and auc of 0.63 ) .
therefore , we suggested researchers only use icd-9 code as a prescreening tool for subsetting patients as in this study .
in addition , our results added further evidence to previous studies that nave bayes model generated the most accurate results even if only pathology report features were used ( kappa of 0.73 , f1 of 0.90 , and auc of 0.92 ) .
this further indicates that pathology report , as an important pierce of synoptic reporting of celiac status , contains invaluable information for celiac classification .
in addition , by combining both clinical and text features altogether , classification accuracy and kappa score could be further improved ( kappa of 0.78 , f1 of 0.92 , and auc of 0.94 ) .
this suggests the complementary effect of using both structured data and unstructured data for celiac classification .
finally , feature selection was found to be a crucial step in celiac detection for further improving the efficiency and accuracy of classification .
feature selection dramatically removed redundant and noninformative terms ( from about 4000 to 60 ) .
although new to the task of cd classification , feature selection methods have been found widely used in other disease classification studies including alzheimer 's disease and asthma . in our case , some variables ( i.e. , number of pathology reports and lab results ) could individually predict celiac at a reasonably high accuracy . however , this result may be problem dependent
. other studies implementing the same metrics may or may not result in similar performance .
even that , we still observed performance gain by adding nlp features with both accuracy and agreement increased by a reasonable amount .
the automated tool we developed could speed up the process of refining the subset population initially identified based on icd-9 code 579.0 .
this refining process can automatically confirm the patient as either positive or negative cd case based on the knowledge of the patient obtained from the laboratories and pathology notes .
the classification accuracy was over 90% of correctness ( f1 ) in our case while it greatly eliminated the need of often time - consuming process of manual review .
first , since the cd classification task was performed on a subset of patients prescreened by icd-9 code rather than on the general patient population , the performance of final classification results may depend on the accuracy of initial icd-9 assignment . for example , if actual cd patients were not assigned an icd-9 code 579.0 initially , they will not have the chance of being further identified by our machine learning system .
it is , therefore , important to make sure the initial icd-9 code assignment will include as many high - risk cases as possible .
second , although feature selection method was found to be critical in yielding high performance , we have not conducted systematic experiments with all possible feature selection methods provided in the weka machine learning system . given our experience , it was likely that some methods may further improve accuracy while others may even lower the accuracy . finally , for most machine learning algorithms we experimented , only the default configuration was used for classification .
it was likely different configurations of parameter setting may lead to very different results of accuracy .
therefore , researchers are encouraged to experiment with different classifier settings to achieve the best possible results for their tasks .
since the cd classification task was performed on a subset of patients prescreened by icd-9 code rather than on the general patient population , the performance of final classification results may depend on the accuracy of initial icd-9 assignment .
for example , if actual cd patients were not assigned an icd-9 code 579.0 initially , they will not have the chance of being further identified by our machine learning system .
it is , therefore , important to make sure the initial icd-9 code assignment will include as many high - risk cases as possible .
second , although feature selection method was found to be critical in yielding high performance , we have not conducted systematic experiments with all possible feature selection methods provided in the weka machine learning system . given our experience , it was likely that some methods may further improve accuracy while others may even lower the accuracy . finally , for most machine learning algorithms we experimented , only the default configuration was used for classification .
it was likely different configurations of parameter setting may lead to very different results of accuracy .
therefore , researchers are encouraged to experiment with different classifier settings to achieve the best possible results for their tasks .
in this paper , we compared results from 96 machine - learning experiments for cd identification based on 12 classification model variations and eight feature set variations .
the logistic model that used a combination of clinical and pathology report features generated the best results : kappa of 0.78 , f1 of 0.92 , and auc of 0.94 .
our results were in agreement with previous studies on the insufficiency of using icd-9 codes alone and the merits of using pathology report features for cd identification .
this study provides new evidence on adopting feature selection techniques to further improve classification efficiency and accuracy based on a subpopulation of prescreened patients using icd-9 code .
this study demonstrated a viable computational approach to automatically reviewing and confirming prescreened cd patients based on icd-9 code at the state of the art accuracy much improved from previous research on the same topic .
 background : celiac disease ( cd ) is a common autoimmune disorder . 
 efficient identification of patients may improve chronic management of the disease . 
 prior studies have shown searching international classification of diseases-9 ( icd-9 ) codes alone is inaccurate for identifying patients with cd . in this study , we developed automated classification algorithms leveraging pathology reports and other clinical data in electronic health records ( ehrs ) to refine the subset population preselected using icd-9 code ( 579.0).materials and methods : ehrs were searched for established icd-9 code ( 579.0 ) suggesting cd , based on which an initial identification of cases was obtained . 
 in addition , laboratory results for tissue transglutaminse were extracted . 
 using natural language processing we analyzed pathology reports from upper endoscopy . 
 twelve machine learning classifiers using different combinations of variables related to icd-9 cd status , laboratory result status , and pathology reports were experimented to find the best possible cd classifier . 
 ten - fold cross - validation was used to assess the results.results:a total of 1498 patient records were used including 363 confirmed cases and 1135 false positive cases that served as controls . logistic model based on both clinical and pathology report 
 features produced the best results : kappa of 0.78 , f1 of 0.92 , and area under the curve ( auc ) of 0.94 , whereas in contrast using icd-9 only generated poor results : kappa of 0.28 , f1 of 0.75 , and auc of 0.63.conclusion:our automated classification system presented an efficient and reliable way to improve the performance of cd patient identification . 
INTRODUCTION
MATERIALS AND METHODS
Feature Extraction from Pathology Reports for Classification
Building Celiac Disease Classifiers
Classification model
Variable configuration
Feature selection
Classifier Evaluation
RESULTS
DISCUSSION
Limitations
CONCLUSION
Financial Support and Sponsorship
Conflicts of Interest