in 2000 , the 24 member boards of the american board of medical specialties , including the american board of pathology , agreed to evolve their recertification programs toward continuous professional development to ensure that physicians are committed to lifelong learning and competency in their specialty areas ( 1 , 2 ) . measuring competencies occurs in a variety of ways according to specialty , which each board determined in 2006 ( 1 ) .
one way competency measures are assessed by the board is practice performance assessment , which involves demonstrating use of best evidence and practices compared to peers and national benchmarks ( 1 ) .
all specialty boards are now in the process of implementing their respective maintenance of certification requirements . to incentivize physicians to participate in new maintenance of certification activities , the affordable care act ( 3 ) required the centers for medicare & medicaid services to implement the physician quality reporting system incentive program , which is a voluntary reporting program that provides an incentive payment to board certified physicians who acceptably report data on quality measures for physician fee schedule covered services furnished to medicare part b fee - for - service recipients ( 4 ) .
the physician quality reporting system represents a stunning change and likely an advancement in physicians professional development ; namely , the opportunity for reporting of and reflection upon actual clinical practice activities .
it is too early to tell what the above changes will mean for improvements in clinical care , or what the future holds in terms of required versus voluntary reporting of clinical care performance .
however , there is much to be learned about how physicians process information about their clinical performance , which is directly relevant to parts 2 and 3 of maintenance of certification , lifelong learning and self- and practice performance assessment ( 1 ) .
reviews on the effectiveness of continuing medical education indicate that for physicians to change their practice behaviors they must understand that a gap exists between their actual performance and what is considered optimal ( 5 ) , which is not always immediately evident .
identifying an existing gap is the initial step , but understanding what might be causing the gap could assist physicians to identify strategies to improve their practice .
interpretation of breast pathology is an area where significant diagnostic variation exists ( 68 ) .
we conducted a study with 92 pathologists across the u.s . that involved administering test sets of 60 breast pathology cases and comparing participants interpretations to a consensus reference diagnosis determined by a north american panel of experienced breast pathologists .
we used the findings from the test set study to design an individualized educational intervention that identified the diagnostic agreement for four categories of breast pathology interpretations : benign without atypia , atypia ( including atypical ductal hyperplasia and intraductal papilloma with atypia ) , ductal carcinoma in situ and invasive cancer . in this paper
, we report on : 1 ) breast pathologists estimates on how their estimated diagnostic interpretations compare to the reference diagnoses before they learned the actual reference diagnosis , 2 ) physician characteristics associated with under- and overestimates of diagnostic agreement with specialist consensus diagnosis , 3 ) the extent to which pathologists recognized the gap after the intervention , and 4 ) what pathologists reported they would change about their clinical practices . the approach we undertook to understand how pathologists process information about breast pathology cases has important implications for maintenance of certification activities in the field of pathology and for pathologist recognition of cases likely to have diagnostic disagreements .
all study activities were health insurance portability and accountability act compliant and were approved by the institutional review boards of the university of washington , dartmouth college , the university of vermont , fred hutchinson cancer research center , and providence health & services of oregon .
a study - specific certificate of confidentiality was also obtained to protect the study findings from forced disclosure of identifiable information .
detailed information on the development of the test set is published elsewhere ( 9 ) .
briefly , test set cases were identified from biopsy specimens obtained from mammography registries with linkages to breast pathology and/or tumor registries in vermont and new hampshire ( 10 ) .
samples were chosen from cases biopsied between january 1 , 2000 and december 31 , 2007 .
a consensus process was undertaken by three experienced breast pathologists to come to a final reference diagnosis for each case in the test set , which is described elsewhere ( 12 ) . using a random stratified approach ,
patient cases were assigned into one of four test sets , which contained 60 cases each ( 240 total unique cases ) and represented the following diagnostic categories : benign without atypia ( 30% ) , atypia ( 30% ) , ductal carcinoma in situ ( 30% ) and invasive cancer ( 10% ) .
this distribution resulted in an oversampling of more atypia and ductal carcinoma in situ cases , which would help quantify the diagnostic challenges to be included in the educational intervention .
pathologists were recruited to participate from eight geographically diverse states ( ak , me , mn , nm , nh , or , wa , vt ) via email , telephone , and street mail . all practicing pathologists in these states were invited to participate . to be eligible for participation ,
pathologists interpret breast biopsies as part of their practice , have been signing out breast biopsies for at least one year post residency or fellowship , and intend to continue interpreting breast biopsies for at least one year post enrollment .
all participating physicians completed a brief 10-minute survey that assessed their demographic characteristics ( age , sex ) , training and clinical experiences ( fellowship , case load , interpretive volume , years interpreting , academic affiliations ) , and perceptions of how challenging breast pathology is to interpret .
two hundred and fifty - two pathologists of 389 eligible participants ( 65% ) agreed to take part in the study . among these , 126 were randomized to the current study ( 126 participants were offered participation in a related future study ) , 115 of these completed interpretation of test set cases and 94 completed the educational intervention upon which this study is based .
the educational intervention was designed to provide a research - based review of differences among pathologists in breast tissue interpretation .
it was a self - paced internet hosted program individualized to participant s interpretive performance on their respective test set .
before they reviewed how their interpretations compared to the reference diagnosis , we asked participants to compare how similar the test set cases were to cases they see in their practice ( response options ranged from 
i always see cases like these ) , and we asked them to indicate the number of continuing medical education hours they have undertaken in breast pathology interpretation over the past year as well as their continuing medical education preferences ( instructor led , self - directed or other ) .
lastly , at the beginning of the continuing medical education program , we asked participants to estimate , globally , how their interpretations would compare to the reference diagnosis ( over interpret , under interpret , do nt know ) and to estimate the proportion of their diagnoses on test cases they thought would agree with the reference diagnoses within each of the four diagnostic categories reviewed ( benign , atypia , ductal carcinoma in situ , invasive ) .
participants estimates for each of the four categories were then assigned a weighted average using the proportion of their diagnoses they estimated agreed with the reference diagnoses and , as the weighting pattern , the number of cases each participant interpreted within each diagnostic class .
subtracting the weighted estimate of agreement with specialist consensus diagnoses from the participants actual agreement gave us the difference between perceived and actual agreement , or 
we then categorized these differences for each of the four diagnostic categories according to three classifications : 1 ) those who were within one standard deviation of the mean ( when compared to the reference diagnosis ) were those who closely estimated their performance ; 2 ) those whose estimates were more than one standard deviation above the mean were those who overestimated their agreement rates ; and 3 ) those whose estimates were more than one standard deviation below the mean were those who underestimated their agreement rates .
the continuing medical education program then proceeded to show the pathologists , on a case - by - case basis , how their independent interpretations actually compared with both the reference diagnosis and with other participating pathologists . by clicking on the case number
, participants could view a digital whole slide image of the case on their computer screens and could dynamically magnify and scan the image similar to a microscope ( virtual digital microscope ) .
participants were given the opportunity to share their own thoughts on each case after seeing their results and reading the experienced pathologists teaching points .
the intervention included an open text field that asked them what they would change in their clinical practice as a result of what they learned in this program .
lastly , a post - test asked study participants to again rate how their interpretations compared to the reference diagnosis , so we could assess the extent to which they recognized any areas of significant disagreement with reference diagnoses .
we then asked them to complete required knowledge questions so that continuing medical education credits could be awarded .
completion of all study activities , including the test set interpretations and the continuing medical education program resulted in awarding up to 20 category 1 continuing medical education hours ( majority awarded 1517 hours ) .
we used descriptive statistics to characterize the demographic and clinical training experience as well as ratings of the test sets and challenges involved in interpreting breast pathology .
histograms were used to compare the pre - test differences between overall perceived and actual diagnostic agreement as well as agreement within each diagnostic group in both pre- and post - test settings . among the 94 participants who completed the continuing medical education program , differences between perceived and actual performance rates could not be assigned to 2 participants ( 2.1% ) due to missing responses .
categorical data are presented as frequencies and percentages and for continuous variables , values are reported as the mean and standard deviation .
cumulative logit models were fit to test the association between each participant characteristic and the ordered three - category dependent variable , perception gaps . each model was adjusted for actual overall agreement .
analyses were performed using a commercially available sas v9.4 ( sas institute , cary nc ) . a classical content analysis
( 12 ) was performed on participants test responses regarding whether there was anything they would change in their practice as a result of what they learned from this program .
this process allowed us to identify and describe thematic areas according to over and underestimates of diagnostic agreement .
all study activities were health insurance portability and accountability act compliant and were approved by the institutional review boards of the university of washington , dartmouth college , the university of vermont , fred hutchinson cancer research center , and providence health & services of oregon .
a study - specific certificate of confidentiality was also obtained to protect the study findings from forced disclosure of identifiable information .
detailed information on the development of the test set is published elsewhere ( 9 ) .
briefly , test set cases were identified from biopsy specimens obtained from mammography registries with linkages to breast pathology and/or tumor registries in vermont and new hampshire ( 10 ) .
samples were chosen from cases biopsied between january 1 , 2000 and december 31 , 2007 .
a consensus process was undertaken by three experienced breast pathologists to come to a final reference diagnosis for each case in the test set , which is described elsewhere ( 12 ) . using a random stratified approach ,
patient cases were assigned into one of four test sets , which contained 60 cases each ( 240 total unique cases ) and represented the following diagnostic categories : benign without atypia ( 30% ) , atypia ( 30% ) , ductal carcinoma in situ ( 30% ) and invasive cancer ( 10% ) .
this distribution resulted in an oversampling of more atypia and ductal carcinoma in situ cases , which would help quantify the diagnostic challenges to be included in the educational intervention .
pathologists were recruited to participate from eight geographically diverse states ( ak , me , mn , nm , nh , or , wa , vt ) via email , telephone , and street mail . all practicing pathologists in these states were invited to participate . to be eligible for participation ,
pathologists interpret breast biopsies as part of their practice , have been signing out breast biopsies for at least one year post residency or fellowship , and intend to continue interpreting breast biopsies for at least one year post enrollment .
all participating physicians completed a brief 10-minute survey that assessed their demographic characteristics ( age , sex ) , training and clinical experiences ( fellowship , case load , interpretive volume , years interpreting , academic affiliations ) , and perceptions of how challenging breast pathology is to interpret .
two hundred and fifty - two pathologists of 389 eligible participants ( 65% ) agreed to take part in the study . among these , 126 were randomized to the current study ( 126 participants were offered participation in a related future study ) , 115 of these completed interpretation of test set cases and 94 completed the educational intervention upon which this study is based .
the educational intervention was designed to provide a research - based review of differences among pathologists in breast tissue interpretation .
it was a self - paced internet hosted program individualized to participant s interpretive performance on their respective test set .
before they reviewed how their interpretations compared to the reference diagnosis , we asked participants to compare how similar the test set cases were to cases they see in their practice ( response options ranged from 
i always see cases like these ) , and we asked them to indicate the number of continuing medical education hours they have undertaken in breast pathology interpretation over the past year as well as their continuing medical education preferences ( instructor led , self - directed or other ) .
lastly , at the beginning of the continuing medical education program , we asked participants to estimate , globally , how their interpretations would compare to the reference diagnosis ( over interpret , under interpret , do nt know ) and to estimate the proportion of their diagnoses on test cases they thought would agree with the reference diagnoses within each of the four diagnostic categories reviewed ( benign , atypia , ductal carcinoma in situ , invasive ) .
participants estimates for each of the four categories were then assigned a weighted average using the proportion of their diagnoses they estimated agreed with the reference diagnoses and , as the weighting pattern , the number of cases each participant interpreted within each diagnostic class . subtracting the weighted estimate of agreement with specialist consensus diagnoses from the participants actual agreement gave us the difference between perceived and actual agreement , or 
we then categorized these differences for each of the four diagnostic categories according to three classifications : 1 ) those who were within one standard deviation of the mean ( when compared to the reference diagnosis ) were those who closely estimated their performance ; 2 ) those whose estimates were more than one standard deviation above the mean were those who overestimated their agreement rates ; and 3 ) those whose estimates were more than one standard deviation below the mean were those who underestimated their agreement rates .
the continuing medical education program then proceeded to show the pathologists , on a case - by - case basis , how their independent interpretations actually compared with both the reference diagnosis and with other participating pathologists . by clicking on the case number
, participants could view a digital whole slide image of the case on their computer screens and could dynamically magnify and scan the image similar to a microscope ( virtual digital microscope ) .
participants were given the opportunity to share their own thoughts on each case after seeing their results and reading the experienced pathologists teaching points .
the intervention included an open text field that asked them what they would change in their clinical practice as a result of what they learned in this program .
lastly , a post - test asked study participants to again rate how their interpretations compared to the reference diagnosis , so we could assess the extent to which they recognized any areas of significant disagreement with reference diagnoses .
we then asked them to complete required knowledge questions so that continuing medical education credits could be awarded .
completion of all study activities , including the test set interpretations and the continuing medical education program resulted in awarding up to 20 category 1 continuing medical education hours ( majority awarded 1517 hours ) .
we used descriptive statistics to characterize the demographic and clinical training experience as well as ratings of the test sets and challenges involved in interpreting breast pathology .
histograms were used to compare the pre - test differences between overall perceived and actual diagnostic agreement as well as agreement within each diagnostic group in both pre- and post - test settings . among the 94 participants who completed the continuing medical education program , differences between perceived and actual performance rates could not be assigned to 2 participants ( 2.1% ) due to missing responses .
categorical data are presented as frequencies and percentages and for continuous variables , values are reported as the mean and standard deviation .
cumulative logit models were fit to test the association between each participant characteristic and the ordered three - category dependent variable , perception gaps . each model was adjusted for actual overall agreement .
analyses were performed using a commercially available sas v9.4 ( sas institute , cary nc ) . a classical content analysis
( 12 ) was performed on participants test responses regarding whether there was anything they would change in their practice as a result of what they learned from this program .
this process allowed us to identify and describe thematic areas according to over and underestimates of diagnostic agreement .
overall ( all diagnostic categories combined ) , participants were very accurate in their estimates of their perceived versus actual diagnostic agreement with specialist consensus diagnosis with a slight tendency to overestimate their agreement , with a mean gap between perceived vs. actual agreement of 5.5% ( mean for actual agreement = 75.9% ; mean for estimated agreement = 81.4% , ) ( figure 1a ) .
figure 1b shows the distribution in performance estimates for under - estimating ( < 6.7% ) , closely estimating ( 75.6% ) and overestimating diagnostic agreement ( > 17.7% ) .
as indicated in table 1 , non - academic affiliated pathologists were more likely to more closely estimate their performance relative to academic affiliated pathologists ( 77.6% versus 48% ; p=0.001 ) , whereas participants affiliated with an academic medical center were more likely to underestimate agreement with their diagnoses compared to non - academic affiliated pathologists ( 40% versus 6% ) .
in addition , those who perceived that their colleagues consider them an expert in breast pathology were more likely to underestimate their agreement ( 41.2% versus 9.3% ; 0.029 ) ( table 1 ) .
the vast majority of participants indicated that , within the entire spectrum of breast pathology , the type of cases included in the sample sets were either often ( 51.7% ) or always ( 24.7% ) seen in their practice , and though not statistically significant , were able to fairly closely estimate their agreement with the specialist consensus diagnoses ( table 2 ) .
prior to the continuing medical education program , nearly 55% ( 54.9% ) of participants could not estimate whether they would over - interpret the cases or under - interpret them relative to the reference diagnosis .
nearly 80% ( 79.8% ) reported learning new information from this individualized web - based continuing medical education program as well as strategies they can apply to their own clinical practice ( table 2 ) .
figure 2 illustrates that participants were most likely to overestimate their performance for atypia and least likely to overestimate their performance for invasive cancer prior to the educational intervention .
figure 3 shows that following the continuing medical education program , participants were more able to recognize the gap in their estimated versus actual agreement .
twenty - two of 92 participants ( 23.9% ) indicated areas in their clinical practice they would change as a result of the continuing medical education program ( table 3 ) . among those who underestimated agreement with their diagnosis , threshold setting and information seeking
were equally described , while among those who overestimated agreement with their diagnoses , more careful review of cases was most frequently described .
this study is , to our knowledge , the first detailed analysis of the perceptions pathologists have regarding the gap in perceived versus actual diagnostic agreement in breast pathology interpretation .
the characteristics most likely associated with underestimating agreement with their diagnoses were affiliation with an academic medical center and their self - report of being perceived as an expert in breast pathology by colleagues .
it may be that pathologists affiliated with an academic medical center or who are perceived by their colleagues as experts in breast pathology by their peers are more aware of areas of poor diagnostic agreement in breast pathology and so have lower expectations for diagnostic agreement than those with other clinical experiences , especially in diagnostic categories like atypia .
it may also be that breast pathologists exposed to a high volume of consults , such as those who practice in tertiary care centers or other high volume centers , see a broader array of difficult cases , which results in less inherent confidence that they can predict the biologic behavior of the disease or be assured that their expert peers would arrive at the same diagnosis .
while research in visual interpretive practice has examined the learning curve post residency with radiologists ( 14 , 15 ) , this topic has not been studied with breast pathologists .
in addition , both prior radiology studies focused on learning curves at the beginning of independent practice , rather than among a broad range of physician ages .
we found that physicians were most likely to overestimate their diagnostic agreement rates with atypia and least likely to over or underestimate these rates for invasive cancer .
many studies have shown less interpretive variability in distinguishing between just two categories : benign and malignant ( 6 , 8) .
it appears to become much more challenging when pathologists must differentiate between atypia and ductal carcinoma in situ ( 68 ) .
given that this clinical conundrum is well documented , understanding how to reduce this variability could improve clinical practice .
our hope is that educational interventions , similar to the one we developed for this study , will result in improved clinical care .
maintenance of certification has stimulated similar novel educational programs for continued professional development , as gaps in professional practice are recognized .
the accreditation council for continuing medical education has adopted the agency for healthcare research and quality s definition of a gap in professional practice as 
the difference between health care processes or outcomes observed in practice , and those potentially achievable on the basis of current professional knowledge ( 16 ) . with changes in maintenance of certification occurring in subspecialties
, many groups are redesigning continuing medical education to examine gaps between actual and optimal practice .
for example , the american college of physicians is currently hosting an educational program on cardiovascular risk reduction ( 17 ) aimed at identifying and addressing gaps in knowledge and practice to improve the care of patients .
as with our study , it will be vitally important to determine whether such educational programs can improve both physician practice and patient outcomes so the changes in maintenance of certification can be linked to actual improvements .
our previously published work identified high diagnostic disagreement rates for atypia in breast pathology , and the current study s continuing medical education intervention identified a gap in pathologists perception of how frequently their diagnoses would agree with the specialist reference diagnoses of atypia .
this continuing medical education is novel because it focused on problematic diagnostic areas ( such as atypia ) that are less clearly defined than areas that are frequently tested with typical continuing medical education activities ( such as invasion versus no invasion ) .
our study is unique in that we studied the gap between perceived versus actual practice when interpreting test sets .
this is important because research shows that for physicians to change practice behaviors they must be motivated by an understanding of a need to change , which was an important focus of this study .
most studies focus on recognizing and rewarding performance that is of high quality ( 18 , 19 ) or report performance gaps at health system levels without detailing how individual physicians processed their own gaps in performance ( 2021 ) .
we found that after recognizing the gap in perceived diagnostic agreement , those who underestimated agreement with their diagnoses reported that setting different clinical thresholds for interpretation and seeking additional information would be steps they would undertake to improve performance .
this suggests that web - based learning loops composed of assessment followed by time - limited access to educational tools and visual diagnostic libraries may be useful in improving performance with a reference standard ; when the time limit expires , the participant would repeat the loop with another assessment .
this  training set  could help improve diagnostic concordance with a reference standard in actual practice .
physicians who overestimated their diagnostic agreement rates reported they would undertake more careful review as a way to improve .
physicians who closely estimated their diagnostic agreement rates most often reported intending to seek additional consultations , likely because of a good perception of how low diagnostic agreement is in breast pathology for diagnoses such as atypia .
these strategies might indeed improve performance if they were implemented , but understanding if this happened and what the impact might be was beyond the scope of this study .
a strength of this study is that a broad range of pathologists participated , including university - affiliated and private - practice pathologists , pathologists with various levels of expertise in breast pathology and pathologists with various total years in practice .
another strength of the study is that we oversampled challenging cases ( atypia ) in a effort to focus on interventions to address the most problematic areas of breast pathology .
limitations of the study include the fact that study was limited to cases in the study test sets .
in addition , the educational program necessarily used digital images of the glass slides to educate physicians about specific features of the slides that contributed to the reference diagnosis .
it is possible that the digital slide images were to some extent dissimilar from the actual glass slides , which could have influenced what pathologists learned from the program . in conclusion ,
our unique individualized web - based educational program helped pathologists to understand gaps in their perceived versus actual diagnostic agreement rates when their interpretations of breast biopsy cases were compared with a reference standard .
we found that pathologists can closely estimate their overall diagnostic agreement with a reference standard in breast pathology but in particularly problematic areas ( such as atypia ) , they tend to over - estimate agreement with their diagnosis .
pathologists who both over and under interpreted their diagnostic agreement rates could identify strategies that could help them improve their clinical practice .
 we examined how pathologists process their perceptions of how their interpretations on diagnoses for breast pathology cases agree with a reference standard . to accomplish this , we created an individualized self - directed continuing medical education program that showed pathologists interpreting breast specimens how their interpretations on a test set compared to a reference diagnosis developed by a consensus panel of experienced breast pathologists . after interpreting a test set of 60 cases , 
 92 participating pathologists were asked to estimate how their interpretations compared to the standard for benign without atypia , atypia , ductal carcinoma in situ and invasive cancer . 
 we then asked pathologists their thoughts about learning about differences in their perceptions compared to actual agreement . 
 overall , participants tended to overestimate their agreement with the reference standard , with a mean difference of 5.5% ( 75.9% actual agreement ; 81.4% estimated agreement ) , especially for atypia and were least likely to overestimate it for invasive breast cancer . 
 non - academic affiliated pathologists were more likely to more closely estimate their performance relative to academic affiliated pathologists ( 77.6% versus 48% ; p=0.001 ) , whereas participants affiliated with an academic medical center were more likely to underestimate agreement with their diagnoses compared to non - academic affiliated pathologists ( 40% versus 6% ) . 
 prior to the continuing medical education program , nearly 55% ( 54.9% ) of participants could not estimate whether they would over - interpret the cases or under - interpret them relative to the reference diagnosis . 
 nearly 80% ( 79.8% ) reported learning new information from this individualized web - based continuing medical education program , and 23.9% of pathologists identified strategies they would change their practice to improve.in conclusion , when evaluating breast pathology specimens , pathologists do a good job of estimating their diagnostic agreement with a reference standard , but for atypia cases , pathologists tend to overestimate diagnostic agreement . 
 many of these were able to identify ways to improve . 
Introduction
Methods
IRB Approval and Test Set Development
Physician Recruitment and Survey
Educational Program Development and Implementation
Data Analyses
Results
Discussion