there are standards for content used in the creation of these checklists as well as formatting .
more recent studies have shown that specific formatting changes to the college of american pathologists ( cap ) cancer protocols significantly improve the completeness of the report , the clerical error rate ( as measured by amendments ) of synoptic reports is associated with the number of required data elements ( rdes ) , and specific formats of the report itself are associated with user preference and speed of data extraction by users .
synoptic  report format as well as specific required elements within the synoptic report for all primary resections of specific diagnoses but has no requirement concerning how that synoptic report is generated . written paper protocols and an electronic product that works through a pathologist 's laboratory information system ( electronic cancer checklist [ ecc ] ) are available from cap .
nevertheless , data concerning the impact of these different methods of generating a synoptic report on the quality of the synoptic report are limited .
in addition , tools have been developed to create  structured data  for registries and other interested parties from these synoptic reports .
this  structured data  consists of having the data elements as discrete elements in a true database ( i.e. , sql , etc . , ) which most commonly is achieved by creating a data file ( i.e. , comma separated value , tab delimited , or excel file for examples ) with discrete data elements rather than a free text narrative text file such as  word  ( most surgical pathology reports ) .
synoptic reporting can be implemented with different features ranging from simply including a table within the free text narrative report to fully structured reporting with binding to other databases including snomed and other databases .
however , data comparing the completeness or accuracy of the subsequently extracted and created structured database for any implementation of synoptic reporting is quite limited . to assess this ,
we compared four different types of synoptic report generating methods , including a web - based method , across a wide range of different quality measures .
from 2004 to 2016 , four different methods were used to generate synoptic reports in our hospital , [ table 1 ] .
all reports were dictated by the pathologists , typed by secretaries , and subsequently edited by the pathologists before sign out .
initially , pathologists used the checklists directly from the cap website ( cap format protocol ) .
next , these checklists were edited as detailed previously ( removal of all optional elements , numbering all elements , and consistent formatting ) ( edited cap protocol ) .
subsequently , word - based macros were made for the secretaries that included all rde headers and into which the secretaries only had to type the response for each header .
finally , a web - based product was made and placed on our hospital system intranet [ figure 1 ] .
this consisted of a web page connected to a json file that contained all the elements required by cap for creating a synoptic report .
this product listed all rde headers and allowed the pathologists or secretary to simply select from a list of the most common responses or free text a response if the desired response was not there . in all cases ,
the pathologist dictated the response to select from looking at the website , and the secretary subsequently went through the website and selected the items .
the website then generated a rich text formatted table - based synoptic report that was then  cut and pasted  directly into our surgical pathology report ( powerpath ) [ figure 2 ] .
performance of four different synoptic report generating methods screen shot of the web based protocol for colon carcinoma screen shot of the synoptic report generated by the web based protocol for colon carcinoma the web page was designed by two of the authors ( mar , sar ) so that only required ( by either cap or our clinicians ) data elements were included .
each question had to be answered , and the pathologists could only select one answer ( no multi - answer questions ) , although he or she was allowed to free text anything they like if they could not find the response they wished for , and  not applicable  was the appropriate response in some cases .
the default response for each question was a flag that read  you missed this item  that the report automatically generated if any item was missing from the report . in this way , if a pathologist skipped an item , the secretary could leave this text in place and still generate a report , and the pathologist could see and fix this response during the editing phase . there were also notes reminding the pathologists which ancillary studies needed to be ordered when associated with an rde in the synoptic report .
these notes appeared on the web page that the pathologists used to create the synoptic report but did not appear on the final synoptic report itself .
for the purposes of this study , we tracked how often the pathologist ordered kras studies on all colon carcinoma cases that were either stage t4 or n1 ( as per the desires of our clinicians ) before and after sign out as a measure of whether they remembered to order this ancillary study or had to go back and order it later .
finally , at the beginning of each rde in the report , the web page placed a pipe  |  specifically designed to improve free text extraction [ figure 3 ] . as a result , all rdes could be easily searched for using a free text search because they all began with the pipe 
multiline responses had no effect on this since the information was in a rich text format table that placed all information related to the data element before any information related to the response .
example of a report used to test the speed and accuracy of information retrieval containing the embedded flag ( pipe ) all cases with a tumor summary were reviewed by the tumor registry for completeness .
data on completeness were collected by the staff of the tumor registry and reported back to the pathologists on a monthly basis .
the definition of completeness was all items listed as required by the cap during the appropriate time period . as those requirements changed over time , so did the definition of completeness .
however , during this study , all methods were updated in a timely manner , and hence that there were no incomplete cases identified because of a new requirement that was not listed in the method .
however , during the 1 month of the web - based report , a single data item ( specimen integrity ) was left off the web - based protocol for endometrium . as a result ,
this omission was subsequently fixed , and this was not included in our results for completeness for the web - based option since the pathologists did correctly include all items that were actually included in the web - based protocol during this time .
only amendments that related to the tumor summary and were not based on additional clinical or pathologic information were included .
specifically , we were trying to track clerical errors such as spelling mistakes and features that did not match that were supposed to ( i.e. , the stage did not match the reported extent ) .
structured data were extracted from the synoptic report section of all surgical pathology reports using free text searches and regular expressions . before the inclusion of the pipe 
|  items were identified using a set of standard regular expression searches looking for the text of the rde . only breast carcinoma cases were extracted since the search required the use of specific headers to identify information . with the use of the pipe
, searches were based on identifying the pipes and the corresponding colons (  :  ) and all synoptic reports were included since the data were extracted regardless of which headers were included .
the data were subsequently collected in tab delimited ( txt ) file , and these results reviewed and compared .
numeric data were extracted from these strings in a subset of cases , but this conversion was not included as part of the definition of a complete report .
successful data extraction consisted of the inclusion of the correct data item and response in the txt file based on manual comparison with the original report . to test whether our embedded flag affected the end users ability to quickly and accurately extract information from a surgical pathology report ,
specifically , the participant is shown a specific phrase that may or may not be in a synoptic report .
when the user presses  enter  the synoptic report appears on the screen , and the timer starts .
the user then examines the report to determine if the phrase is or is not present .
if it is present the user types the number  2 ,  if it is not they type  1 ,  and then press  enter .  the timer stops when  enter  is pressed .
the program automatically records the time and whether the answer was correct , and this data are then transferred to a comma - separated values ( csvs ) file for further analysis .
we constructed our synoptic report for this test from an invasive breast carcinoma using the tumor protocol from cap .
all elements were identical except for changes in the reporting of focality , nipple involvement , skin involvement , and lymphovascular invasion .
the formats were tested in a quiz that contained 32 total questions half of which had the pipe on the left side of the report [ figure 3 ] and half of which did not [ figure 4 ] .
all questions were presented in random order . at the end of the quiz , each participant was asked whether they felt the pipe interfered with their reading of the report .
example of a report used to test the speed and accuracy of information retrieval without the embedded flag ( pipe ) twenty - six participants completed the quiz .
they were all nonpathologists and included , six cancer registrars , 15 medical personnel ( 4 md , 11 non - mds ) , and five nonmedical personnel ( administrative assistants , other professionals ) .
we specifically excluded pathologists from this testing , since we wanted to measure the performance of a user other than a pathologist . to allow comparison between these uses , times were normalized to the mean of the standard format for each user . as a result , the normalized time for the format without the pipe was the control with a normalized time of one , and the time for the pipe format was in comparison with that time .
the results for these are reported as mean  standard deviation ( no units since they are normalized ) .
statistical analysis was performed using a fisher 's exact test for categorical data and a student 's t - test for continuous data .
use of the printed tumor protocols directly from the cap website had the lowest completeness ( 84% ) and highest amendment ( 1.8% ) rates .
reformatting these protocols was associated with higher completeness ( 94% , p < 0.001 ) and reduced amendment ( 1% , p = 0.20 ) rates .
extraction failed most often due to the headers for the presence of ductal carcinoma in situ ( dcis ) and the margin status of dcis not being consistently unique ( as copied from the cap protocol directly , since the fact that one of the elements is related to margin status is located well away from the actual line for dcis ) .
word - based macros improved completeness ( 98% vs. 94% , p < 0.001 ) but not amendment rates ( 1.5% ) .
in contrast , a web - based product with a reminder flag when items were missing , an embedded flag for data extraction , and a reminder to order kras when appropriate resulted in improved completeness ( 100% , p = 0.005 ) , amendment rates ( 0.3% , p = 0.03 ) , kras ordering before sign out ( 100% , p = 0.23 ) , and structured data extraction ( 100% p < 0.001 ) .
the contents of the data extraction was compared to that of the original report on a subset of cases ( 100 ) and found to be 100% accurate .
there were a total of 832 responses for the quiz , half with the pipe and half without .
the speed with which readers were able to answer the questions was not significantly different with the pipe than without it ( 0.98 + 0.66 vs. 1.00 + 0.49 , p = 0.65 ) , and both formats were 100% accurate ( p = 1.00 ) .
all readers agreed that after initially being shown the pipe , they ignored it when answering the questions in the test .
the results of this study consist of a range of quality performance measures comparing a variety of different methods to generate synoptic reports .
the goal of these quality performance measures is to ensure that a complete and accurate report is generated that can easily be converted into a structured data file and does not interfere with the ability of the reader to accurately and rapidly extract information from the report .
first , consistent formatting of the checklists makes it easier to generate a complete report .
the completeness rates for all of the methods examined except the cap format protocols , all of which have a consistent format , are well above 90% , which is a level of performance most studies have failed to achieve .
second , selection of responses from a list rather than free text typing reduces errors ( amendments ) .
third , reminders or warning flags to identify missing items or additional studies that need to be performed improves the completeness of the report , and when these reminders are appropriately designed may reduce the need for force functions to ensure completeness .
finally , embedding a flag into the synoptic report within the surgical pathology report itself made free text extraction of data elements from the surgical pathology report into a structured data file easy and highly accurate without impacting the reader 's ability to extract information from the report .
, it appeared that the secretaries were not sure what to do when a pathologist skipped an item in the macro in their dictation . in some cases
, they assumed that it was skipped on purpose and deleted that entry from the report .
additional training may be of value in further improving the completeness of reports with the use of macros .
however , there was no reduction in the amendment rate with the use of word macros . on review
, we noted that the majority of amendments were in the response rather than the header part of the report , and these macros did not change how this section of the report was created ; secretaries still had to type in the response even when using a macro .
in contrast , the web - based option reduced the number of responses that had to be typed , and this may have led to a reduction in the amendment rate . while it is possible to create drop - down boxes in word that might have the same result , we were unable to make these compatible with our current laboratory information system .
in contrast , the web - based protocol was associated with a very high level of accuracy and completeness , including completeness of ordering additional studies , a measure that has rarely been examined in previous studies .
first is that it was simply easier for both the pathologists and secretaries to use than other methods ( improved  usability  ) .
for both the pathologists and secretaries , the web - based method is setup with very simple rules so that each question must be answered and always requires a single response .
there is very little thinking involved concerning optional questions or multiple select items that may distract the user from the task at hand .
in addition , when reading from a computer ( typically but not exclusively internet explorer in this study ) the browser scrolled through the questions in a much more uniform and consistent fashion than microsoft word scrolled through word - based templates . in a word , the checklist often scrolled either too slowly or jumped from page to page , making it more likely for the pathologist to miss an item , and possibly distracting them from dictating the correct response . for the secretaries ,
there was significantly less free text entry , and it was more difficult for them to delete an item , and the flag made it clear that they were supposed to flag a missing item rather than simply delete it .
in addition , unlike word macros , it was also possible to embed instructions and reminders for the pathologists in the website without these appearing on the subsequent report .
our data clearly show that these reminders improved the consistent ordering of appropriate ancillary studies , without the need for force functions .
this study shows that a structured data file (  structured data  ) can be extracted from free text synoptic reports and used to create a structured data set in a conventional database , but the success depends on the way those reports are structured .
 structured data  is a broad and generic term , and there are many types of  structured data  not all of which may be of value in creating a structured data file or data in a true database system .
indeed , a synoptic report is required by the cap to have data in a particular structure (  structured data  ) composed of data arranged in a tabular form , but this structure is of very little value for extracting a structured data file ( i.e. , csv , txt , excel files ) for entering into a true database which is really what most authors mean when they say  structured data  ( although limited data suggest that this structure likely improves the speed with which readers can extract information ) .
the success of our free text searches was highly dependent on embedding an alternative  structure  into the synoptic report ( beyond that stipulated by the cap ) in the surgical pathology report itself , although this  structured  data are not the same as a data file .
when we did not embed this structure ( straight free text searching using regular expressions ) , the algorithm to extract the data was complex and completely type specific since we had to search on the specific headers themselves . as a result
, we were only able to develop an algorithm that worked for one tumor site ( breast ) , and this method was only modestly reliable .
in contrast , with our embedded structure of pipes , a single simple algorithm allowed extraction of 100% of data from all tumor sites .
thus , we believe that embedding an appropriate structure into the synoptic report can significantly improve the success of extracting a structured data file from that report using free text searches . we have also shown that the embedded structure that is used in the synoptic report does not interfere with the reader 's ability to extract information .
in addition , the structure that we employ ( pipes  |  ) to extract a structured data file using text - based searches is not dependent on the tabular format that is required by the cap .
one could easily list one or more items on a line , or even embed these pipes within narrative text , and still our system would be able to extract the data into a structured data file .
we believe it can also easily be extended to data within an addendum , for example , ancillary studies that may take some time to come back .
in addition , any cases with amendments can be re - extracted and a corrected structured data file created .
while other methods may be able to re - extract data from an amended report where the synoptic report itself is edited and fixed , most other report methods can only extract data from the synoptic report area , and can not extract any data from other areas of the report ( addendums , amendments where the new information is in a separate amended area , etc . ) .
finally , as we have shown the use of a pipe as we describe is in no way limited to the website we use , and any interested laboratory could use this technique regardless of how their synoptic reports are generated . while we compared the performance of four different methods in this study ,
cap also offers the ecc product through a variety of different laboratory information system vendors , but this product was not available to us to compare .
we would imagine that in some ways the performance of this product would be similar to that of our web - based product .
since the ecc typically contains a force function to ensure that all elements are included , one would expect the reports that are generated to be complete by necessity .
in addition , the ecc by design generates a structured data report from the case just as our web - based product does .
however , since our method ensures completeness and allowed extraction of virtually 100% of the data , it would appear difficult for the ecc to show an advantage in either of these two features
. nevertheless , there are important differences between the web - based product we used in this study and the ecc . stylistically , the two programs are very different .
the web page is designed to only include required questions ( including questions required by our clinicians ) so all questions must be answered and all question are single question , single answer questions . as such it is as simple a synoptic report as we can make and requires as few  clicks  as possible on the part of the pathologist .
in addition , we allow free text answers to every single question ( although the majority of responses were not free text ) allowing the user a very wide range of answers .
the downside to this is that the final report may include more questions that are  not applicable  than other types of programs do , though there is some evidence that this may , in fact , make the reports easier to read .
it contains single answer questions , multi - answer questions , conditional questions all of which are multi - question answers ( the user needs to answer more than one question to answer a single question that appears in the report ) and optional questions and answers . as such there are fewer  not applicable  responses in the final report , but it takes more questions and more  clicks  on the part of the pathologist to create it . in addition , the responses to some questions are restricted to specific types ( distances must be a number , regardless of the complexity of a particular case ) . as such if a pathologist is uncomfortable answering a question in the way that the program is structured , the only real choice is to state that the answer can not be determined .
additional testing of different formats to determine which ones are preferred by the end users may be appropriate .
perhaps more importantly , the ecc generates its structured data at the time the data is entered , whereas our web based product extracts this data from the report itself after it is signed out . as a result while we can be sure that the data we extract exactly matches the data in the final surgical pathology report ( and we have confirmed this in a subset of our cases ) , the ecc may not always be able to do this ( since the surgical report may be altered after the structured data is extracted ) , depending on the exact design used by an individual vendor .
the cap laboratory accreditation program already has standards to ensure the accuracy of all electronic interfaces .
whether confirming the accuracy of the data that is extracted by these programs might be an appropriate topic for standards in this program might be worth considering .
in addition , the ecc can only generate data from the synoptic report itself , and it can not extract data from addendums or any other field of the report . the only way to extract this data for any purpose ( research , tumor registries , etc .
however , we already know that error rates as measured by amendments are associated with the number of rdes in a synoptic report .
we also know that the amount of text in the report is associated with the speed with which readers of the report can extract information .
thus , this practice may be both reducing the accuracy of the synoptic report and its utility to convey information to the clinician .
in contrast , our web - based product can extract data from any section of the report , including notes , addendums , or even the gross , as long as our embedded structure is present . as a result , it is possible to have multiple  synoptic reports  within a single surgical pathology report , each designed for the needs of a different set of users . on the other hand , the ecc may have additional functionality that our website may not yet offer .
it is possible that the ecc in addition to extracting the data provides additional tools to allow that data to be linked to other data sets such as snomed codes .
the use of our website led to the timely ordering of the most appropriate ancillary studies every time before the cases was signed out .
in addition , while not the subject of our study , during this study the ancillary studies that our clinicians wanted for a number of different tumors changed several times as they responded to the very rapidly changing world of  precision  medicine . given this rapid pace of change
, it is becoming harder and harder for pathologists and the laboratories that they work in to ensure that the most up to date and appropriate tests are being ordered in every case .
we have found the ability to embed notes reminding the pathologists about which tests to order in which situation invaluable to the success of our practice .
since the website was designed so that changes could easily be made to accommodate these differences in practice , we were able to make these changes for all pathologists in all sites in a matter of minutes from a single source , without going through our laboratory information system vendor and without having to make changes to multiple different source documents that pathologists and secretaries were using .
such flexibility may be of value as pathologists try to keep pace with the rapidly evolving field of  precision  medicine .
first , we used the opinion of the people in the tumor registry as a gold standard for completeness .
in a few cases , there were disagreements about whether the report was complete or not .
furthermore , this study takes place in a busy community hospital with general pathologists . it may be possible that different results could be obtained by specialists or in practices that are less busy . perhaps most importantly
, we can not exclude that at least some of the improvement in performance is related to increasing use and experience over time and not entirely related to the methods used for synoptic report generation .
however , the sudden improvement in performance with the web - based product to essentially perfect performance after over a decade of creating synoptic reports by other methods without such success would suggest that increased practice and experience are unlikely to be the sole cause of this improvement in performance .
completeness , amendment rates , ancillary test ordering rates , and data extraction rates vary significantly with the method used to construct the synoptic report .
our web - based protocol appears to be very competitive with most other synoptic report generating methods .
 context : the college of american pathologists ( cap ) requires synoptic reporting of all tumor excisions.objective:to compare the performance of different methods of generating synoptic reports.methods:completeness , amendment rates , rate of timely ordering of ancillary studies ( kras in t4/n1 colon carcinoma ) , and structured data file extraction were compared for four different synoptic report generating methods.results:use of the printed tumor protocols directly from the cap website had the lowest completeness ( 84% ) and highest amendment ( 1.8% ) rates . 
 reformatting these protocols was associated with higher completeness ( 94% , p < 0.001 ) and reduced amendment ( 1% , p = 0.20 ) rates . 
 extraction into a structured data file was successful 93% of the time . 
 word - based macros improved completeness ( 98% vs. 94% , p < 0.001 ) but not amendment rates ( 1.5% ) . 
 kras was ordered before sign out 89% of the time . 
 in contrast , a web - based product with a reminder flag when items were missing , an embedded flag for data extraction , and a reminder to order kras when appropriate resulted in improved completeness ( 100% , p = 0.005 ) , amendment rates ( 0.3% , p = 0.03 ) , kras ordering before sign out ( 100% , p = 0.23 ) , and structured data extraction ( 100% , p < 
 0.001 ) without reducing the speed ( p = 0.34 ) or accuracy ( p = 1.00 ) of data extraction by the reader.conclusion:completeness , amendment rates , ancillary test ordering rates , and data extraction rates vary significantly with the method used to construct the synoptic report . 
 a web - based method compares favorably with all other methods examined and does not reduce reader usability . 
I
M
R
D
C
Financial support and sponsorship
Conflicts of interest