the world wide web has become an excellent resource for distribution of educational materials in an electronic format for independent study with computer - aided instruction .
students in the health sciences can utilize their own electronic devices with web browsers to access a growing number of web - based educational content .
though web - based educational materials may be popular with students as indicated on evaluations , are online resources used in the manner for which they were intended , and do they yield results that suggest students are achieving mastery of subject matter they have studied ? students evaluations and feedback sessions in the preclinical years at the author 's medical school have called for greater availability of practice examinations to prepare for  real  examinations , including both those in the pathology discipline taught at this school as well as the united states medical licensing examination ( usmle ) step i examination .
in particular , students have shown the greatest interest in access to practice examinations that incorporate imaging .
this study was performed to analyze the usage and results of web - based pathology practice examinations that were developed to assist students using formative assessments of their knowledge .
these practice examinations were developed to provide a freely available resource for review and self - assessment in major subject areas of pathology , but not to be so extensive or comprehensive that a large amount of student time would be required to complete them .
a web site with public internet access to a site entitled web path was developed by the author , starting in 1994 , with no direct funding throughout the site 's history .
this site provides content for pathology education , primarily using images with text descriptions arranged by subject , along with examination questions .
the site began receiving an average million web page  hits  per month at its inception , and access continued to increase in the 1990 's until average monthly usage ranged from 6.5 to 8.5 million hits / month , and that has remained stable over the past 15 years .
usage paralleled the academic calendar of northern hemisphere medical schools , with more  hits  in fall and spring .
web site analysis revealed that over 95% of accesses originated from dynamic internet protocol ( ip ) addresses of commercial internet service providers , so it is impossible to assess the location of users . on average , over half of users gained access to web path via referral from internet search engines .
the menu [ figure 2 ] for the examinations is accessed at the following url : http://library.med.utah.edu/webpath/exam/examidx.html the login window for the timed examinations is shown in figure 3 .
web path home page , with the link to the examinations highlighted by the red arrow examination web page main menu , with links to the nontimed general pathology and organ system pathology banks , and links to the timed review quizzes login popup window showing the data fields collected anonymously , before the start of each timed examination the author developed examination questions for the web site that were representative of content for chapters in the robbins and cotran pathologic basis of disease textbook that followed the usmle content outline .
all examination questions were developed in usmle style , and had 5 foils / question .
these patient vignette style questions integrated anatomical pathology with laboratory and physical examination findings [ figures 4 and 5 ] .
the examination templates were developed using javascript for client - side html documents accessible via a web browser .
a bank of 431 questions in 9 topic areas for general pathology emphasizing mechanisms of disease , and a bank of 509 questions in 11 topic areas covering organ system pathology , was placed onto the web path site for user - chosen access in a nontimed format , as shown in table 1 .
these questions provided feedback for each answer chosen by the user , along with a continuously compiled examination score using javascript with cookies . however , the scores were generated client side on the user 's computer and not stored on the web server .
sample nontimed examination format from the hematopathology examination , with the question stem in the lower frame , the feedback for the answer in the upper frame , and the cumulative score in the right frame sample timed examination format from the pulmonary subject examination .
the stem of this question provides clues to the diagnosis of restrictive lung disease , and the image shows the gross pathology of honeycomb lung , so foil e is the best choice the author developed a smaller bank of timed examinations to mimic a real testing environment .
each of the four multi subject review examinations had 30 questions , including 6 questions with images .
the topic areas for the timed examinations in order of their display on the web pages are shown in table 1 . after clicking on a menu item to open a timed examination ,
a popup window appeared on the user 's screen , shown in figure 3 , with fields for login that recorded via radio buttons choices for the health science field of study for the user , along with free text field for voluntary , self - reported typed entry of the higher educational affiliation by the user .
no e - mail address , ip address , or other identifier was collected , so the data remained anonymous and institutional review board exempt . after submitting this form ,
a sample timed format question is shown in figure 5 . following user submission and exit of an examination
, scoring occurred using a server - side common gateway interface program using a perl 5 program script . upon completion of the exam ,
the web server received the information using the html form postmethod and then passed the data to the perl script .
the captured data elements written in comma - delimited sequence to an ascii text file included : user 's field of study , user 's academic affiliation , total score , and chosen answer for each question . after a user submitted an examination for scoring , or the exam timed out , only those questions marked were scored .
the user was then automatically directed to a web page that provided the overall percentage score for correct answers for the examination taken , along with a key of correct answers for that examination . as in a real examination ,
the perl script generated a cumulative log file that recorded the individual logins to each examination for each line of the text file .
the log files for each examination accumulated following migration of the web site to a new web server and updating with revision of questions in june 2006 , until the log files were downloaded in january 2014 . during that period ,
the ascii text formatted log files were downloaded to the author 's computer and loaded into microsoft excel ( 2011 ) spreadsheet software ( microsoft corporation , redmond , washington ) for each examination .
the log files of anonymous user data were analyzed for total number of logins , number of logins with all questions completed , scores by question and health science field of study for each of the 21 examinations .
the 4 general review examinations [ table 2 ] had 31,639 accesses with completion of all questions , for an overall completion rate of 53.7% , and average score of 75.2% with standard deviation of 3.2 .
review examination 3 with the average farthest from the mean was not a significant outlier ( p > 0.05 ) by grubbs test to detect an outlier . in top to bottom web page menu order , review examination usage was 43.9% , 23.7% , 17.5% , and 14.9% of all accessions [ table 2 ] .
general review examination usage the 17 subject examinations [ table 3 ] had 103,028 completions , with overall completion rate 72.6% , and average score 74.0% with standard deviation of 4.2 .
the pulmonary subject examination with the average farthest from the mean was not a significant outlier ( p > 0.05 ) by grubbs test to detect an outlier .
the first three menu items on the subject examination web page accounted for 12.6% , 10.0% , and 8.2% of all completions , and the bottom three menu items accounted for no more than 2% each [ table 3 ] .
subject specific examination usage analysis of scores by user 's field of study [ table 4 ] showed the highest overall scores were obtained by md and do users , 75.3% for the four general review examinations and
users in other health sciences had overall lower averages , ranging from 54.6% to 75.4% for the general review examinations , and from 50.9% to 73.7% for the subject examinations .
none of the field of study categories was a significant outlier ( p > 0.05 ) by grubbs test to detect an outlier .
examination scores by user 's field of study for the four general review examinations , a score of 100% was achieved by 7% of all users , 21% of users achieved a score 90% , and 95% of users achieved 50% [ table 4 ] .
for the 17 subject examinations , a score of 100% was achieved by 20% of users overall , 37% of users achieved a score 90% , and 90% achieved a score 50% [ table 5 ] .
web - based curricular tools have become popular for teaching students since the early 1990 's .
the value of web - based formative examinations ( giving a score ) has been shown to positively influence student performance in curricular assessments that count for a grade .
web - based instruction can also promote approaches to deep learning ( seeking to understand underlying concepts ) that are more effective for students than surface learning ( focused on passing the next examination ) .
web - based examinations can provide resources to support distance learning , education beyond a single institution , and share expertise on a national scale .
the web path site was designed to provide a variety of pathologic material for review by health science students , but it can not replace a standard curriculum .
it serves as an example of what can be done to supplement pathology education with no direct funding for development and no direct cost to users .
the total time required for completion of the web path web site question banks , estimating 11.5 min / question , is up to 23 h for untimed questions and up to 6 h for the timed questions .
there are now numerous question banks available on the internet , both free and through subscription .
novice students are challenged to select material that may be of value to their education .
usefulness of educational web sites can be evaluated with a published rating method . the web page menu for this study that leads to the login screen for the examinations
is located four clicks away , including a popup login window [ figure 3 ] , from the main menu [ figure 1 ] of the web path web site , tending to discourage casual surfing .
there is no way from log analysis to determine how many persons with minimal or no prior education in pathology logged in and took the exams , but of total users , only 4.3% for the review examinations , and only 6.1% for the subject examinations , picked  other  as a field of study category , and their scores were well within the range of other users who indicated a specific field of study .
however , the login did not distinguish students from persons at higher levels of training , such as residents , or practice professionals .
the value of anonymous data for research , such as the findings presented here , comes from the large amount of aggregate data that can be collected from users anywhere on earth and over a long period of time and at no cost to users .
the major limitation is that the data can not be verified for individual users by location , course of study , scores obtained , or conditions of usage .
educational research studies recruiting student volunteers for more rigorous conditions of assessment and analysis generally obtain smaller subject numbers .
national organizations providing qualifying examinations for certification can generate extensive statistical analyses , but generally do not publish their findings , results are released only to users and their educational institutions , and the examinations are costly to users . in this study , limiting the data analysis to users completing the examinations served to increase validity of results . reliability was increased through aggregation of data over 9 years with an unchanging set of 290 questions .
does posting web - based educational tools insure usage to justify the time needed to develop them ? in this study , the completion rates indicated that the shorter the exercise , the more likely the user would complete it .
the general review exams were closer to mimicking a  real  timed exam , but completion rates were even lower , 53.7% overall , or little more than half the accessions .
users may have been accessing the examinations only to assess their potential content and possible educational value , planning to return later .
though completion rates were below 100% in this study , the scores achieved on these web - based examinations suggest that some users completing the examinations had sufficient preparation to make use of the examinations to support their pathology education .
a high score 90% was achieved by 21% of users of review examinations and 37% of users for the subject examinations [ table 5 ] .
< 50% suggests that few users were just randomly marking answers [ table 5 ] .
given that a correct answer key was returned to users following submission of the examination for scoring , it is possible that some users may have retaken an examination one or more times to achieve a higher score , and that skewed the average higher .
however , these examinations are not  high - stakes  for a grade , so there is no tangible reward to the user for obtaining a higher score . if users are striving for a higher score , then that contributes to their education and fulfills a goal of the web site .
the variation in scores obtained per examination is primarily due to the nature of the mix of questions on these relatively short examinations .
one or two more difficult questions on a 10 questions examination will reduce the average scores achieved [ table 3 ] . however , none of the examinations was a statistically significant outlier .
the most difficult of the 290 total questions , scoring at 38.8% overall , in the cardiovascular subject examination had an image showing a dense collagenous scar with remote myocardial infarction , and prompted for the most likely current correlate , which was congestive heart failure , but 37.4% of users picked the foil for troponin elevation , reflective of an acute coronary syndrome . however , the foils were nonhomogeneous .
the author has used the information from log file analysis to revise the most difficult questions . in this study ,
to some extent this was a self - fulfilling prophecy , since the order of examinations in the menu , at least for the subject examinations , was arranged by the author according to past experience of subject popularity in a prior study of this web site .
similar internet user behavior has been observed even for the web site of the college of american pathologists ( cap ) , where a web page for complex cancer protocols , presumably of interest only to pathologists , recorded the highest number of  hits  for the adrenal protocol , even though this is an obscure site for malignancies ( personal communication ) .
the protocols were arranged alphabetically on that cap web page , with adrenal first . based upon user behavior
observed in this study , developers of web pages should place items of greatest interest or impact at the top of a menu .
designers of computer - aided instruction should bear in mind that , the longer the exercise , fewer users will complete it in the manner intended .
web page authors can evaluate whether developing more content is worth the additional effort , given a declining rate of usage for more items at the bottom of a menu .
providing more exercises of a similar nature does not guarantee equivalent usage at the web site .
would whole - slide imaging ( wsi ) lend itself to this kind of examination format ?
high - stakes timed examinations for students generally provide additional time , such as 1.5 min / question with imaging , and less for text - only questions .
however , use of wsi would require even more time , and reduce the total number of questions for examinations limited to a specific time period .
students in health science educational programs have a heavy curricular load , and pathology is just one of many subject areas that require testing .
guided instruction with a small number of labeled pathologic images fits within the time available for group learning exercises , such as laboratory sessions , in most curricula .
novice students are used to viewing video screens , and are more likely to tolerate such a medium than the multi headed microscope driven by a pathologist moving the slide at so rapid a rate that motion sickness develops in some students . however
, there is no practical way to test on wsi with large class sizes , and so students can safely ignore it .
though wsi has been applied at the medical student level , it is more effectively suited to training programs in pathology , such as residencies and fellowships .
wsi has been evaluated for testing pathology residents taking specialty board examinations administered by the american board of pathology .
this study showed that web - based examination completion rates were higher for shorter examinations in a single subject area .
usage was higher for shorter 10questions examinations , and for examinations listed closer to the top of the web page menu .
users identifying themselves as md / do scored higher than other users , averaging 75% , but this was only slightly higher than the overall average for all users , regardless of their voluntarily stated health science field of study .
scores 90% were achieved by 21% of users for review examinations , and by 37% of users for subject examinations and suggest that there were serious users completing the examinations who had sufficient preparation to make use of the examinations to support their pathology education . 

examinations of shorter length have higher completion ratesusage is highest for items nearer the top of a web page menudeveloping more examinations does not guarantee their equivalent usagea cohort of users had scores 90% , validating the effort in web site development . 
 
examinations of shorter length have higher completion rates usage is highest for items nearer the top of a web page menu developing more examinations does not guarantee their equivalent usage a cohort of users had scores 90% , validating the effort in web site development .
 context : general and subject specific practice examinations for students in health sciences studying pathology were placed onto a free public internet web site entitled web path and were accessed four clicks from the home web site menu.subjects and methods : multiple choice questions were coded into . html files with javascript functions for web browser viewing in a timed format . a perl programming language script with common gateway interface for web page forms scored examinations and placed results into a log file on an internet computer server . 
 the four general review examinations of 30 questions each could be completed in up to 30 min . 
 the 17 subject specific examinations of 10 questions each with accompanying images could be completed in up to 15 min each . 
 the results of scores and user educational field of study from log files were compiled from june 2006 to january 2014.results:the four general review examinations had 31,639 accesses with completion of all questions , for a completion rate of 54% and average score of 75% . 
 a score of 100% was achieved by 7% of users , 90% by 21% , and 50% score by 95% of users . in top to bottom web page menu order , review examination usage was 44% , 24% , 17% , and 15% of all accessions . 
 the 17 subject specific examinations had 103,028 completions , with completion rate 73% and average score 74% . 
 scoring at 100% was 20% overall , 90% by 37% , and 50% score by 90% of users . 
 the first three menu items on the web page accounted for 12.6% , 10.0% , and 8.2% of all completions , and the bottom three accounted for no more than 2.2% each.conclusions:completion rates were higher for shorter 10 questions subject examinations . 
 users identifying themselves as md / do scored higher than other users , averaging 75% . 
 usage was higher for examinations at the top of the web page menu . 
 scores achieved suggest that a cohort of serious users fully completing the examinations had sufficient preparation to use them to support their pathology education . 
INTRODUCTION
SUBJECTS AND METHODS
RESULTS
DISCUSSION
CONCLUSION