the importance of evidence - based practice ( ebp ) in allied health is well documented in the literature.1,2 clinical decisions that are based on patients unique circumstances , sound clinical expertise , and the best available research evidence are known to deliver the best outcomes for patients and their families.35 allied health practitioners hold positive attitudes toward ebp and believe in the value of research evidence in informing their clinical decisions . however , applying research findings to clinical decisions is not a simple process and is often difficult to achieve .
one of the most commonly reported barriers to evidence uptake in allied health is the lack of knowledge of the ebp process and lack of skill in critically appraising research.68 teaching ebp is therefore an important step in promoting evidence - based clinical decision making .
allied health practitioners need to understand the principles of ebp before they can apply it .
early ebp educational programs include the development of clinical questions , literature searches , and critical appraisal.9 to evaluate the impact of such educational programs and document competence of individual practitioners , educators need objective and psychometrically sound instruments or assessment tools .
based on a review of the literature , the fresno test is the only available instrument that comprehensively assesses ebp competence across all relevant domains.10 the fresno test consists of two clinical scenarios and 12 short - answer questions that require respondents to formulate a focused question , identify the most appropriate research design that will address the question , show knowledge of electronic database searching , identify issues important for determining the relevance and validity of a research paper , and discuss the magnitude and importance of research findings.11 the test is scored by using a standardized grading rubric that describes explicit grading criteria . the fresno test has content validity , good - to - excellent interrater reliability for all questions , and excellent internal consistency.11 however , this tool focuses on assessing competence in medical students only , and therefore it can not be used across different health disciplines . in 2009 ,
mccluskey and bishop modified the fresno test to measure the change in ebp skills and knowledge of occupational therapists following exposure to an ebp workshop.12 new clinical scenarios ( ie , versions 1 and 2 ) were developed to suit rehabilitation professionals , such as physiotherapists and occupational therapists .
the 12 questions in the original fresno test were reduced to seven ( ie , questions 17 ) , removing questions about diagnosis and complex statistics ( ie , questions 812 ) .
similar to the original fresno test , the seven - item adapted fresno test ( aft ) measures the following : the ability to develop a focused clinical question using the pico ( population , intervention , comparison , and outcome ) format , the ability to develop a search strategy , the ability to interpret and critically appraise a research paper , and knowledge associated with understanding of the hierarchy of evidence and methodological biases in study designs , databases , and other sources of evidence and study designs .
the aft has been reported to have acceptable psychometric properties : interrater reliability ranged from good to excellent for individual items ( version 1 , intraclass correlation coefficient [ icc ] 0.800.96 ; version 2 , 0.680.94 ) and excellent for the total score ( version 1 , 0.96 ; version 2 , 0.91 ) ; acceptable internal consistency ( cronbach s  0.74 ) ; and responsive to change in novice learners.12 the current versions of the aft are limited to physiotherapists and occupational therapists , and new scenarios and scoring rubrics are required for other allied health disciplines .
therefore , the aim of this study was to examine the validity , interrater reliability , and internal consistency of aft versions developed for speech pathologists , social workers , and dieticians / nutritionists .
this study was approved by the human research ethics committee of the university of south australia and the ethics review board of the university of tasmania .
an expert panel consisting of four practitioners from each discipline was formed to content - validate the aft .
content validity refers to   how well the combined elements used to construct the instrument truly describe the conceptual domain of interest.13 the panel represented practitioners with more than 10 years of clinical experience and with previous exposure to ebp training or research .
the panel members were presented with the original fresno test and aft , and were asked to examine the questionnaire and comment on which questions should be included in the new versions for speech pathologists , social workers , and dieticians / nutritionists .
all members agreed that only questions adapted by aft should be included for these disciplines .
the scoring rubric of the aft was considered applicable to the new versions except for questions 1 (  write a focused clinical question for one scenario to help you organize a search of the literature  ) , 2 (  where might you find answers to these and other similar clinical questions ?
name as many possible sources of information as you can , not just the ones you think are good sources  ) , and 4 (  if you were to search for medline for original research to answer your question , describe the search strategy you might use  ) .
discipline - specific information was required to revise the scoring key for these questions . following consultation with the expert panel ,
a draft instrument including the clinical scenarios , questionnaire , and scoring rubric was prepared by the primary author .
the draft instrument was emailed to the experts for feedback on the clarity of the entire instrument and completeness of the scoring rubric .
the instrument and scoring rubric for each discipline were revised based on comments from the expert panel and returned to them for a final round of feedback .
the new aft versions were completed by ten speech pathologists , 16 social workers , and 12 dieticians / nutritionists who agreed to participate in a larger study aimed at examining the impact of a journal club on the ebp knowledge and skills of allied health professionals.14 they were asked to individually complete either a paper - and - pencil version or electronic version of the questionnaire at a time convenient for them .
less than half had previous training in research or ebp , and the majority had been in clinical practice for less than 10 years .
interrater reliability is the   degree to which measurements of the same phenomenon by different raters will yield the same results , or the consistency of results between raters.15 interrater reliability was calculated for individual items and the total aft score using iccs ( 2,1 ) and 95% confidence intervals . for interpretation of results , icc values of 0.80 indicate excellent reliability , values between 0.60 and 0.79 denote moderate reliability , and values < 0.60 mean questionable reliability.16 four individuals experienced in research and teaching ebp for allied health students served as raters for the study . before the study began , the raters reviewed and discussed the aft test , and collaboratively scored a sample test for each discipline .
they were then given a practice period , where they scored another set of sample tests , then compared and discussed their differences in scoring .
following discussion , the raters were instructed to score each test independently without conferring or comparing ratings .
initial examination of the interrater reliability showed poor reliability between raters for questions 2 , 4 , and 5 of all versions ( ie , aft for speech pathology , social work , and dietetics / nutrition ) and question 7 for social work and dietetics / nutrition .
this prompted the first author , who has experience in using the previous aft versions , to provide further training and discussion of the scoring procedure to the raters .
the training involved an explanation of the rating system , discussion of common rater errors , advice on process for decision making , and practice on interpreting the rubric .
reflects the coherence of the components of a scale or instrument.17 the internal consistency of the aft was examined using cronbach s .
an expert panel consisting of four practitioners from each discipline was formed to content - validate the aft .
content validity refers to   how well the combined elements used to construct the instrument truly describe the conceptual domain of interest.13 the panel represented practitioners with more than 10 years of clinical experience and with previous exposure to ebp training or research .
the panel members were presented with the original fresno test and aft , and were asked to examine the questionnaire and comment on which questions should be included in the new versions for speech pathologists , social workers , and dieticians / nutritionists .
all members agreed that only questions adapted by aft should be included for these disciplines .
the scoring rubric of the aft was considered applicable to the new versions except for questions 1 (  write a focused clinical question for one scenario to help you organize a search of the literature  ) , 2 (  where might you find answers to these and other similar clinical questions ?
name as many possible sources of information as you can , not just the ones you think are good sources  ) , and 4 (  if you were to search for medline for original research to answer your question , describe the search strategy you might use  ) . discipline - specific information was required to revise the scoring key for these questions . following consultation with the expert panel ,
a draft instrument including the clinical scenarios , questionnaire , and scoring rubric was prepared by the primary author .
the draft instrument was emailed to the experts for feedback on the clarity of the entire instrument and completeness of the scoring rubric .
the instrument and scoring rubric for each discipline were revised based on comments from the expert panel and returned to them for a final round of feedback .
the new aft versions were completed by ten speech pathologists , 16 social workers , and 12 dieticians / nutritionists who agreed to participate in a larger study aimed at examining the impact of a journal club on the ebp knowledge and skills of allied health professionals.14 they were asked to individually complete either a paper - and - pencil version or electronic version of the questionnaire at a time convenient for them .
less than half had previous training in research or ebp , and the majority had been in clinical practice for less than 10 years .
interrater reliability is the   degree to which measurements of the same phenomenon by different raters will yield the same results , or the consistency of results between raters.15 interrater reliability was calculated for individual items and the total aft score using iccs ( 2,1 ) and 95% confidence intervals . for interpretation of results , icc values of 0.80 indicate excellent reliability , values between 0.60 and 0.79 denote moderate reliability , and values < 0.60 mean questionable reliability.16 four individuals experienced in research and teaching ebp for allied health students served as raters for the study . before the study began , the raters reviewed and discussed the aft test , and collaboratively scored a sample test for each discipline .
they were then given a practice period , where they scored another set of sample tests , then compared and discussed their differences in scoring .
following discussion , the raters were instructed to score each test independently without conferring or comparing ratings .
initial examination of the interrater reliability showed poor reliability between raters for questions 2 , 4 , and 5 of all versions ( ie , aft for speech pathology , social work , and dietetics / nutrition ) and question 7 for social work and dietetics / nutrition .
this prompted the first author , who has experience in using the previous aft versions , to provide further training and discussion of the scoring procedure to the raters .
the training involved an explanation of the rating system , discussion of common rater errors , advice on process for decision making , and practice on interpreting the rubric .
internal consistency reflects the coherence of the components of a scale or instrument.17 the internal consistency of the aft was examined using cronbach s .
the content validity of the aft instrument was established through formal feedback from the expert panel .
the comments received were consistent across disciplines , and involved issues associated with the wording of the clinical scenarios .
no comments were made on the questionnaire itself ; however , additional possible answers were suggested for the scoring rubric .
for example , in question 1 , where respondents are asked to write a focused clinical question , the expert panel provided additional pico terms or synonyms .
some members of the panel suggested further sources of research information for question 2 , such as discipline - specific electronic databases , websites , and professional organizations . two new clinical scenarios and a revised scoring rubric were developed for each discipline .
a copy of the scoring rubric may be obtained from the primary author upon request .
the reliability among raters was excellent for questions 1 , 3 , and 6 across all disciplines , as shown in table 3 .
question 7 showed excellent reliability for speech pathology , but not for social work or dietetics / nutrition .
all other reliability coefficients increased to moderate or excellent levels following further training and discussion .
cronbach s  was 0.71 for speech pathology , 0.68 for social work , and 0.74 for dietetics / nutrition , indicating internal consistency was acceptable for all disciplines .
deletion of any of the items did not improve the internal consistency of the aft for any discipline .
the content validity of the aft instrument was established through formal feedback from the expert panel .
the comments received were consistent across disciplines , and involved issues associated with the wording of the clinical scenarios .
no comments were made on the questionnaire itself ; however , additional possible answers were suggested for the scoring rubric .
for example , in question 1 , where respondents are asked to write a focused clinical question , the expert panel provided additional pico terms or synonyms .
some members of the panel suggested further sources of research information for question 2 , such as discipline - specific electronic databases , websites , and professional organizations . two new clinical scenarios and a revised scoring rubric were developed for each discipline .
a copy of the scoring rubric may be obtained from the primary author upon request .
the reliability among raters was excellent for questions 1 , 3 , and 6 across all disciplines , as shown in table 3 .
question 7 showed excellent reliability for speech pathology , but not for social work or dietetics / nutrition .
all other reliability coefficients increased to moderate or excellent levels following further training and discussion .
cronbach s  was 0.71 for speech pathology , 0.68 for social work , and 0.74 for dietetics / nutrition , indicating internal consistency was acceptable for all disciplines .
deletion of any of the items did not improve the internal consistency of the aft for any discipline .
the results provide preliminary evidence of the psychometric integrity of the aft , and support its use in the assessment of ebp knowledge and skills of speech pathologists , social workers , and dieticians / nutritionists .
similar to the original aft , the new versions assess knowledge and skills of the key processes involved in ebp , including the development of clinical questions , searching for literature , critical appraisal , and interpretation of research findings .
the new aft has content validity , moderate - to - excellent reliability and acceptable internal consistency .
these results are consistent with the previously reported validity and reliability of the original fresno test11 and aft versions for rehabilitation professionals ( ie , occupational therapist and physiotherapist).12 the importance of ebp training in facilitating an evidence - based approach to clinical practice has been highlighted by a number of systematic reviews.1821 many of the training programs reported in these reviews relied on self - report data , which potentially reflect inaccuracies in actual knowledge.22 measuring the effectiveness of such training programs therefore requires objective and robust instruments to document changes in the competence of the individuals being trained . to the authors knowledge
, the aft is the only objective measure of ebp knowledge and skills that has been tested and applied in allied health .
mccluskey and bishop , who first reported about the validity and reliability of the aft , urged researchers to develop new clinical scenarios and modify the instrument to suit other health disciplines.12 the current study addressed this gap and provided researchers and educators an instrument to measure ebp skills and knowledge in speech pathologists , social workers , and dieticians / nutritionists .
the new versions of the aft were content - validated , and although the internal consistency of the different versions was slightly lower than the original aft , the cronbach s -values were still acceptable .
the reliability estimates for some of the items ( questions 2 , 4 , 5 , and 7 ) were questionable ; however , after further training , the iccs increased considerably , indicating moderate - to - excellent reliability of scores for these items .
this finding highlights the importance of providing training to raters as a strategy to improve interrater reliability .
rater training has been shown to increase consistency of scoring between raters.23 it emphasizes developing a common understanding among raters so they will apply the rating system as consistently as possible.24 this common understanding , also called 
frame of reference  , addresses the common sources of rater disagreements , which include lack of overlap among what is observed , discrepant interpretations of descriptor meanings , and personal beliefs or biases.24 however , research also suggests that even comprehensive training will not ensure rater agreement.25 studies have suggested that a rater s expertise may improve accuracy,23,26 which implies that rater characteristics are also an important consideration in ensuring consistency between raters .
reliability in examination scoring can be expected if the raters are highly knowledgeable in the domain in which ratings are made .
studies have found a relationship between rater expertise and rating accuracy , as well as the ability to differentiate between different domains in a rating scale.24,26 the raters involved in this study are experienced ebp educators and researchers , and these attributes could have contributed to the consistency in scoring .
because of their exposure to teaching , the raters may have already gained a wealth of experience in examination assessment , and could be expected to respond well to training .
it is therefore not surprising to find that following training in aft rating , the reliability estimates improved significantly for the previously questionable items .
based on the results of the current study , it appears that there are three important variables that can contribute to rater reliability : an explicit scoring criteria ( ie , scoring rubric ) , raters training , and raters professional experience . as with any study
first , the sample size may have been too small to produce sufficiently reliable results .
second , the expert panel was limited to four practitioners , which may not represent the collective set of views in the different professions .
third , the ability of the test to detect change following educational programs has not been tested . despite these limitations ,
the results of this study provide a valuable resource for ebp educators and researchers who require an objective instrument to measure knowledge and skills among social workers , speech pathologists , and dieticians / nutritionists .
the authors propose the use of aft in evaluating the ebp knowledge and skills of social workers , speech pathologists , and dieticians / nutritionists .
ebp educators and researchers should identify raters with experience in ebp teaching or those with previous ebp training , who should then receive training for aft scoring .
while the content validity , internal consistency , and reliability of the aft have been shown in this study , further research is required to establish its sensitivity to detect change in knowledge and skills following an educational intervention for dieticians , speech pathologists , and social workers .
 purposethe current versions of the adapted fresno test ( aft ) are limited to physiotherapists and occupational therapists , and new scenarios and scoring rubrics are required for other allied health disciplines . 
 the aim of this study was to examine the validity , reliability , and internal consistency of the aft developed for speech pathologists ( sps ) , social workers ( sws ) , and dieticians / nutritionists ( dns).materials and methodsan expert panel from each discipline was formed to content - validate the aft . 
 a draft instrument , including clinical scenarios , questionnaire , and scoring rubric , was developed . 
 the new versions were completed by ten sps , 16 sws , and 12 dns , and scored by four raters . 
 interrater reliability was calculated using intraclass correlation coefficients ( 2,1 ) for the individual aft items and the total score . 
 the internal consistency of the aft was examined using cronbach s .resultstwo new clinical scenarios and a revised scoring rubric were developed for each discipline . 
 the reliability among raters was excellent for questions 1 , 3 , and 6 across all disciplines . 
 question 7 showed excellent reliability for sps , but not for sws and dns . 
 all other reliability coefficients increased to moderate or excellent levels following training . 
 cronbach s  was 0.71 for sps , 0.68 for sws , and 0.74 for dns , indicating that internal consistency was acceptable for all disciplines.conclusionthere is preliminary evidence to show that aft is a valid and reliable tool for the assessment of evidence - based practice knowledge and skills of sps , sws , and dns . 
 further research is required to establish its sensitivity to detect change in knowledge and skills following an educational program . 
Introduction
Materials and methods
Development and content validation of AFT for speech pathology, social work, and dietetics/nutrition
Participants
Interrater reliability of the AFT
Internal consistency of the AFT
Results
Content validity of the AFT for speech pathologists, social workers, and dieticians/nutritionists
Interrater reliability of the AFT
Internal consistency of the AFT
Discussion
Conclusion