An evaluation instrument or assessment measure such as a test or quiz is said to have evidence of validity if we can make inferences about a specific group of people or specific purposes from the results. What this means is that if a particular computational thinking test
An evaluation instrument or assessment measure such as a test or quiz is said to have evidence of validity if we can make inferences about a specific group of people or specific purposes from the results. What this means is that if a particular computational thinking test is designed, tested, and validated with 7th grade students, it does not mean automatically that it would also be valid for high school students. Without additional testing for validity for the high school student population, we cannot make that jump. The specific sub-group of the population used to validate the instrument may also be important—for instance, an instrument measuring motivation to learn CS among 7th grade students who are primarily White may not be valid for use with 7th grade students who are primarily African American.
The same thing applies to specific purposes from the results. If an instrument is designed, tested, and measured for validity with 7th grade students in order to measure self-efficacy with respect to learning programming, the evidence of validity may differ if the same group of students were given the instrument for measuring self-efficacy with respect to learning science. There may be correlations, but without further investigation of the instrument, automatic assumptions about the evidence of validity in the other subject (science) cannot be firmly made.
Face validity and criterion validity are the most commonly used forms of testing for validity in evaluation instruments for education. But there are many options to consider. Validity is a bit more subjective than reliability and there is no one pure method of “proving” validity–we can only gather evidence of validity. You can obtain deeper levels of evidence by evaluating it through different measures (described below), but you can only do so much with limited resources. Researchers are not typically expected to go to extraordinary lengths to demonstrate evidence of validity–however, those researchers with more extensive time and resources may.
Face validity (or Content-related Evidence)
Face validity is simply that–on its surface (or face) as reviewed by content specialists, are the items and questions on the instrument appropriate? An instrument that “passes” face validity would meet each of the following criteria, as defined by the experts, if when taken as a whole the items on the instrument are:
- Representative of the sample of the content being assessed,
- In a format appropriate for using the instrument, and
- Designed appropriately for the target population.
What this means is that if an instrument is measuring cognitive load of 4th grade students learning computational thinking through the use of a tool such as Scratch, the content represents a variety of questions designed to measure cognitive load across the targeted goals (such as variable assignments and mathematical expressions). It may also have images of Scratch blocks to contextualize the questions on the instrument. Wording of questions and the appropriateness of the answers (perhaps multiple choice) will all be designed at the 4th grade level.
Measuring face validity can be done by the researchers creating the survey. However, given the inherent biases of experts when evaluating their own work, this carries more meaning if the instrument is evaluated by other experts whenever possible.
Another method of assessing the face validity of an instrument involves conducting interviews with individuals who are similar to those whom you are researching. By asking these individuals to review the items on your survey or interview protocol and answer them as if they were participating, they can give an indication of which items make sense to them and which are worded poorly or may be measuring something other than what the investigator believed.
Criterion-Related (or Concrete) Validity
Criterion related validity evaluates to what extent the instrument or constructs in the instrument predict a variable that is designated as a criterion—or its outcome. Concurrent validity and predictive validity are forms of criterion validity. This form of validity is related to external validity, discussed in the next section.
- Predictive Validity – This measures how likely the instrument measures a variable that can be used to predict a future related variable. For example, measuring the interest of 11th grade students in computer science careers may be used to predict whether or not it can be used to determine whether those students will pursue computer science as a major in college.
- Concurrent Validity – This concrete validity measure defines how well the instrument measures against existing instruments/tests. In this case, the results of using the instrument may be compared against similar instruments in related fields or in the same field or against similar instruments targeting other population groups. For example, if you are measuring attitudes of students towards computers in your instrument, then you may want to, concurrently, or at the same time, administer a second instrument that has already been determined to measure student attitudes toward computers. If the results of your instrument’s measures are statistically the same or better than the other instrument’s results, then your instrument has met concurrent validity.
Internal and External Validity
Internal validity is the extent to which results of the instrument can be generalized in the same population group and context. External validity, on the other hand, is the extent to which the results of the instrument can be general