statements about validity from published tests:
1. “You can help ensure that the test will be valid and equitable for all students.” (From an examiner’s manual for a statewide assessment program, 2005)
2. “Evidence of test validity … should be made publicly available.” (From a major publication of a prominent testing organization, 2002)
3. “In the assessment realm, this is referred to as the validity of the test.” (From an introductory assessment textbook, 2005)
4. “[Test name] has proven itself in use for more than 50 years as a … valid test.” (From the website of a prominent test publisher, 2005)
5. “Such efforts represent the cornerstone of test validity.” (From the technical manual of a prominent achievement test, 2003). (Frisbie, 2005, p. 22)
Challenges like that presented by Frisbie notwithstanding, the instrumental perspective still dominates.
Criterion-related validity, construct validity, and content validity contain certain requirements if a test is deemed valid from the instrumental perspective. To establish criterion-related validity for an assessment from the instrumental perspective, a researcher typically computes a correlation coefficient between the newly developed test and some other assessment considered to already be a valid measure of the topic. This second assessment is referred to as the criterion measure; hence the term criterion-related validity. A test is considered valid for any criterion it predicts accurately (Guilford, 1946).
The major problem with criterion-related validity is that it is difficult in some cases to identify an appropriate criterion measure. Citing the work of Roderick M. Chisholm (1973), Hathcoat (2013) exemplifies the criterion problem using the ability to determine the quality of apples:
If we wish to identify apple quality then we need a criterion to distinguish “good” apples from “bad” apples. We may choose to sort apples into different piles based upon their color, though any criterion is adequate for this example. The problem arises whenever we ask whether our criterion worked in that color actually separated good apples from bad apples. How can we investigate our criterion without already knowing something about which apples are good and bad? (pp. 2–3)
In effect, identifying an appropriate criterion measure renders criterion-related validity very difficult for test designers in general and for classroom teachers in particular.
Construct validity became prominent about halfway through the 1900s. According to Hathcoat (2013), a seminal article in 1955 by Lee J. Cronbach and Paul E. Meehl led to a focus on construct validity. Hathcoat (2013) notes that “Cronbach and Meehl were concerned about situations wherein a target domain and/or a relevant criterion remained ambiguous” (p. 3).
Cronbach and Meehl (1955) were saying that construct validity must be established for any type of content for which it is difficult to find a criterion measure (as cited in Hathcoat, 2013). For example, where it is rather easy to find a criterion measure for content like fifth-grade geometry, it is quite difficult to find criterion measures for content like students’ abilities to apply knowledge in unique situations or students’ abilities to make good decisions. Any instrument designed to measure these topics must establish construct validity evidence of what such an ability entails.
In the middle of the 20th century, around the same time Cronbach and Meehl (1955) established the need for construct validity, statistical procedures became readily available that allowed psychometricians to induce the nature of an otherwise ambiguous construct. One such statistical procedure is factor analysis, which mathematically provides evidence that specific items on a test measure the same construct. (For a technical discussion of factor analysis, see Kline, 1994.) This type of analysis is also beyond the resources of the typical classroom teacher. In fact, from the perspective of the classroom teacher, construct validity is probably more a function of the standard that he or she is trying to assess than the assessment he or she is designing. For example, assume a teacher is trying to design an assessment for the following standard: “Students will be able to work effectively in cooperative teams.” Construct validity would address the extent to which this standard represents a definable set of knowledge and skill—something that could actually be taught and measured.
For criterion-related validity and construct validity, the classroom teacher has few, if any, resources to address them. However, the classroom teacher can address content validity, which basically reaffirms the early definition of validity—the test measures what it is purported to measure. For the classroom teacher, this simply involves ensuring that the CA addresses the content in the standard that is the focus of instruction and assessment.
In summary, from the instrumental perspective, the classroom teacher has limited or no control over two of the three types of validity associated with CAs he or she is designing. However, from the argument-based perspective, the teacher has some control over all three types of validity.
The Argument-Based Perspective
The argument-based perspective of validity is relatively new. Although it can be traced back to work in the 1970s and 1980s around the importance of test interpretation articulated by Messick (1975, 1993), the argument-based approach became popular because of a series of works by Michael T. Kane (1992, 2001, 2009). At its core, argument-based validity involves an interpretive argument that “lays out the network of inferences leading from the test scores to the conclusions to be drawn and any decisions to be based on these conclusions” (Kane, 2001, p. 329).
From the instrumental perspective, it is the assessment itself that possesses a specific type of validity (criterion, construct, or content). In contrast, from the argument-based perspective, validity is a function of how the data generated from the assessment are used to craft an argument regarding a particular student’s knowledge or skill. This type of validity applies nicely to the classroom teacher.
From the argument-based perspective, then, criterion-related validity for a CA is determined by a teacher’s ability to use data from the assessment to predict students’ performance on interim assessments and end-of-course assessments. If students do well on the CAs for a particular topic, they should also do well on the more formal assessments on that topic designed outside of the classroom.
Construct validity for CAs is determined by the extent to which a teacher can use data from these assessments to identify specific knowledge and skills that should be directly taught. If a teacher can translate scores on the CAs into specific types of instruction for specific students on specific content, then the information generated from the CAs is judged to have construct validity.
From the argument-based perspective, content validity for CAs is determined by a teacher’s ability to use the information generated from CAs as evidence regarding students’ current knowledge and skill on a specific topic. If the teacher can use the scores on the CAs to determine what content students know and what content they don’t know on a specific progression of knowledge, then the information generated from the CAs is judged to have content validity.
The distinction between the instrumental and argument-based perspectives is critical to establishing the validity of CAs. Table 1.1 summarizes these differences.
Table 1.1: CA Validity From the Instrumental and Argument-Based Perspectives
| Validity Type | Instrumental Perspective | Argument-Based Perspective |
| Criterion-Related Validity |
Scores on a
|
