Testing Whether Reported Treatment Effects are Unduly Dependent on the Specific Outcome Measure Used
Peter Halpin Joshua Gilbert


 




 








Testing Whether Reported Treatment Effects are Unduly Dependent on the Specific Outcome Measure Used
    Peter F. Halpin
 

	   School of Education

	   University of North Carolina at Chapel Hill 
 
	   Chapel Hill, NC 27514 

	    
	   Joshua B. Gilbert 
 
	   Graduate School of Education 

	   Harvard University 

	   Cambridge, MA 02138 

	   
	   
    
=============================================================================================================================================================================================================================================================

§ ABSTRACT 

This paper addresses the situation in which treatment effects are reported using educational or psychological outcome measures comprised of multiple questions or “items.” A distinction is made between a treatment effect on the construct being measured, which is referred to as impact, and item-specific treatment effects that are not due to impact, which are referred to as differential item functioning (DIF). By definition, impact generalizes to other measures of the same construct (i.e., measures that use different items), while DIF is dependent upon the specific items that make up the outcome measure. To distinguish these two cases, two estimators of impact are compared: an estimator that naively aggregates over items, and a less efficient one that is highly robust to DIF. The null hypothesis that both are consistent estimators of the true treatment impact leads to a Hausman-like specification test of whether the naive estimate is affected by item-level variation that would not be expected to generalize beyond the specific outcome measure used. The performance of the test is illustrated with simulation studies and a re-analysis of 34 item-level datasets from 22 randomized evaluations of educational interventions. In the empirical example, the dependence of reported effect sizes on the type of outcome measure (researcher-developed or independently developed) was substantially reduced after accounting for DIF. Implications for the ongoing debate about the role of researcher-developed assessments in education sciences are discussed.This material is based upon work supported by the National Science Foundation under Grant No. 2400864. 






Meta-analyses of educational interventions have consistently documented the importance of methodological factors related to the choice of outcome measures. In particular, when interventions are evaluated using measures developed by researchers involved with the intervention or its evaluation, the effect sizes tend to be larger than when using independently developed measures of the same or similar constructs <cit.>. For example, two recent meta-analyses focusing on PreK-12 math <cit.> and science <cit.> documented average effect sizes that were, respectively, 0.30 SD and 0.27 SD larger when using researcher-developed outcomes. In both studies, the type of outcome measure was more strongly related to reported effect sizes than any other methodological factor, including whether or not the intervention was randomized. The differences associated with the type of outcome were an order of magnitude larger than the focal comparisons among the interventions. 

Several hypotheses have been proposed about why researcher-developed outcome measures are associated with larger effect sizes <cit.>. In the usual case that the outcome measures are educational or psychological assessments (e.g., achievement tests, self-report surveys), some of these hypotheses can be interpreted in terms of how treatment effects depend on the specific questions or “items” that make up the assessments. For example, it has been argued that researcher-developed assessments may be more narrow in scope  and more closely aligned with target learning domains than independently developed assessments <cit.>. The rationale here is that assessment items less closely aligned with the intervention should show diminished treatment effects. If independently developed assessments contain relatively larger proportions of such items, this could explain the reported discrepancies between researcher-developed and independently developed outcome measures. 

Others have argued that the content of researcher-developed measures may be either directly or indirectly pre-exposed to participants in the treatment group <cit.>, or may be otherwise over-aligned with the intervention being evaluated (What Works Clearinghouse, ). For example, some of the questions on a researcher-developed assessment may be drawn directly from program materials seen only by the treatment group. In this situation, we would expect relatively larger “effects” on items that included pre-exposed material. More generally, the pattern of effects on pre-exposed items is unlikely to be consistent with the pattern implied by a treatment effect on the underlying construct being measured, a point we elaborate on below.

A related area of research has addressed item-level heterogeneous treatment effects in the context of program evaluations. A recent re-analysis of 46 randomized control trials (RCTs) in education, economics, and health documented extensive item-level heterogeneity, with 46 of 73 outcome measures exhibiting statistically significant item-level variation <cit.>. Various implications of item-level treatment effect heterogeneity have also been documented. For example, reported treatment effects can be highly sensitive to the removal of even just a single item <cit.>, and statistically significant item-level or subscale-level effects can be “masked” by aggregate null effects <cit.>. Variation in item-level effects can also inflate the standard errors of reported treatment effects <cit.>, and correlations between item-level effects and item difficulty may lead to bias in the estimation of treatment-by-covariate interactions <cit.>. 

The present paper contributes to this growing area of research by showing how item-level treatment effect heterogeneity can lead to systematic bias (technically, inconsistency) in reported treatment effects. To formalize this idea, we draw from the psychometric literature to distinguish between (a) an overall treatment effect on the construct being measured, which is referred to as impact, and (b) item-level treatment effects that are incompatible with an overall treatment effect, which are referred to as differential item functioning <cit.>. By definition, impact generalizes to other assessments of the same construct (i.e., assessments that use different items) whereas DIF is dependent on the specific items that make up the outcome measure. We argue that this distinction is an important but overlooked consideration about the internal validity of research studies in education and related fields. 

Building on this distinction, an asymptotic test is proposed for inferring whether reported treatment effects are affected by DIF. The test is constructed by comparing an estimator of impact that naively aggregates over assessment items with a less efficient estimator that is highly robust to DIF. The former estimator corresponds to the usual practice of reporting treatment effects using the unweighted mean over assessment items. The latter was recently developed to facilitate item-by-item tests of DIF <cit.>. The null hypothesis that both are consistent estimators of the true treatment impact leads to a Hausman-like specification test <cit.> of whether reported treatment effects are due to impact on the target construct instead of item-level variation that would not be expected to generalize beyond the specific outcome measured used. 

The main result of the paper shows that the robust estimator is consistent for the true treatment impact so long as fewer than 1/2 of items exhibit DIF. This result is complemented by Monte Carlo simulations that illustrate the how the specification test performs when more than 1/2 of the items are systematically biased in the same direction (e.g., toward 0). Together, these results indicate that, while the sign of the test statistic will depend on the proportion of items that exhibit DIF, its population value will be non-zero whenever item-level variability leads to inconsistency of the naive estimate. Consequently, the specification test remains useful regardless of the proportion of assessment items that exhibit DIF.  

The specification test is illustrated using 34 item-level datasets obtained from 22 randomized control trials (RCTs) of educational interventions. The studies were implemented in 13 countries, addressed a wide range of age groups and educational outcomes, and the majority (21 out of 22) involved cluster randomization of treatment conditions. Although the collection of studies is a convenience sample based on public availability of item-level data, the empirical analyses offer an illustration of the testing procedure and an initial indication that reported treatment effects may often be dependent on the specific outcome measured used.  

In the discussion, we return to the question of how the proposed methodology can address ongoing concerns about the role of researcher-developed assessments in education sciences. Not all of these concerns can be framed in terms of item-to-item variability of treatment effects. For example, the choice of outcome measure may be systematically related to other aspects of study design <cit.>. It is also the case that item-level effects may represent scientifically interesting sources of treatment effect heterogeneity that can inform education theory  <cit.>. Consequently, it is important to understand the limitations of the proposed methodology as well as the potential for future research directions. 



§ DIF IN PROGRAM EVALUATION RESEARCH
 

The conceptual model underlying this research is summarized in Figure 1. Panel (a) depicts a situation in which treatment status, T, directly affects a target construct, θ, that is measured by the assessment items, X_i. There are two main assumptions at play in this panel. First, it is assumed that the path from T to θ is not confounded, which can be assured through random assignment of treatment status. Second is the psychometric assumption of conditional independence, which implies that E[X_i |θ, W] = E[X_i |θ]. Traditionally, this assumption has focused on the case in which W = X_j≠ i are other items on the assessment <cit.>. Later, researchers also considered situations in which W may be another variable external to the assessment <cit.>. In panel (b) of the figure, the assumption of conditional independence is relaxed for W =T, so that treatment status directly affects both the target construct and the assessment items. 



As recently recognized in the literature on causal inference <cit.>, the model in panel (a) is tacitly assumed when taking a univariate summary of the assessment items (e.g., the unweighted mean) and using it to study the treatment effect on the target construct. This can be seen most clearly in the case where the measurement model relating  X_i and θ is linear. Writing E[X_i |θ] = a_i θ + b_i implies 


    E[X_i | T = t]     = E[E[X_i |θ, T = t] | T = t] 
        = E[E[X_i |θ] | T = t] 
        = a_i E[θ| T = t] + b_i.


The crucial step in this derivation is the use of conditional independence in the second line. The overall conclusion is that, under the model in panel (a) of Figure 1, the item-level treatment effects are known functions of the treatment effect on the target construct. Importantly, this leads to testable implications about the pattern of treatment effects over items. 

Under the model in panel (b) of Figure 1, Equation impact can be re-written as


    E[X_i | T = t]  = a_i E[θ| T = t] + b_i + c_i I(T = 1),

 
where I(T = 1) is an indicator function for treatment status and c_i is the item-specific treatment effect. This situation has been interpreted in terms of item-level treatment effect heterogeneity. As summarized in the introduction, recent research indicates that this phenomenon is quite pervasive in education and related fields. 

In the psychometric literature, the treatment effect on the target construct (i.e., the solid path from T to θ) is called impact, and the item-specific treatment effects (i.e., the dashed paths from T to X_i) are referred to as DIF <cit.>. An extensive methodological literature on DIF has developed since the 1990s <cit.>.[In particular, Theorem 1 of <cit.> was anticipated by <cit.>.] The present paper follows the psychometric literature by building from the conceptual model in Figure 1 towards more realistic data-analytic models that allow for non-additive DIF and different link functions in the measurement equations. 

The DIF literature has focused on applications in which the predictor of interest is a pre-existing characteristic of the respondents (e.g., gender, race) rather than treatment status. In this context, DIF is typically interpreted in terms of measurement bias and regarded as a threat to test fairness <cit.>. In terms of  Figure 1, the overall goal of DIF analysis is to identify any non-zero red paths in panel (b) and omit those items from the assessment, leaving us with the model in panel (a). 

The interpretation of DIF in the context of program evaluation research is quite different. Rather than fairness, the focal issue is whether the observed treatment effect is attributable to impact on the target construct or may also reflect other sources of variation. It is argued that distinguishing these two cases is important for establishing the internal validity of studies that evaluate educational interventions. By definition, impact generalizes to other assessments of the same construct. For this reason, impact is a valid causal mechanism through which observed treatment effects can arise. On the other hand, DIF with respect to treatment is dependent on the specific items that appear on an assessment, and therefore does not generalize to assessments that do not use those same items. DIF becomes a threat to internal validity when it leads researchers to make incorrect inferences about impact. There are many potential sources of item-level treatment effect heterogeneity, and documenting and theorizing them may provide fertile grounds for future research <cit.>. However, the focus of the present paper is the question of internal validity: Do reported treatment effects reflect impact on the target construct, or are they dependent upon the specific items used to measure the construct?

In summary, distinguishing between impact and DIF is important for understanding the causal mechanism underlying observed treatment effects. In particular, this distinction directly addresses whether treatment effects reported using one outcome measure would be expected to generalize to other measures of the same construct. The extent to which observed treatment effects are due to impact or DIF is an empirical question, but one which has received very little attention in the psychometric literature. 



 §.§ Previous Approaches to DIF


The methodology developed in this paper can be motivated by considering the following three reasons that traditional methods for DIF analysis are not well suited to the context of program evaluation research. 

1. DIF with respect to treatment cannot be discovered until it is too late – i.e., after the intervention has been administered. In this context, the usual practice of addressing DIF by omitting some items from the outcome measure would be highly questionable. In terms of Figure 1, this means we are squarely stuck with panel (b), and, in particular, with the problem of estimating impact in the presence of DIF. This problem has received very little attention in the psychometric literature, because, as mentioned, the usual goal has been to identify and remove problematic items during test development. 

2. The shortcomings associated with commonly used item-by-item tests of DIF also become highly problematic in the present setting. These procedures typically involve conducting a large number of statistical tests, often in combination with iteratively re-fitting many IRT models to the data, and manually selecting so-called “anchor items” that are assumed not to exhibit DIF <cit.>. The choice of anchor items is especially problematic, as it can dramatically affect the results of DIF analyses, even changing the direction of impact <cit.>. It has also been shown that these methods can fail badly when a relatively large proportion of items (> 20%) exhibit DIF in the same direction <cit.>. In general, item-by-item tests of DIF are inherently exploratory procedures that are not well suited to program evaluation settings. 

3. Item-by-item tests of DIF can be replaced with omnibus tests of whether any item exhibits DIF, which are often referred to in terms of measurement invariance <cit.>. However, inferring whether or not any item exhibits DIF is not the same as inferring whether DIF affects conclusions about impact. DIF on one item may be offset by countervailing DIF on another item, a phenomenon that has been referred to as “DIF cancellation" <cit.>. When DIF is isolated to only a small proportion of items, its overall effect on estimates of impact can be negligible, which is a point illustrated in the simulations studies reported below. If an omnibus test of measurement invariance indicates the presence of DIF, researchers will typically proceed with the usual item-by-item tests to identify potentially problematic items, and then refit a “final" model that omits those items or allows their parameters to vary over groups. After all this, we still lack a standard procedure for inferring whether there is a non-zero difference between the estimates of impact in the original model and the final model, because the covariance of the two estimates is not available from the separate model fits. 

In summary, conventional item-by-item tests of DIF have many practical shortcomings, and, ultimately, are not directly relevant to the question of whether DIF affects conclusions about impact. While this question has received some attention in the psychometric literature, current approaches require that items with DIF are specified beforehand <cit.>. The methodology described in the following section seeks to circumvent this issue by directly comparing two estimators of impact.



§ STATISTICAL METHODOLOGY


This section derives the asymptotic distribution of the difference between an estimator of impact that naively aggregates over assessment items and a less efficient estimator that is highly robust to DIF. The former corresponds to the usual practice of estimating treatment effects using the unweighted mean over items. The robust estimator was recently developed in the context of item-by-item tests of DIF <cit.>. Theorem 1 of the present paper provides a sufficient condition for the robust estimator to be consistent for the true treatment impact on the target construct. The condition is seen to be quite weak, requiring only that there are more “good” items without DIF than there are “bad” items with numerically identical bias. 

The null hypothesis that both the naive and robust estimators are consistent for the true treatment impact leads to a test of whether DIF affects the naive estimator. As noted, the rationale here is a lot like Hausman's () specification test. However, the derivation of the asymptotic distribution of the difference between the estimators does not require Hausman's lemma regarding the covariance of an efficient estimator and its difference with an inefficient estimator. This is a practical advantage, since Hausman's lemma is an asymptotic result that does not always obtain in finite samples <cit.>. Additionally, the alternative hypothesis considered by Hausman was that only the less efficient estimator is consistent. In the present context, the alternative hypothesis is less clear cut: the robust estimator will remain consistent up to a certain point, but both estimators may become inconsistent if a sufficiently large proportion of items exhibit DIF that systematically favors one group over the other. This limitation is discussed further in the simulation studies reported below. 

Both estimators are developed using IRT-based scaling with the common items non-equivalent groups (CINEG) design <cit.>. In practice, this means that the proposed methodology can be implemented as a post-estimation procedure following maximum likelihood estimation of a focal psychometric model, estimated separately in the treatment and control groups. This paper focuses on the two-parameter logistic (2PL) IRT model for binary response data, although the main results extend directly to other IRT models (e.g., for ordered categorical data).



 §.§ Model Specification


Begin by specifying the 2PL model in slope-intercept form:


    logit(p_gi) =   a_giθ_g + b_gi,


where: 

 

  * g = 0 ,1 denotes the control and treatment groups, respectively. 

  * i = 1 … m denotes the assessment items. 

  * p is the probability of endorsing an item. 

  * a > 0 is the item slope (also called discrimination).  

  * b ∈ℝ is the item intercept (also called an item threshold or “easiness”). 

  * θ_g = (θ^*_g- μ_g)/ σ_g with θ^*_g ∼ N(μ_g, σ_g^2).  



In this paper, we refer to a_gi and b_gi as the item parameters, μ_g and σ_g as scaling parameters, and θ_g as the latent trait. The constraints required to identify the model are imposed by standardizing the latent trait in both groups, which is discussed further in the next section. 
 
The results presented below assume that maximum likelihood estimates (MLEs) of the item parameters are available. To this end, let ν_i = [a_i0, b_i0,  a_i1, b_i1]^T denote the vector of parameters for item i and ν= [ν_1^T, ν_2^T , ⋯, ν_m^T]^T. The MLEs, ν̂, have asymptotic distribution denoted by √(n) (ν̂ - ν) d→ N(0, V(ν̂)) where n = n_0 + n_1 is the total sample size and n_1 / n_0 → (0, ∞) <cit.>. The focus will be on plug-in estimators constructed using ν̂, and the asymptotic distribution of these estimators will be obtained by applying the Delta method <cit.>. 




 §.§ Estimating impact using IRT-based scaling


Standardizing the latent trait separately in both groups, as in Equation glm, implies that impact is ignored when estimating the model. However, it is possible to recover group differences on the latent trait in a post-estimation step, which is often referred to as test scaling or linking. This section outlines a variation on so-called moment-based scaling using comparable items non-equivalent groups (CINEG) procedure <cit.>. The mathematical details follow closely to the established procedures in IRT, but the notation and terminology are adapted to emphasize that the groups of respondents are defined with respect to treatment status. 

The scaling parameter of interest is defined as the average treatment effect on the target construct, standardized in the control group:


    δ_0 = (μ_1 - μ_0) / σ_0.


We refer to this as impact on the target construct. It also commonly referred to as an effect size <cit.>. Because the variance of the latent trait is not identified without additional model constraints, we do not distinguish between (unstandardized) treatment effects and (standardized) effect sizes in the present paper. 

Next, define the item-level treatment effects:


    δ_i(ν) = (b_i1 - b_i0) / a_i0.


In the Section <ref> of the Appendix, it is shown that δ_i(ν) = δ_0 if and only if item i does not exhibit DIF. Thus, the δ_i(ν) will be constant over items when the treatment effect on each item is due exclusively to impact. This corresponds to the situation shown in panel (a) of Figure 1 (i.e. “no DIF”). When item-level treatment effects arise from sources other than impact, the δ_i(ν) will vary over items. This corresponds to panel (b) of Figure 1. 

Aggregating over items leads to the following test-level treatment effect:


    δ(ν)= ∑_i=1^m w_i δ_i(ν),


with convex weights w_i  ∈ [0, 1] and ∑_i w_i = 1. The assumption that the item-level treatment effects are due exclusively to impact leads the unweighted mean as a natural test-level treatment effect. In the IRT literature <cit.>, this is referred to as “mean scaling." The unweighted mean will be written  as δ_U(ν) with weights w_Ui = 1/m.  In the present context, the plug-in estimator δ̂_U = δ_U(ν̂) can be interpreted as the IRT analog of estimating average treatment effects using the unweighted mean over assessment items. Because δ̂_U is motivated by the assumption that no item exhibits DIF, it is referred to as a naive estimate of impact. 

To address the situation in which a subset of items i ∈𝒟 exhibit DIF, we can re-write Equation delta as 


    δ(ν) = ∑_i ∈𝒟 w_i δ_i(ν) +  δ_0 ∑_j ∉𝒟 w_j.


Here it is assumed that 0 < m_D < m, where m_D is the number of items with DIF.  From Equation delta-dif, it is apparent that there are two situations in which a plug-in estimator δ(ν̂) will remain consistent for δ_0 in the presence of DIF. The first case arises when w_i = 0 for all items that exhibit DIF. This provides an intuitive motivation for the robust estimator of impact presented in the next section – i.e., the goal is to down-weight items that exhibit DIF. 

The second case arises when 


    ∑_i ∈𝒟 w_i δ_i(ν) = (1 - ∑_j ∉𝒟 w_j) δ_0.

In particular, δ̂_U will remain consistent so long as DIF “cancels out” over items:


    δ_0 = 1/m_D∑_i ∈𝒟δ_i(ν).


Consequently, testing whether a naive estimator of impact is consistent for δ_0 is less restrictive than testing whether any item exhibits DIF, because we are purposely ignoring item-to-item fluctuations that do not have a cumulative effect on impact. 



 §.§ Estimating Impact in the Presence of DIF
 

This section outlines a recently developed test-level treatment effect estimator that is highly robust to DIF <cit.> and presents a new result about its consistency for the true treatment impact. The construction of the estimator borrows computational techniques from robust estimation of a location parameter <cit.>. In the present setting, the item-level treatment effects δ_i(ν) play the role of “data points” whose location we wish “estimate.” This analogy is seen to lead to a robust version of Equation delta. However, it is important to clarify that the asymptotic distribution of the resulting plug-in estimator will be obtained via the Delta method and does not otherwise make use of the asymptotic theory of robust statistics. 

Begin by writing the weights in Equation delta in terms of a scalar-valued function ψ, to be chosen subsequently: 


    w_i = ψ(u_i) / u_i/∑_r = 1^m ψ(u_r) / u_r.
 

By convention, the ratio ψ(u)/u is set to 1 when u = 0. The argument u_i = u_i(ν) = δ_i(ν) - δ is a function of both the item parameters ν and a “preliminary value” of the test-level treatment effect, δ. The dependence on δ leads to iteratively re-weighted least squares (IRLS) as a computational strategy <cit.>. 

In the present context, it is desirable to choose ψ to be a so-called redescending function <cit.>. While many redescending functions are available, Tukey's bi-square function performs well in practice and is convenient for illustrating the main ideas. The bi-square is defined as 


    ψ(u) = {[ u (1 - ( u/k)^2)^2                for            |u| ≤ k;                  0                for            |u| > k;                    ]. .


The overall idea behind the bi-square is to choose the tuning parameter k such that ψ = 0 for outliers. This has the effect of setting the weights w_i = 0, so that outliers are down-weighted to zero during estimation. It can also be confirmed that the ratio ψ(u)/u is non-negative and reaches a maximum of 1 when u = 0. Thus, in the idealized case that u_i = 0 for all items without DIF and |u_i| > k for all items with DIF, the test-level treatment effect in Equation delta becomes the unweighted mean of the items without DIF. As noted in the previous section, this would ensure that the resulting plug-in estimator remains consistent for true treatment impact by down-weighting any items with DIF, which is the desired behavior.  

The performance of the bi-square depends strongly on the choice of the tuning parameter k <cit.>. In usual applications, k is chosen to be proportional to the scale (e.g., median absolute deviation) of the data points. In the present context, we can follow a similar logic by choosing k based on the asymptotic distribution of the item-level treatment effects δ̂_i = δ_i(ν̂). Under the null hypothesis that item i does not exhibit DIF (i.e., δ_i(ν) = δ_0), this distribution can be written as √(n)(δ̂_i - δ_0) d→ N(0, V_0(δ̂_i)). Letting α denote the desired Type I Error (false positive) rate for down-weighting items with DIF, we may choose item-specific tuning parameters


    k_i = z_1-α/2×√(V_0(δ̂_i) / n ),
 

where z_q denotes the q-th quantile of the standard normal distribution. This choice of k_i implies that the weights w_i will be set to zero whenever δ̂_i is beyond the (1 - α) × 100 % asymptotic confidence interval centered at δ. 

Weights that are computed using Equations w through k will be referred to as robust weights and denoted w_Ri. The plug-in estimator constructed by using these weights in Equation delta will be denoted as δ̂_R = δ_R(ν̂). Halpin () discussed some additional details that improve the robustness and efficiency of δ̂_R; these are summarized in Section <ref> of the the Appendix. The following theorem addresses the consistency of δ̂_R. The proof is provided in Section <ref> of the the Appendix.


 Assume that there exists a value δ such that the number of items whose treatment effect is equal to that value (i.e., δ_i(ν) = δ) exceeds the number of items whose treatment effect is equal to any other particular value, say δ^* (i.e., δ_i(ν) = δ^* ≠δ). Then, as the sample size n →∞, the plug-in estimator δ̂_R defined in Equations delta, w - k converges in probability to δ. 
 

The theorem provides a relatively weak condition under which δ̂_R will remain consistent for the true treatment impact, δ_0. In the worst-case scenario that all items that exhibit DIF have item-specific treatment effects equal to the same aberrant value δ^*, then δ̂_R will remain consistent for δ_0 so long as fewer than 1/2 of items exhibit DIF. Perhaps a more natural assumption is that DIF is idiosyncratic – i.e., δ_i(ν) ≠δ_j(ν) unless they are both equal to δ_0. In this case, δ̂_R will remain consistent if just 2 items do not exhibit DIF.

Importantly, the proof makes use of the modified tuning parameters k^*_i = max{k_i,  ϵ}, where ϵ > 0 is a constant chosen to be less than 1/2 the smallest non-zero distance between any two item-level treatment effects, | δ_r(ν) - δ_s(ν)| for r, s = 1 … m. In practical terms, this means that standard errors of the δ̂_i must be substantially smaller than the variation among the δ_i(ν) before the mechanism ensuring consistency “kicks in." Consequently, the performance of the estimator with realistic sample sizes is an important consideration, which is addressed in the simulation studies reported below. 



 §.§ Testing Whether Naive Estimates of Impact are Affected by DIF


Having the naive and robust test-level treatment effects in hand, the next step is to consider their difference: 


    Δ (ν) = δ_U (ν) - δ_R (ν) = ∑_i=1^m(1/m - w_Ri) δ_i(ν).
 

Under the condition described in Theorem 1, δ_R (ν) = δ_0. Thus, when the condition holds, testing the null hypothesis that Δ (ν) = 0 is equivalent to testing δ_U (ν) = δ_0. A Wald test of this hypothesis is available via the asymptotic distribution of the plug-in estimator Δ̂= Δ (ν̂). In the Section <ref> of the Appendix, the Delta method <cit.> is used to derive this distribution. The relevant result is 


    √(n) (Δ̂- Δ (ν)) d→ N(0, V(Δ̂))
  

where 


    V(Δ̂) = ∑_i=1^m(1/m - v_Ri)^2 V(δ̂_i)    and    v_Ri = ψ'(u_i)/∑_r = 1^m ψ(u_r) / u_r .



A closed-form expression for V(δ̂_i) is also given in the Appendix. 

Equations test and var-Delta define the proposed specification test. The plug-in estimate Δ̂ will be referred to as the Delta statistic and z = Δ̂/ SE(Δ̂) as the Delta test. When the condition stated in Theorem 1 holds, the null hypothesis Δ (ν) = 0 implies that the naive test-level treatment effect  δ̂_U is a consistent estimate of the true treatment impact on the target construct, δ_0. When the null hypothesis is false, the naive effect is inconsistent for δ_0 due to item-level sources variation. In this situation, a pragmatic interpretation is that reported treatment effects are unduly dependent upon the specific items that make up the outcome measure.  It is important to emphasize that, if the condition stated in Theorem 1 does not hold, then Δ (ν) = 0 does not necessarily imply that δ_U (ν) = δ_0. This is an important limitation of the proposed test and is addressed further in the simulations reported below. 



§ SIMULATION STUDIES


The overall goal of the simulations is to complement Theorem 1 by describing the finite sample performance of the test-level treatment effects and Delta test under varying degrees of DIF. In particular, the theorem implies that the robust estimator δ̂_R will be consistent for whichever value of δ_i occurs most frequently. When fewer than 1/2 of item exhibit DIF, this implies that δ̂_R is consistent for the true treatment impact δ_0. However, when more than 1/2 of items are systematically biased towards the same value (e.g., 0), the theorem implies that the robust estimator will be a consistent estimate of that value instead of δ_0.  We are especially interested in the performance of the Delta test in such situations. Additionally, the asymptotic result stated in Theorem 1 requires that the sampling error of the δ̂_i must be substantially smaller than the variation among the item-level treatment effects. Thus, a secondary interest is to compare the bias of two estimators in finite samples. 

We report two simulation studies that are motivated by the two theoretical cases considered in the introduction. In the first case, the concern was that a non-null treatment effect could be “washed out" when the outcome measure includes a large proportion of items that are not aligned with the intervention. This case is modeled using an assessment in which items without DIF exhibit moderate impact on the latent trait (δ_i = 0.4), while items with DIF exhibit no impact (δ_i = 0). The value of 0.4 was chosen to reflect the estimated effect sizes on researcher-developed assessments reported in meta-analyses of math and science education <cit.>. 

In the second case, the concern was that a null treatment effect could be masked by pre-exposure of assessment content in the treatment group. This is modeled using a null treatment impact on items without DIF (δ_i = 0), while for items DIF the logit of a correct response systematically favors the treatment group (b_i1 = b_i0 + e where e ∼ U(0.4, 0.5)). Note that, in contrast with the first simulation study, the treatment effects on the biased items are not constant. This is because (a) e_i varies randomly over items, and (b) the item discrimination α_0i, which is used to compute the item-level treatment effects δ_i, also varies over items. The resulting range of possible values for the item-level treatment effects is δ_i ∈ [0.2, 1]. This is intended to reflect the situation in which non-zero item-level treatment effects are driven by a mechanism (pre-exposure) that is not related to impact on the target construct and consequently vary over items. 

The focal factor in both simulations is the proportion of items that exhibit DIF, p ∈ [0, 1]. As noted, this allows us to study the performance of the specification test in cases where the robust estimator is consistent for the true treatment impact (p < 1/2), as well as cases where it is inconsistent (p ≥ 1/2). Items that exhibit DIF are chosen at random in each replication. The other details of the simulation studies are summarized in Table 1. The number of assessment items is m = 16 and the number of respondents per group is n_g = 500. These are not focal factors of the study and are fixed over conditions. Item parameters vary randomly over replications. The range of item slopes is [.5, 2], representing predicted change on the logit scale for a 1 standard deviation increase in the latent trait. The range of item thresholds corresponds to [-1.5, 1.5] standard deviations on the latent trait. In each simulation, the two estimators are compared using half-violin plots, and the specification test is summarized in terms of false positive (Type I Error) and true positive (power) rates. 


   


 §.§ Simulation 1: Wash-out
 

In this simulation, the true treatment impact is δ_0 = 0.4 and all items that exhibit DIF have null effects. Theorem 1 implies that the robust estimator will be consistent for δ_0 when p < 1/2, but will instead be consistent for 0 when p > 1/2. When p = 1/2, both values are solutions and the estimator is inconsistent. Thus δ_R(ν) ∈{0, 0.4} for all simulation conditions. By contrast, δ_U(ν) will always fall between these two extreme values when 0 < p < 1. 

This behavior is illustrated in Figure <ref>. When p = 0, it can be seen that both estimators are centered near the true treatment effect (δ_0 = 0.4), although the naive estimator is slightly more efficient. As the number of items with null treatment effects increases up to 1/2, the robust estimator remains much less biased than the naive estimator. At p = 1/2 the distribution of the robust estimator becomes strongly bi-modal, and this is also apparent to a lesser degree for values of p close to 1/2. The robust estimator  becomes a consistent estimate of the treatment effect on the biased items (i.e., δ_i(ν) = 0) when p > 1/2, and the two estimators are both centered at this value when all items exhibit null effects (p = 1). 



The figure shows that the behavior of the two estimators when p < 1/2 is a mirror image (reflection about the horizontal line at 0.2) of their behavior when p > 1/2. Consequently, the sign of the difference between the two estimators “flips" depending on the proportion of items with DIF. However, on average, the absolute value of the difference remains non-zero when 0 < p < 1, and is only equal to zero in the two extreme cases where p = 0 or p = 1. In both of these cases, the item-level treatment effects are constant over items, so it is correct to conclude that item-level treatment effect heterogeneity does not affect conclusions about impact. 

Based on these considerations, we may expect the Delta test to attain the nominal false positive rate when p = 0 or p = 1, and have non-zero power to reject the null hypothesis that Δ = 0 when 0 < p < 1. This conclusion is supported by the insets in Figure <ref>, which provide the rejection rates of the Delta test over the 500 replications, using a nominal rejection rate of α = .05. Overall, the results of this simulation support the conclusion that, while the sign of the Delta statistic depends on the number of items that exhibit DIF, its magnitude will generally be non-zero when a strict subset of items are systematically biased towards the same value. 



 §.§ Simulation 2: Pre-exposure
 

In this simulation, the true treatment impact is δ_0 = 0 and items that exhibit DIF are uniformly distributed in the range δ_i(ν) = [0.2, 1]. This situation is importantly different from the first simulation, because it is less obvious how variation in the item-level treatment effects will affect the finite sample performance of the robust estimator when the proportion of items that exhibit DIF exceeds 1/2. The theorem implies that the robust estimator should remains consistent when as few as 2 items do not exhibit DIF. However, it is also expected that large sample sizes will be required to realize this result. 

Figure <ref> compares the naive and robust estimators. In contrast to Simulation 1, upward rather than downward bias is induced by DIF. However, the qualitative pattern of results in Simulation 2 is very similar to Simulation 1. The robust estimator is less biased than the naive estimator, up until about p = 1/2 of the items exhibit DIF (8/16). The robust estimator becomes strongly bi-modal close to p = 1/2, and the behavior for p > 1/2 is an (approximate) mirror image of the behavior for p < 1/2. Insets show the false positive rate (when p = 0) and true positive rate (p > 0) of the specification test, using a nominal rejection rate of α = .05. The results of this simulation suggest that the specification test can be useful when p > 1/2  items are biased in the same direction, but not necessary by the same amount. 






 §.§ Summary

 
The simulations show that, when the proportion of items with DIF is less than 1/2, the robust estimator is much less biased than the naive estimator and consequently the Delta test performs as intended. When the number of items with DIF is greater than or equal to 1/2, the robust estimator may no longer be consistent for the true treatment impact. Consequently, the difference between the robust estimator and the naive estimator, (i.e., the Delta statistic), is no longer a good approximation of the difference between the true naive test-level treatment effect and the true treatment impact on the target construct. However, the simulations showed that, even after the robust estimator becomes inconsistent, the empirical distributions of the two estimators continued to have different expected values. As an intuitive explanation, we may reason as follows: Theorem 1 shows that robust estimator should behave the like the mode (most common value) of item-level treatment effects, where as the naive estimator is their mean. When the distribution of the item-level effects is highly asymmetrical (e.g., when DIF systematically favors one group over the other), these two values will generally differ. 

The simulations support the overall conclusion that, while the sign of the test statistic depends on the number of items that exhibit DIF, its magnitude will generally be non-zero when items with DIF are systematically biased in the same direction. In such situations, the proposed specification can be useful for inferring whether the naive estimator is affected by item-level variation, regardless of how many items exhibit DIF. 



§ EMPIRICAL ILLUSTRATION


In this section, we re-analyze 34 datasets from 22 unique RCTs. The data are a subset of a collection of 73 publicly available datasets from 46 RCTs with item-level data <cit.>.[The full collection of datasets will be publicly available at the following URL: https://doi.org/10.7910/DVN/C4TJCAhttps://doi.org/10.7910/DVN/C4TJCA.] In this re-analysis, we focus on educational interventions with children or adolescents. The datasets are summarized in Table <ref>. Together, the datasets include responses from 224, 572 individuals to 784 items, representing a wide range of geographic settings and outcome measures. 




Our re-analysis used initial treatment assignment and considered the first post-treatment followup assessment. We included only datasets with more than 500 subjects and more than 10 items, after excluding any items with endorsement rates outside of the 2-98% range, items with abnormally high (>10) or low (< .25) discrimination parameter estimates, or items responded to by less than 10% of the total sample. A handful of items with polytomous responses were re-coded to binary. All IRT models were fitted separately to the treatment and control samples in each dataset. For estimation, we used Stata's IRT module and, where relevant, standard errors were clustered using the same cluster variable as the original studies. The IRT-based estimates of impact and Delta tests were computed using the [https://github.com/peterhalpin/robustDIFCode is available at https://github.com/peterhalpin/robustDIF] package written in the  language. 

Figure <ref> shows the Delta statistic for each dataset. At the 95%-level, the estimates were significantly different from zero in 8 out of 34 datasets (23.5 %), although 3 of these were from a single study. Standard errors showed a strong association with sample size (r = -.59, using log(N students)) and a moderate association with number of items (r = - .36 using log(N items)), indicating that large studies with longer outcome measures were best suited to revealing statistical significance. In particular, two studies with large negative estimates and wide confidence bands (Datasets 18 and 20) had smaller sample sizes and relative few items. Using a more lenient 80%-level of confidence, 15 out of 34 (44.1 %) datasets showed some evidence that the naive estimate may be inconsistent for the true treatment impact. Also note that, although the Delta statistic took on both positive and negative values, the simulation studies reported above caution against interpreting the direction of the difference without taking further steps to ensure that the proportion of items that exhibit DIF is less than 1/2. This point is revisited in the Discussion.  




Figure <ref> shows how the absolute value of the Delta statistic was related to the naive estimates of impact, for researcher-developed (RD) and independently developed (ID) outcomes. In general, the discrepancy between the naive and robust estimates was larger when the naive treatment effects were larger. The independently developed measures generally showed smaller naive estimates (range = [-0.05, 0.22]) and the discrepancy with the robust estimates was also smaller (range = [0.01, 0.32]). By contrast, the researcher-developed outcomes tended to shower larger naive effects (range = [-0.12, 1.00])) that tended to be more discrepant from the robust estimates (range =[0.01, 0.57]). 



Regressing the naive estimates on the type of outcome revealed that researcher-developed assessments predict larger treatment effects (b= 0.30, SE = 0.10). This regression coefficient is in standard deviation units of the latent trait and is consistent with the research reviewed in the introduction to this paper. In a model that also included the absolute value of the Delta statistic (b = 1.53, SE = 0.32), the relationship with the type of outcome was substantially diminished (b = 0.16, SE = 0.09) and no longer significant at the 5% level. This result suggests that the well-documented relationship between reported effect sizes and the type of outcome measure may be largely due to systematic bias induced by DIF. These regression results accounted for dependence of Datasets within Studies by including a random effect for Study, and they used precision weights to account for sampling error in the naive estimates. However, when interpreting these results, it is important to keep in mind the small sample size (34) and that 5 out of only 11 independently developed outcomes were from a single study.
 
Figure <ref> shows the item-level effects. The datasets are shown in ascending order of the naive test-level effects, and the lighter data points indicate item-level effects that were down-weighted when estimating the robust test-level effect. The figure indicates substantial item-level treatment effect heterogeneity, which has already been documented using these data <cit.>. The standard deviation of the item-level effects was positively associated naive test-level effect (r = .41), showing that datasets that reported larger effects also exhibited more item-level variability. 

Some of the observed item-to-item variability is due to sampling error. However, some of this variability may also be due to idiosyncratic item-level sources of variation that do not reflect impact on the underlying construct (i.e., DIF). These potentially problematic item-level effects are represented by the lighter data points in the figure. In a total of 13 of 34 datasets (38.2%), at least 10% of assessment items were down-weighted to zero (i.e., were statistically different from the robust estimate at the .05 level). Moreover, the proportion of items that were down-weighted to zero was strongly associated with the naive estimate of impact (r = .63). However, it was also the case that many datasets (11 out of 34) had no items that were down-weighted to zero, and several of these were larger studies with relatively precise estimates. Together these results indicate that DIF with respect to treatment was pervasive but not ubiquitous in this sample of studies, and was especially apparent in studies that reported larger treatment effects. 





§ DISCUSSION


This paper contributes to a growing body of literature documenting the prevalence and importance of item-level heterogeneous treatment effects in program evaluation research <cit.>. Previous approaches to studying this phenomenon have focused on random-effects models whose identification constraints require omitting parameters that describe impact on the target construct <cit.>. Consequently, these models do not characterize the extent to which individual item-level treatment effects, or test-level treatment effects that aggregate over items, may be incompatible with impact on the target construct. The present paper addressed this issue by drawing on the psychometric distinction between impact and DIF <cit.>. Taking this approach, it was shown that DIF with respect to treatment status can cause the unweighted mean of assessment items to be inconsistent for the true impact on the target construct. Consequently, DIF is a potential source of systematic bias that may affect reported treatment effects. To address this issue, a Hausman-like specification test was proposed, which compares the naive (unweighted mean) estimate of impact to a more robust alternative. Pragmatically, a statistically significant test result indicates that reported treatment effects may be unduly dependent upon the specific items that make up the outcome measure, and therefore would not be expected to generalize to other measures that do not use those same items. 

Theoretical and simulation-based results show that the proposed test performs as intended when fewer than 1/2 of assessment items exhibit DIF. The simulations also suggested that, while the sign of the test statistic depends on the proportion of items that exhibit DIF, its magnitude will generally be non-zero when items with DIF are systematically biased in the same direction. Thus, in these situations, the proposed specification can be useful for inferring whether the naive estimates of impact are robust to item-level sources of variation, regardless of how many items exhibit DIF. This is an important advantage over traditional methods of DIF analysis, which can fail badly when a relatively large proportion of items (> 20%) exhibit DIF in the same direction <cit.>. While recent research has improved the performance of DIF detection methods when more items (< 50%) may exhibit DIF <cit.>, the present paper is the first to explicitly consider the implications for inferences about impact. 

An important shortcoming of the proposed test statistic is that, by itself, its sign is not a reliable indication of whether DIF leads to upwards or downwards bias in reported treatment effects. This point can be illustrated with reference to Figure <ref>. Focusing specifically on the bottom row (Dataset 26) we see that a small minority of item-level effects are positive outliers while the overall effect is slightly negative. This could be a small subset of items with a true non-zero effect (“wash-out”), or a large subset of items with a small negative effect. Unless we are willing to make the additional assumption that fewer than 1/2 of items exhibit DIF, the test statistic itself does not allow us to differentiate these two situations. Rather, the test only tells us whether conclusions about impact would differ depending on whether or not the positive outliers were included when reporting the overall treatment effect. If the reported treatment effect does not depend on this small subset of items, the matter can be concluded. However, if the reported treatment effect is strongly influenced by the positive outliers, it would be important to address this issue when interpreting the results of the study. Similar considerations apply in general, not just to this single example. 

This shortcoming also points to a clear direction for future research: developing open data practices <cit.> and statistical methodology <cit.> to study how item-level effects depend on pre-exising features of the items. Documenting and theorizing DIF that arises due to scientifically interesting item characteristics (e.g., content, difficulty) can present exciting opportunities to refine education theory. For example, research in language literacy has used item-level analysis to evaluate how treatment effects depend on the alignment between the intervention and item content to assess learning transfer <cit.>. Application of similar models in psychopathology show that the effect of selective serotonin re-uptake inhibitors (SSRIs) on patient-reported depression surveys is mostly driven by effects on mood rather than physical symptoms such as appetite, and these results align with the pathways by which SSRIs affect brain chemistry <cit.>. More generally, it may often be fruitful to pre-register item-specific hypotheses during the design of a study, or to consider them as ad hoc hypothesis that can nonetheless inform education theory. This can be useful for addressing issues related to treatment effect heterogeneity <cit.> as well as broader concerns about the role of measurement in evaluation contexts <cit.>. 

Our empirical example illustrated that DIF with respect to treatment can also play an important role in addressing ongoing concerns about the role of researcher-developed assessments in education sciences <cit.>. In this sample of studies, the average naive treatment effect was 0.30 SD larger for researcher-developed assessments as compared to independently-developed assessments (SE = 0.10). After after controlling for the discrepancy between the naive estimate and the robust estimate, this relationship was substantially reduced to 0.15 SD (SE = 0.09) and no longer significant at the 5% level. In pragmatic terms, this means that the well-documented relationship between reported effect sizes and the type of outcome measure was largely explained by systematic bias induced by DIF. As noted above, this result should be interpreted tentatively due to the small size of the empirical example (34 datasets) and because 5 out of only 11 independently developed outcomes were from a single study. In addition to replicating this finding in a wider range of studies, it is important for future research to investigate how the performance of the test can be improved in realistic settings, for example by using alternative specifications of the underlying psychometric models, model-free approaches based on resampling of items <cit.>, or alternative procedures. 

In summary, the proposed specification test can provide a clear signal to researchers, meta-analysts, and policy makers about whether reported treatment effects are unduly dependent on the specific items that make up the outcome measure. The test is straightforward to implement as a post-estimation step using standard IRT models that are widely available in general-purpose statistical software. Consequently, it is a relatively stand-alone procedure that can be implemented alongside other checks on internal validity (e.g., balance, attrition bias). Future research along these lines can help move education sciences beyond inherently subjective heuristics about who developed an assessment, and towards statistical methodology and empirical results that directly evaluate the sensitivity of treatment effects to the specific outcome measures used.



§ APPENDIX




 §.§ Derivation of item-level treatment effects
 

Models that can be represented using Equation glm are identified only up to an affine transformation of the latent trait <cit.>. In Equation glm this is addressed by standardizing the latent trait in each group. When the groups may differ in the distribution of the latent trait, an alternative way to identify the model is to (a) standardize the latent trait in only one of the groups, and (b) assume some of the item parameters are equal over groups, which serves to identify the scaling parameters in the second group. In the IRT literature, this two-part scaling procedure is referred to as the “common items non-equivalent groups" (CINEG) procedure for IRT-based scaling <cit.>. 

To obtain Equation delta using the CINEG procedure, part (a) is achieved by standardizing the latent trait in the treatment group (i.e., μ_1 = 0 and σ_1 = 1). For part (b), we can use the following relation


    a_i0θ_0 + b_i0 = a^*_i0θ^*_0 + b^*_i0


in which θ_0 = (θ^*_0- μ_0)/ σ_0. This relation is implied by Equation glm and the invariance of the model under affine transformation of the latent trait. Algebraic manipulation leads to 


    - μ_0 / σ_0 = (b^*_i0 - b_i0) / a_i0.


Since we have set μ_1 = 0 to identify the model, Equation scale1 can equivalently be written as 


    δ_0 = (μ_1 - μ_0 ) / σ_0 = (b^*_i0 - b_i0) / a_i0.


Equation scale2 relates impact on the latent trait to the item parameters. However, the “unscaled" item thresholds in the control group, b^*_i0, are not observable. To address this situation, one may assume that the thresholds are equal across groups (i.e. b^*_i0 = b_i1) for some items. This is equivalent to assuming that those items do not exhibit DIF on their thresholds. The assumption leads to 


    δ_0 = (μ_1 - μ_0 ) / σ_0 = (b_i1 - b_i0) / a_i0 = δ_i(ν),


which establishes the relation between Equations delta_0 and delta-i. 
Note that when an item exhibits DIF on its threshold (i.e. b^*_i0≠ b_i1), Equation scale2 remains valid but second equality in Equation scale3 no longer holds. It is also important to note that both Equation scale2 and scale3 hold regardless of whether the item slopes exhibit DIF. 




 §.§ Computational details of δ_R
  

Robust estimation of the test-level treatment effect has been implemented in the  package , which is used for the numerical examples reported in this paper. This software fine-tunes the computation of the robust estimator in a few ways, which are briefly summarized here. First, one may choose the tuning parameters in Equation k using V_0(δ̂_i - δ̂_R) rather than V_0(δ̂_i). This is possible because, under the null hypothesis that no item exhibits DIF, the V_0(δ̂_i - δ̂_R) depends only on the V_0(δ̂_i) <cit.>. Because δ̂_i and δ̂_R are positively correlated, V_0(δ̂_i - δ̂_R) ≤ V_0(δ̂_i), which results in more accurate false positive rates when flagging items with DIF in finite samples. 

A second point concerns the computation of V_0(δ̂_i). This asymptotic variance is a function of the individual item-level treatment effect, δ_i(ν) <cit.>. This is problematic because it implies that  DIF not only biases the point estimates δ̂_i but also their variance. However, under the null hypothesis δ_i(ν)= δ_0, one can substitute a consistent estimate of δ_0 for the individual item-level effects when computing the plug-in estimate V_0(δ̂_i). In particular, using a robust estimate such as median(δ̂_i) in place of the δ̂_i leads to a highly robust version of δ_R <cit.>. 

A third detail concerns precision weighting. Computing the robust estimator using u^*_i = (δ̂_i -δ) / V_0(δ̂_i) in place u_i = δ̂_i -δ in Equations w through k leads asymptotic efficiency of the resulting estimator under the null hypothesis that no items exhibit DIF <cit.>. However, the robust estimator is not efficient when one or more items exhibit DIF.  

Finally, it is important to note that redescending loss functions, including the bi-square function used to compute δ̂_R, can result in local “bad” solutions. In particular, Lemma 1 of the Section <ref> shows that, if any items exhibit DIF, local solutions will necessarily arise as n →∞ when using the tuning parameters in Equation k. Thus, when many items exhibit DIF, different starting values (i.e., choices of δ in Equation w) may lead to different solutions when computing δ̂_R. In practice, this can be dealt with by comparing solutions using multiple starting values. If the different starting values converge to different solutions, the minimum of the solutions is taken as the estimated value of δ_R. Because estimation problem is unidimensional, this procedure is not computationally complicated or expensive. Moreover, because local solutions can arise only due to DIF, their presence can serve as a useful diagnostic procedure.  

 


 §.§ Proof of Theorem 1
 
The set-up for the proof is adapted from Huber's () discussion of the finite sample breakdown of M-estimators of location. To simplify notation, x_i, i = 1 … m, will denote a finite collection of “data points,” which correspond to the item-level treatment effects. The “estimator” in question is defined as the global minimum of 


    R(μ) = ∑_i=1^m ρ(x_i - μ),
 

where ρ = ∫ψ(u) du is assumed to exist. Other than the change of notation, the loss function representation in Equation R is equivalent to the weighted mean representation in Equation delta <cit.>

To address the case of redescending M-estimators, it is assumed that ρ satisfies the following assumptions: 

 

  A1 ρ: ℝ→ [-1, 0] is continuous.

  A2 ρ(0) = -1 is the unique minimum.

  A3 There exists a value of k > 0 such that ρ(u) = 0 for |u| > k. 
 

These assumptions include the (appropriately rescaled) bi-square function defined in Equation bsq as well as other commonly used redescending M-estimators <cit.>.  

The proof appeals to Theorem 5.7 of van der Vaart (), which states two conditions that are together sufficient to ensure the consistency of M-estimators. One condition is that the population loss function must have a unique global minimum, say μ_0. The second condition is that the sample loss function must uniformly converge in probability to the population loss function. Together these conditions imply that the minimizing argument of the sample loss function converges in probability to μ_0. 

The following two lemmas serve to establish these two conditions for δ̂_R. Lemma 1 describes a population loss function that is computed using a finite number of items, m. Lemma 2 shows that, as the sample size n increases without bound, the loss function of δ̂_R uniformly converges in probability to the loss function described in Lemma 1. The proof of Theorem 1 follows as an immediate corollary of the two lemmas. 

  Let x_i, i = 1, … m, denote a finite collection of data points and let S denote the set of unique values of x_i. For each s ∈ S let m_s denote the number of data points such that x_i = s and let ϵ be the smallest distance between any two values s_i, s_j ∈ S. If we choose k ≤ϵ / 2, then R(s) = -m_s is a local minimum of R for each s ∈ S, where R is defined by Equation R and assumptions A1-A3.  
 

Proof of Lemma 1. For each s ∈ S, A2 implies that ρ(x_i - s) = -1 for x_i = s, while A3 and the choice of k imply that ρ(x_j - s) = 0 for x_j ≠ s. Thus, R(s) = -m_s is the minimum of R (by A2) on the interval [s - ϵ/2, s + ϵ/2] (by A3 and choice of k). □ 

 Let R(δ; ν̂) denote the sample loss function of the plug-in estimator δ̂_R =  δ_R(ν̂) defined via Equations delta, w, bsq, using tuning parameters k_i^* = max{k_i, ϵ/2} with k_i defined in Equation k and ϵ defined in Lemma 1. Then R(δ; ν̂) uniformly converges in probability to a function that satisfies Lemma 1. 
 

Proof of Lemma 2. Uniform convergence of R(δ; ν̂) to R(δ; ν) follows from A1 and applying the continuous mapping theorem to the MLEs ν̂ <cit.>. From Equation k, it is apparent that k_i → 0 as n →∞. Thus, k^*_i →ϵ/2 so that tuning parameter of R(δ; ν) deterministically satisfies the condition stated in Lemma 1. □


An immediate corollary of Lemma 1 is that the population loss function will have a unique global minimum when there is a unique modal (most frequent) value of the item-level treatment effects. This is the condition stated in Theorem 1. As noted in the main text, Lemma 2 utilizes a “modified” tuning parameter that simplifies the proof, but does not affect the computation δ̂_R in finite samples. More subtle proofs may be available. Together with Theorem 1, the two lemmas imply that δ̂_R satisfies conditions stated in Theorem 5.7 of van der Vaart (). 




 §.§ Derivation of Equation var-Delta
 

The asymptotic distribution of the plug-in estimators described in the main text can be derived using the Delta method <cit.>. This approach is applicable whenever a continuously differentiable function g: ℝ^K →ℝ is applied to the MLEs, ν̂∈ℝ^k. The relevant result is


    √(n) (g(ν̂ ) - g(ν) )d→ N(0, ∇ g(ν) ^T   V(ν̂ )  ∇ g(ν ))


where ∇ g is the K-dimensional graident vector of g and V(ν̂ ) is the asmptotic covariance matrix ν̂. The following derivation focusses on g = δ_R(ν̂) defined using Equation delta with the weights given by Equations w through k. Parallel results for δ_P and Δ are seen to follow directly. 

Start by writing


    ∂/∂ν_kδ_R(ν)     
     = ∑_i = 1^m∂/∂ν_k w_i(ν) δ_i(ν)


Note δ_i(ν) depends on only the parameters of item i but the weights w_i(ν) depend on the parameters of all items. Thus, if ν_k ∈ν_i we require the product rule to compute the partial derivatives in Equation partial-1, but if ν_k ∈ν_r for r ≠ i we require only the partial derivatives of the weights. These considerations lead to the following expression, in which the dependence on ν has been made implicit to simplify notation: 


    ∂/∂ν_kδ_R(ν) 
       =  w_i∂/∂ν_kδ_i
     + δ_i∂/∂ν_k w_i
     + ∑_r ≠ iδ_r∂/∂ν_k w_r.


Computing the necessary derivatives for the second term of the sum in Equation partial-2 gives


    δ_i∂/∂ν_k w_i    = δ_i  (1 - w_i)  /∑_s w̃_s∂/∂ν_kw̃_i


where w̃_i denotes the non-normed weights in Equation w. The third term in the sum is 


    ∑_r ≠ iδ_r∂/∂ν_k w_r  = - ∑_r ≠ iδ_r w_r/∑_s w̃_s∂/∂ν_kw̃_i .


Adding Equations partial-3 and partial-4 gives


    δ_i - ∑_r = 1^m  δ_r w_r/∑_s w̃_s∂/∂ν_kw̃_i  
    =  δ_i - δ_R/∑_s w̃_s∂/∂ν_kw̃_i


Equation partial-5 is further simplified by using the result


    ∂/∂ν_kw̃_i    =  ψ'(δ_i - δ) - w̃_i/δ_i - δ∂/∂ν_kδ_i.
 
 
Substituting Equation partial-6 into Equation partial-5 gives


    δ_i - δ_R/δ_i - δψ'(δ_i - δ) - w̃_i/∑_s w̃_s∂/∂ν_kδ_i.


In the above expression, recall that δ_R is a function of the item parameters whose gradient we desire. The quantity δ is a “preliminary value" of the target scaling parameter that is treated as a constant when computing δ_R. In practice, estimation via IRLS continues until the difference between these two values is arbitrarily small, leading to δ = δ_R in Equation partial-7. The simulation studies reported above confirm that analytical expressions for variance of δ_R(ν̂) and Δ(ν̂) obtained using this substitution correspond closely to the values obtained by directly computing their variances over simulated datasets. 

Using δ = δ_R in Equation partial-7 and substituting the resulting expression for the last two terms in Equation partial-2 yields  


    ∂/∂ν_kδ_R(ν)     = 
    w_i∂/∂ν_kδ_i  + ψ'(δ_i - δ) - w̃_i/∑_s w̃_s∂/∂ν_kδ_i
        =  ψ'(δ_i - δ)/∑_s w̃_s∂/∂ν_kδ_i.


Finally, using the notation


    v_i ≡ψ'(δ_i - δ)/∑_s w̃_s,


leads to the following expression for the asymptotic variance of δ_R(ν̂):  


    V(δ_R(ν̂)) = ∑_i = 1^m [v_i∇δ_i(ν)]^T  V(ν̂ ) [v_i∇δ_i(ν)]  =  ∑_i = 1^m v_i^2 V( δ_i(ν̂ ))


For the precision-weighted versions, treating the precision weights as ancillary constants <cit.> leads to good finite sample performance, as illustrated in the simulation studies reported in the main text. For g = δ_P, this leads to using w_Pi in place of v_i in Equation var-delta. For the precision-weighted robust estimator, Equation v becomes


    v^*_i ≡ψ'(δ_i - δ) / V(δ̂_i)/∑_s w̃_s / V(δ̂_s).


For the differences between the precision weighted estimators, the Delta method leads to substituting v_i = (w_Pi- v^*_i) in Equation var-delta, which gives in Equation var-Delta.  

All that remains is to obtain the partial derivatives ∂δ_i / ∂ν_k, where δ_i was defined in Equation delta-i. The non-zero elements are listed below: 


    ∂/∂ a_i0δ_i  = - δ_i/a_i0      ∂/∂ b_i0δ_i =  -1/a_i0      ∂/∂ b_i1δ_i =   1/a_i0.


The derivation presented in this section applies to a relatively general choice of item-level treatment effects – this simply requires using the relevant partial derivatives in place of Equation partial-delta-i. The derivation uses Equation w but not Equation bsq, so it applies to other robust test-level treatment effects for which the function ψ is continuously differentiable. 

Interestingly, the derivation presented here leads to a slightly different result than provided by Halpin () and Wang et al. () Those papers also used the Delta method to obtain the variance of IRT scaling functions, but chose the function g to be estimating equation ∑_i ψ(u_i) = 0, which implicitly defines the target parameter δ. By contrast, the present paper instead chose g to be the weighted mean in Equation delta, which explicitly defines the target parameter.  Comparing the weights v_i in Equation v with the corresponding expression from Halpin (2024; Equation (32)), reveals the main difference: Using the estimating equations, we must divide by the sum of the ψ'(u_i) / u_i rather than the sum of the w̃_i = ψ(u_i) / u_i. This difference is important when ψ is non-monotone (i.e., for redescending functions), in which case ψ' can take on positive and negative values, so the sum can be equal or close to zero, leading the weights to diverge to infinity. By contrast, w̃_i is non-negative under the (usual) assumption that ψ is odd, and by convention, it is set to 1 when u_i = 0. Consequently, the weights v_i in Equation v do not diverge (although they may be negative due to the numerator). Unreported simulation studies conducted by the author show that these two approaches generally lead to similar results, except when the δ̂_i are symmetrically distributed around δ̂, in which case the approach described in this paper is much more accurate. 


apalike

 



