Paper Title: Unsupervised Pairwise Causal Discovery on Heterogeneous Data using Mutual Information Measures.


          








A]Alexandre Trilla,
B]Nenad Mijatovic

[A]Alstom, Santa Perpètua de la Mogoda, Barcelona, 08130, Spain
	
[B]Alstom, Saint Ouen, Paris, 93482, France

§ ABSTRACT 

A fundamental task in science is to determine the underlying
causal relations because it is the knowledge of this functional
structure what leads to the correct interpretation of an effect 
given the apparent associations in the observed data.
In this sense, 
Causal Discovery is a technique that tackles this
challenge by analyzing the statistical properties of the 
constituent variables.

In this work, we target the generalizability of the 
discovery method by following a reductionist approach that 
only involves two variables, i.e., the pairwise or bi-variate
setting.

We question the current (possibly misleading)
baseline results on the basis that they were obtained through 
supervised learning, which is arguably contrary
to this genuinely exploratory endeavor. In consequence, we
approach this problem in an unsupervised way, using robust Mutual
Information measures, and observing the impact of the different variable
types, which is oftentimes ignored in the design of solutions.
Thus, we provide a novel set of standard unbiased results
that can serve as a reference to guide future discovery tasks 
in completely unknown environments.



Causal Discovery, Functional Causal Model, Unconditional Independence Test, Mutual Information, Unsupervised Learning, Heterogeneous Data








§ INTRODUCTION

Causal Discovery methods are able to identify the causal structure
from the joint distribution of the data by introducing assumptions 
that restrict the model of their generating process <cit.>.
In a multivariate setting, the traditional constraint-based and 
score-based methods exploit conditional independence 
relationships in the data <cit.>.
These approaches have found a great deal of success in 
challenging environments such as biology and the Earth
sciences <cit.>. Nevertheless,
they do not necessarily provide complete causal information 
because they output Markov Equivalence Classes, i.e., a set of
causal structures that satisfy the same conditional independence
statements. Moreover, they cannot handle isolated cause-effect
settings that lack the diversity of other variables for the 
conditional tests. Therefore, the single reduced cause-effect association,
also known as the pairwise or bivariate scenario, constitutes an 
essential building block of more complex causal structures. 

To distinguish cause from effect in this pairwise setting, 
one needs to find a way to capture the asymmetry between the
two variables <cit.>. In this sense,
computational methods based on properly defined Functional Causal 
Models (FCMs) are able to distinguish different structures in the 
same equivalence class.
A FCM represents the effect variable Y as a function of its direct
causes X and some noise term ε, i.e., 
Y = f(X, ε), where ε is
independent of X. Thanks to the restricted functional classes,
the causal direction between X and Y is generally identifiable because
the independence between the noise and the cause holds
only for the true causal direction and is violated for the wrong
direction <cit.> (unless the variables are jointly Gaussian,
which render the orientation unidentifiable). Additionally, the
identifiability of pairwise FCM generalizes to the identifiability 
of multivariate FCM <cit.>.

In this work, we argue that some reference benchmark results for pairwise 
causal discovery could be misleading because they tackle this genuinely 
exploratory objective as a classical supervised classification problem, 
inducing them to possibly learn specific contextual details beyond causality 
that tend to overestimate their actual performance <cit.>.
Moreover, there exists a tight relationship between the nature of data 
and the specific tools from statistics that could also accrue the estimation 
of skewed results, which was not originally addressed. Consequently, we 
propose a reevaluation of the benchmark driven by the heterogeneity 
of the data and also by following an unsupervised 
learning approach. Our method leverages linear functional causal models and 
combines different unconditional independence tests based on robust
Mutual Information measures.
The paper is organized as follows: Section <ref>
introduces the fundamental concepts for pairwise causal
discovery, Section <ref> describes the enhanced method
proposed in this work, Section <ref> shows the novel
benchmarked results, Section <ref>
discusses the approach from a causal perspective, along with its
limitations, and
Section <ref> concludes the article.






§ BACKGROUND
 

This section reviews the fundamental concepts that support the
discovery of causality in the bivariate or pairwise setting.



 §.§ Identifiability of a Linear Model with Non-Gaussian Noise

The fundamental principle that enables the identification of cause
and effect lies in the asymmetry of the association between them: 
the causal direction, i.e., from cause to effect, is functionally 
simpler than the anticausal one, i.e., from effect to cause, which
requires a more complex application <cit.>.
Thus, a given model with limited capacity will be able to 
faithfully capture the expressiveness 
of only one of these two causal orientations.

Linear additive noise models assume that the effect Y is a linear 
function of the cause X plus a noise term ε that
is independent of the cause, formally defined as: Y = bX + ε.
The linear FCM is learnable if at most one of X and ε
is Gaussian <cit.>. Pairwise discovery approaches such as this one are regarded to be very flexible for identifying causal models in the general case because they don't require a third variable for conditioning.
In specific scenarios such as multivariate time series, they can even
improve the performance of traditional constraint-based
approaches such as the Peter-Clark algorithm and others that
utilize Granger causality <cit.>.

The linear model with non-Gaussian noise can be estimated from 
(unconfounded) observational data by exploiting the inherent 
asymmetry between cause and effect through a regression as 
follows: for both directions (i.e., causal and anticausal), the 
FCM is fit; then, the amount of association between the estimated 
noise term (i.e., the regression residual) and the hypothetical 
cause is computed; finally, the direction assignment which gives an 
independent noise term is considered plausible <cit.>.

Obviously, the cornerstone of this regression-based method is the
measure of dependence between the regression residual and its
associated potential cause. The next section 
describes a solution to estimate the strength of this association.





 §.§ Mutual Information as a Measure of Association

The amount of association between the regression residual RR and
the hypothetical cause HC signals the causal direction. However,
these variables are uncorrelated by construction of the regression. To 
properly quantify this dependence, the Mutual Information (MI) measure 
provides a reliable indicator, which is defined as


    MI(RR,HC) = ∑_RR,HC P_RR,HClog ( P_RR,HC/P_RR  P_HC )  ,


where P_RR,HC represents the (discrete) joint probability between
RR and HC, and P_RR and P_HC represent their marginal
distributions, respectively.

For illustrative purposes, the following variable dependencies 
are defined. First, a random Uniform noise sample U is
independently assigned to Z, X_ind and Y_ind. Then,
the following structural variable associations are created: 
Y_dep← X_ind + Z, X_conf← X_ind + Z,
and Y_conf← Y_ind + Z, which yield the subsequent relationships:


  Causal Between X_ind and Y_dep: X → Y

  Anticausal Between Y_dep and X_ind: Y ← X

  Independent Between X_ind and Y_ind: X  Y

  Confounded Between X_conf and Y_conf: X ↔ Y


Figure <ref> shows the distribution of MI between
the regression residual and the hypothetical cause considering
the preceding types of structural variable associations for a random
sample of 1000 instances. The different density plots
show how the causal orientation can be reliably recovered using
MI when the model assumptions are respected. Note that
the overlap between Causal and Independent structures 
will be disambiguated when both experiments are conducted regarding 
the different causal direction hypotheses.
Finally, MI is also able to detect latent confounding, which is 
oftentimes assumed (and required) not to occur for 
discovery <cit.>, i.e., the causal 
sufficiency principle <cit.>. This feature 
adds value to the robustness of the MI indicator.



While the distribution of MI is a useful tool to visualize the
different causal structures, there is still a missing criterion to 
make a classification decision in order to construct the underlying
causal structure from the data. The next section addresses this point.




 §.§ Pearson's χ^2 as an Unconditional Independence Test


In a real-word scenario, one cannot expect to get a null measure
of association because of sampling noise, the sample size, etc.
In this case, the use of an (unconditional) independence
test is the proper way to handle this situation. In general,
this is known as the REgression with Subsequent Independence Test
(RESIT) procedure <cit.>. 
Once the independence test is in place, a confidence interval needs 
to be observed to obtain actionable results. To this end, Pearson's 
χ^2 test is here introduced as a fundamental 
method to inform the decision process <cit.>.

The Pearson's χ^2 test is a statistical hypothesis test used 
in the analysis of contingency tables for categorical variables.
It is used here to determine whether there is a statistically 
significant difference between the expected frequencies E
for independence (i.e., the marginal probabilities)
and the observed frequencies O in one or more categories i
of the contingency table (RR,HC), formally defined as

    χ^2(RR,HC) = ∑_i ∈ (RR,HC)(O_i - E_i)^2/E_i .


The test is valid as long as its statistic is χ^2 distributed 
under the null hypothesis.
Note that χ^2(RR,HC) is in its essence related to the MI measure of 
association as is used for the discovery of causal structure.

While the χ^2 test is most suitable for discrete variables <cit.>,
a general
causal discovery method should be able to seamlessly deal with 
continuous variables. Typically, a uniform grid is applied to discretize
a real-valued space prior to conducting the test.
Following the former illustrative example, Table <ref>
shows the p-values for the different causal structures.
Once a threshold is introduced through the confidence interval,
e.g., p<0.05 for a given number of degrees of freedom related to
the resolution of the grid,
passing the test becomes the rule for making further decisions.
Note that in all cases, there is a unique combination of results
for the two hypothesized causal directions that identifies the true
underlying causal structure.
Also note that the threshold may be adjusted to require further decision strength because the closer the variables get to a Gaussian distribution, the harder it is to distinguish the direction of causation <cit.>.



While these results
may be sufficient as an indication, they may also have some 
limitations for discovering causal associations between numerical variables.
The next section addresses this shortcoming and describes
an enhanced solution for this type of variables.







 §.§ Total Information Coefficient as a Robust Independence Test



The Total Information Coefficient (TIC) is a robust independence
test based on MI for real-valued variables 
that features the two important heuristic properties of generality and 
equitability <cit.>, which are defined as follows:



  Generality With sufficient sample size the statistic 
should capture a wide range of interesting associations, not limited 
to specific function types. 

  Equitability The statistic should give similar scores to 
equally noisy relationships of different types.


For a pair of (RR,RC) variables, the TIC algorithm applies a search
procedure to partition their joint probability function and find the
grid with the highest induced MI.
TIC operates by summing over optimal grids G on the joint 
density distribution of the two variables, such that it exhibits 
a stronger power 
against independence <cit.>, formally expressed as

    TIC(RR,HC) = ∑_G MI((RR,HC)|_G)/log∥ G ∥ ,

where ∥ G ∥ denotes the minimum of the number 
of rows and columns of G. This can be viewed as a regularized 
version of MI that penalizes complicated grids.
Figure <ref> shows qualitatively the impact of the
optimum grids G.




The performance of TIC has been positively rated with
other successful approaches for causal discovery dealing with
continuous variables, such as the
Hilbert-Schmidt independence criterion <cit.>.
Finally, an additional motivation for
considering TIC in this work is to increase the odds of success
in case of model hypothesis violations, i.e., the functional form
of the relationship Y = f(X), which is crucial for causal
identification.






§ METHOD


In the pursuit of a general approach to causal discovery
combining several methods <cit.>,
one common point of success is found in the use
of information-theoretic measures, such as Mutual Information,
to quantify the amount of regularity in the 
data <cit.>. 

However, the nature of the variables, i.e., discrete or continuous, 
can be problematic in an heterogeneous context if the statistical 
methods are confused <cit.>.


Moreover, supervised learning approaches are excluded from the
general objective because these methods do not yet work as standalone 
techniques for causal learning <cit.>. Therefore,
the desideratum in causal discovery is to have methods that work on a 
broad range of problems under different conditions with relaxed
assumptions <cit.>, ideally showing a 
certain degree of robustness regarding violations of the model 
hypothesis <cit.>.


In this work we propose a general method to discover the causal 
structure in heterogeneous data based on a bivariate linear FCM by
implementing a flexible RESIT procedure, see 
Section <ref>, where the
unconditional independence test, e.g., χ^2 or TIC,
also see Section <ref>,
is driven by the nature of the variables involved. The proposed 
strategy is described with comments in Algorithm <ref>.
Note that the orientation decision rules follow from the evidence 
given by the illustrative setting in Section <ref>, 
which is regarded as self-evident.
Such explicitly-stated logic rules are meant to increase the 
interpretability and explainability of the proposed method.
Also note that discrete variables are assimilated to real-valued variables for computing the regression residuals on which the RESIT method is based. Finally,
since our contribution targets the problem of applying one single discovery technique on data with different types of variables, and we focus our effort at the integration level rather than at the fundamental level, we refer the interested reader to the references for the specific methods being integrated for further details about their comparison to other methods from the literature.
We rely on the advice from Peters, Reshef, and 
colleagues <cit.>, to select the best methods that
we integrate in RESFIT in order to cover a broad range of techniques, 
and we provide an analysis on how these separate tools perform when
they are put together.















§ RESULTS


This section describes the reference benchmark dataset that was used
to evaluate the introduction of the flexible independence test selection, 
the results that were obtained, and the tools for the implementation 
of the research.



 §.§ Reference Benchmark Dataset



The “ChaLearn cause-effect pair (SUP2)” is taken for reference as
the recommended benchmark dataset to evaluate the performance
of pairwise causal discovery <cit.>.
It comprises pairs of artificially generated dependent 
variables with different types 
(numerical, categorical and binary) for the full 
causal discovery task (orientation, independence and confounding).
The dataset features a balanced number of unique values across all 
classes and includes around 6000 pairs. In terms of data type balance,
the majority is comprised of numerical and mixed variable pairs. Finally, 
the average median length of an instance is around 2000 values.



 §.§ Performance Scores


The accuracy classification score, i.e., the total rate of correct
predictions statistically given by the overall amount of
true positives and true negatives,
is used in this research following the previous benchmark
evaluation approaches <cit.>, yielding a baseline around 
65±2% for the supervised learning setting <cit.>.
To further assess the stability of our scores in the unsupervised
scenario, statistical
bootstrapping is introduced in line with this former work.
Moreover, Gaussianity is asserted with the Lilliefors normality
test <cit.>,  and the main descriptive
statistics are extracted despite the low amount of available performance
samples, well under 30, which are commonly
required to obtain reliable estimations <cit.>.
Finally, in terms of statistically significant comparisons, the t-test has been
used <cit.>.
Table <ref> shows the results of the experiments.
Note that they are presented separately by data type and independence
test, whereas the proposed RESFIT algorithm automatically integrates
them. This is done for illustrative purposes and for enriching 
the ensuing discussion.




The first straightforward conclusion that the results show is that,
in all cases, there is a statistically significant difference between
the accuracy averages for the two unconditional independence tests
within each data type stratum. Therefore, introducing a selection 
action on the test function is expected to impact 
on the discovery of causal associations.
Additionally, TIC yields a slightly smaller variance in all
the scores. Finally, when numerical data types are present,
the standard deviation in accuracy drops an order of magnitude.

The code for this research is available 
here[Code repository: https://github.com/atrilla/ccia24].
The next section discusses the global results with respect to the
perspective of a causal effect, and addresses the limitations
of the proposed approach.





§ DISCUSSION


This section delves into the finer details of the results that
were obtained, and sheds light on the actual value introduced by
the flexibility in selecting the unconditional independence test
for causal discovery.



 §.§ Average Causal Effect of the Flexible Test Selection


To properly quantify the impact of introducing the flexibility
on the unconditional independence test, this section 
treats this selection action as a “treatment” variable X 
and studies its impact on the “outcome” discovery accuracy
score Y.
The Average Causal Effect (ACE) of X → Y is
expressed using the following counterfactual notation 


    ACE =    𝔼 [Y_1 - Y_0] = 𝔼 [Y_1] - 𝔼 [Y_0] = 𝔼 [Y(X=1)] - 𝔼 [Y(X=0)] 
    
    	=    𝔼 [Y|X=1] - 𝔼 [Y|X=0]  ,


where Y_x refers to the value the Y accuracy result would have
if X was set to x, i.e., Y(X=x). Here, X
could take the values x=1 to indicate
flexibility of independence test, and x=0 to indicate no
preference (i.e., the null random choice). The conditionals that
eventually follow are the values that are actually observed in the results.
Note that all these expected quantities can be exactly calculated 
because the different experiments can be conducted on the same data
(also dispelling any doubts about latent confounding).
This setting is thus not subject to the fundamental problem of 
causal inference where only one of the potential outcomes
can be observed <cit.>.

Equation (<ref>) is shown to be an unbiased estimator for the
ACE <cit.>. For computing the expectations in the
causal effect comparison, a weighted mixture of Gaussian random
variables is required.
The quantity of the reference potential outcome 
𝔼 [Y_0], where no smart selection occurs, is given by
the arithmetic mean value of the total results, and it yields
an accuracy score of 0.3531±0.0357. Alternatively, the potential outcome
𝔼 [Y_1], where the smart flexible test selection is introduced, 
is given by the average of the best performance results among 
the different variable
types, i.e., a sum weighted by their balance in the dataset,
and it yields a value of 0.3866±0.0690. Therefore, the ACE
is of 3.36%, and this estimated difference is statistically significant.
However, the absolute results are far from the 65% baseline, which
suggests that the original supervised approach may have leaked
statistical patterns beyond causation. Although the unsupervised
learning approach does not seem to offer any pragmatic quantifiable
advantage, we believe that its adoption is a must for tackling the
inherently hard exploratory objective of causal discovery and avoid
any qualms with regards to learned biases.




 §.§ Limitations

The realization that TIC, which was conceived for
numerical data, performed better on discrete data is surprising.
Also, χ^2, which is defined for discrete data, performed 
better on numerical data. This counterintuitive behavior 
where theory and practice disagree may
suggest that the independence tests that were used are subject 
to some inherent limitations such as the sample size. 
We regard the shortage of samples to be the main limitation
of our approach on real-world data applications such as the
Tuebingen dataset <cit.>.

Finally, while the approach that we propose is able to welcome
any testing technique, we selected RESIT flexibly paired with TIC and 
χ^2 following the advice of Peters, Reshef, and 
colleagues <cit.>. To the best
of our knowledge, this limited choice should be sufficient to 
generally cover the spectrum of approaches.





§ CONCLUSION



Environments with heterogeneous data pose challenging questions to
the discovery of causal structure, and leveraging one single method may lead to erroneous results in general.
In this work, we propose an
unsupervised pairwise approach using linear functional causal 
models and different unconditional independence tests based on 
Mutual Information measures, the selection of which 
is driven by the nature of data and the empirical results 
from a reference benchmark.
The introduction of this independence test selection flexibility is
estimated to have a positive and statistically significant average
causal effect over 3% in accuracy.
These scores may establish the first 
standard baseline for this kind of flexible focus, but more
research is needed because the limitations of the
data size and the statistical techniques can result in 
counterintuitive results.







vancouver





 Paper Title: An overview of methods for receiver operating characteristic analysis, with an application to SARS-CoV-2 vaccine-induced humoral responses in solid organ transplant recipients.







An overview of methods for receiver operating characteristic analysis, with an application to SARS-CoV-2 vaccine-induced humoral responses in solid organ transplant recipients
    Nathaniel P. Dowd nathaniel.p.dowd@vanderbilt.edu 

       Department of Biostatistics

       Vanderbilt University

       Nashville, TN 37203, USA
       Bryan Blette bryan.blette@vumc.org 

       Department of Biostatitics

       Vanderbilt University Medical Center

       Nashville, TN 37203, USA
       James D. Chappell jim.chappell@vumc.org

       Division of Pediatric Infectious Diseases

       Vanderbilt University Medical Center

       Nashville, TN 37232, USA
       Natasha B. Halasa natasha.halasa@vumc.org

       Division of Pediatric Infectious Diseases

       Vanderbilt University Medical Center

       Nashville, TN 37232, USA
       Andrew J. Spieker andrew.spieker@vumc.org 

       Department of Biostatitics

       Vanderbilt University Medical Center

       Nashville, TN 37203, USA
       
    August 5, 2024
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 

Receiver operating characteristic (ROC) analysis is a tool to evaluate the capacity of a numeric measure to distinguish between groups, often employed in the evaluation of diagnostic tests. Overall classification ability is sometimes crudely summarized by a single numeric measure such as the area under the empirical ROC curve. However, it may also be of interest to estimate the full ROC curve while leveraging assumptions regarding the nature of the data (parametric) or about the ROC curve directly (semiparametric). Although there has been recent interest in methods to conduct comparisons by way of stochastic ordering, nuances surrounding ROC geometry and estimation are not widely known in the broader scientific and statistical community. The overarching goals of this manuscript are to (1) provide an overview of existing frameworks for ROC curve estimation with examples, (2) offer intuition for and considerations regarding methodological trade-offs, and (3) supply sample R code to guide implementation. We utilize simulations to demonstrate the bias-variance trade-off across various methods. As an illustrative example, we analyze data from a recent cohort study in order to compare responses to SARS-CoV-2 vaccination between solid organ transplant recipients and healthy controls.




  Nonparametric, Parametric, Receiver Operating Characteristic, Semiparametric, SARS-CoV-2, Software, Tutorial




§ INTRODUCTION


Receiver operating characteristic (ROC) analysis is a tool to summarize the extent to which a numeric measure accurately distinguishes between groups. ROC methodology has historical roots in electrical engineering as a method to detect enemy objects in battlefields, but became heavily associated with assessment of diagnostic utility shortly after its development (). In clinical practice, it is often of interest to determine whether a diagnostic test based on some numeric measure has the ability to discriminate between two states of health. In particular, it is often a goal to develop classification measures that help reduce the invasiveness and/or costs associated with diagnostic tests while maintaining accuracy of classification. For example, a “gold standard” test in the diagnosis of prostate cancer as determined by biopsy is invasive and impractical as a routine clinical screening measure; the prostate-specific antigen is a biomarker that has been used for prostate cancer detection (). Numerous examples of ROC analysis have been published, and its use is still widespread in the modern era (). Summary measures associated with ROC analysis have also long been used to compare classification ability across machine learning methods ().

More recently, researchers have more broadly recognized stochastic ordering as a scientifically relevant measure in its own right to compare groups—including in contexts outside of diagnostic tests (). Proper procedures for model selection, implementation, inference, and interpretation are less widely known in the statistical and scientific community as compared to, for example, methods to compare means across population strata ().

Parametric, nonparametric, and semiparametric approaches to ROC estimation have been characterized in the literature, largely separately (). A bias-variance trade-off is to be expected when comparing estimators across these classes (with more structure generally offering greater efficiency under correct assumptions, but introducing bias under assumption violations), though the nature of that trade-off in the space of ROC methods is not as well explored. Further, inference regarding the ROC curve is deceptively simple due to the fact that both the x- and y-coordinates of the overall curve are estimated. Subtleties and nuances regarding the fundamental geometry of ROC-based methods, as well as considerations surrounding estimation, merit more explicit characterization and further illustration. The fundamental goal of this manuscript is to address this need.

Throughout this manuscript, we anchor our illustrations using two foundational examples for ease of characterizing various trade-offs: exponential methods and normal methods. The remainder of this manuscript is organized as follows. In Section 2, we provide definitions and an overview of the ROC curve and considerations regarding its geometry. In Section 3, we discuss and provide sample R code for estimation procedures associated with the ROC curve, including related summary measures. In Section 4, we describe and present the results of various simulation studies to highlight the trade-offs between approaches. In Section 5, we apply several methods to a recent study seeking to characterize immune responses to SARS-CoV-2 vaccination in solid organ transplant (SOT) recipients. Finally, we conclude with a discussion of our findings in Section 6.



§ THE ROC CURVE AND ITS GEOMETRY




 §.§ Notation


Throughout this manuscript, we let i = 1, …, n_0 index independently sampled observations from a reference distribution with outcomes Y_01, …, Y_0n_0, each having common cumulative distribution function (CDF) F_0(t) = (Y_0 ≤ t), survivor function S_0(t) = 1 - F_0(t), and quantile function F_0^-1(p). Analogously, we let j = 1, … n_1 index independently sampled observations from a comparator distribution with outcomes Y_11, …, Y_1n_1, each having common CDF F_1(t), survivor function S_1(t), and quantile function F_1^-1(p). Note that in cases where F_0 and F_1 are not invertible (e.g., associated with discrete-valued outcomes), F_0^-1(·) and F_1^-1(·) denote generalized inverses (for instance, F_0^-1(p) = inf{ t : F_0(t) > p}). For ease of notation, we let n = n_0 + n_1 denote the combined sample size, and we will often refer generically to the measure of interest as Y.

The utility of the measure Y as a classifier is most easily described first at a particular cut-off point, c ∈supp(Y). The true positive rate (TPR) marks the proportion of observations in the comparator group correctly classified at that cut-off; the false positive rate (FPR) marks the proportion of observations in the reference group incorrectly classified. The graph of an ROC curve summarizes these two complementary classifying features of the measure Y across all possible cut-off points, and is defined as a plot of the FPR (i.e., one minus the specificity) at a particular cut-off point on the x-axis and the TPR (i.e., sensitivity) at that cut-off on the y-axis. The ROC curve is a monotone, surjective function : [0, 1] ⟶ [0, 1], so that p, the FPR, denotes the input and (p), the TPR, denotes the output ().



 §.§ Conventions and symmetries


To characterize the ROC curve as a function of observable data, we must specify whether higher or lower values of Y signify a less desirable outcome. In many settings, scientific knowledge will provide a researcher with this information a priori: for example, higher hemoglobin A1c (HbA1c) is generally known to be less desirable in patients with type 2 diabetes mellitus (), and lower CD4^+ T cell count is known to be less desirable in patients with human immunodeficiency virus (). When lower values of Y are considered less desirable, the ROC function is best defined as (p) = F_1(F_0^-1(p)), for 0 ≤ p ≤ 1. On the other hand, when higher values of Y are considered less desirable, the ROC function is instead best defined as (p) = S_1(S_0^-1(p)). In either case, the ROC curve can be thought of as a sensible composition of functions: the inner function determines the cut-off value, c_p, such that 100 × (1 - p)% of those in the reference group are correctly classified, and the outer function determines the relative frequency of those in the comparator group correctly classified at said cut-off. This is illustrated in Figure 1.



In addition to setting a directional convention, we must also decide which group serves as the reference group (0) and which is the comparator (1). This will often have a natural synergy with the convention of direction described above (for example, healthy controls will tend to have lower HbA1c as compared to patients with type 2 diabetes). However, in a comparative study of two active treatments (e.g., the effect of proton vs. photon therapy for cancer treatment), context will not always allow the researcher to so easily intuit which group should serve as a reference in advance (). Further, it may not even always be possible to characterize a higher value of Y as “better” or “worse” (e.g., a comparison of hormone levels between males and non-males), such that two independent researchers analyzing the same data could reasonably choose opposite conventions (). To maintain the typical convention of an ROC curve that, loosely speaking—and with further elaboration in Section 2.3—lies mostly on the upper-left side of the unit square [0, 1] × [0, 1], a change in convention of directionality necessitates a simultaneous reversal of groups. This translates to an important symmetry in the resulting ROC curve, described as follows: a change in directional convention corresponds to the mapping (x, y) ↦ (1 - x, 1 - y), and a reversal of groups corresponds to the mapping (x, y) ↦ (y, x), so that if (p, (p)) denotes the graph of the original ROC curve, the graph of the updated ROC curve is denoted by (1 - (p), 1 - p). This phenomenon is illustrated in Figure 2.



In line with our ultimate motivating example (Section 5), we will follow the convention that higher values are more desirable (largely without loss of generality, with a notable exception serving as the topic of discussion in Section 3.4).



 §.§ Area under the curve


The overall quality of Y as a classifier can be summarized by the area under the ROC curve (AUC), given by:

    AUC = ∫_0^1 (p) dp = ∫_0^1 F_1(F_0^-1(p)) dp = (Y_0 > Y_1).

The value of the AUC is bounded below by 0 and above by 1; a change of directional convention alone or a reversal of groups alone would each update the AUC via the relationship AUC^' = 1 - AUC, which is to say that the AUC is preserved by the convention change illustrated in Figure 2; typically, a convention is chosen such that the AUC is no smaller than 0.5. An ROC curve for two well-separated distributions will pass through the point (0, 1), with an AUC of 1.0 (). An ROC curve for two identical distributions will lie along the line y=x, with an AUC of 0.5. Importantly, equality of F_0(t) and F_1(t) for all values of t implies an AUC of 0.5, but not the other way around (i.e., it is possible to achieve an AUC of 0.5 when comparing two distributions that are different; an example will be provided in Section 2.5). This distinction is important, as there are two fundamental tests that may be of interest when evaluating the predictive capacity of Y: a test of a weak null hypothesis, H_0 : AUC = 0.5, and a test of a strong null hypothesis, H_0 : (p) = p for all 0 ≤ p ≤ 1 (i.e., the ROC curve lies along the line y=x). Of note, and as previously pointed out (), a test of the weak null hypothesis is sometimes mischaracterized as a test of a difference in medians: it can be shown that two groups can have the same median value of Y when the weak null is false, and that two groups can have different median values of Y when the weak null is true. Although not our emphasis, we offer methodological guidance on both characterizing and testing the weak and the strong null hypotheses.



 §.§ ROC curves emerging from exponential data


Suppose that Y_0 ∼Exponential(λ_0), with CDF given by F_0(t) = 1 - exp(-λ_0 t) and quantile function given by F_0^-1(p) = -λ_0^-1log(1 - p); analogously, assume that Y_1 ∼Exponential(λ_1). It is readily shown in this setting that the family of ROC curves indexed by this parameterization is given by (p) = 1 - (1 - p)^α, where α = λ_1/λ_0 ≥ 1. We choose to refer to this family of ROC curves as the biexponential ROC family in order to follow the previously established terminology used for normal data (Section 2.5). This family is illustrated in Figure 3(A) for different choices of α. It is straightforward to show that the AUC in this example can be expressed in closed form as AUC=1-(α+1)^-1, α≥ 1, and that both the weak and strong null hypotheses are represented by H_0 : α = 1. A convention change as described in Section 2.2 and illustrated in Figure 2 updates the form of the ROC curve as (p) = p^α^', 0 < α^' < 1, which, quite notably, indexes a different family of ROC curves than the family indexed by the original convention. Together, these facts signify a certain restrictiveness in the form of the single-parameter biexponential ROC family.



 §.§ ROC curves emerging from normal data


Suppose now that Y_0 ∼𝒩(μ_0, σ_0^2), with CDF given by F_0(t) = Φ((t - μ_0)/σ_0) and quantile function given by F_0^-1(p) = μ_0 + σ_0Φ^-1(p); analogously, assume that Y_1 ∼𝒩(μ_1, σ_1^2). It is straightforward to show that the family of ROC curves indexed by this parameterization is given by (p) = Φ(δσ_1^-1 + σ_0σ_1^-1Φ^-1(p)), where δ = μ_0 - μ_1. This family is often referred to as the binormal ROC family (), and is illustrated in Figures 3(B) and 3(C) for different choices of δ and σ_0/σ_1. For reasons that will become more apparent when we discuss estimation, the binormal family is often expressed as (p) = Φ(β_0 + β_1Φ^-1(p)), with β_0 = δσ_1^-1 and β_1 = σ_0σ_1^-1. The binormal ROC curve offers certain flexibility not offered by the biexponential family. First, note that although no closed-form expression exists for the AUC of a binormal ROC curve, the weak null is represented by H_0 : β_0 = 0 (that is, μ_0 = μ_1), and the strong null is represented by H_0 : β_0 = 0  and β_1 = 1 (that is, μ_0 = μ_1 and σ_0 = σ_1). Secondly, note that a convention change corresponds to a re-parameterization that remains in the same binormal family under the original convention, with β_0 ↦β_0/β_1 and β_1 ↦ -1/β_1. The latter fact, in particular, speaks to the broader range of shapes that can be taken by the binormal family under a single convention.





 §.§ More on the biexponential and binormal curves


We have now derived two forms for the ROC curve: one that emerges from exponentially distributed data (Section 2.4) and one that emerges from normally distributed data (Section 2.5). Importantly, ROC curves taking either of these forms can emerge from other combinations of distributions. That is to say, for example, that exponentially distributed data imply a biexponential form for the ROC curve, but not the reverse. Consider the following examples. Suppose first that Y_0 follows a normal distribution with quantile function F_0^-1(p) = μ_0 + σ_0Φ^-1(p). It is straightforward to verify that a biexponential ROC curve is achieved by assigning the following CDF to Y_1: F_1(t) = 1 - (1 - Φ((t - μ_0)/σ_0))^α. As another example, suppose that Y_0 follows an exponential distribution with quantile function F_0^-1(p) = -λ_0^-1log(1 - p). It is again straightforward to verify that a binormal ROC curve is achieved by assigning the following CDF to Y_1: F_1(t) = Φ(β_0 + β_1Φ^-1(1 - exp(-λ_0 t))). The CDFs that induce a desired structure on the form of the ROC curve do not go by known names, and are unlikely to offer practical relevance except to illustrate the lack of a one-to-one relationship between parametric assumptions on the data and the form of the ROC family. Nevertheless, the idea that multiple parametric structures on the data can produce the same family of ROC curves is crucial to understand why direct modeling of the ROC curve without parametric assumptions on the underlying data can be conceptualized as a semiparametric method. These examples offer a heuristic but nevertheless constructive argument that approaches to directly model the ROC curve do not implicitly impose a single parametric structure on the data (which, if they did, would render the direct modeling approaches trivial).




§ ESTIMATION OF THE ROC CURVE


In this section, we outline various methods for estimation of the ROC curve. For ease of exposition, we assume for the majority of our examples that we are interested in obtaining point and interval estimates for the sensitivity, (p), at given values of the FPR, p. In Section 3.4, we briefly discuss alternative ways of characterizing uncertainty.



 §.§ Nonparametric estimation


Let _0(t) = n_0^-1∑_i = 1^n_0 I(Y_0i≤ t) denote the empirical CDF for the reference group, and _0^-1(p) = inf{ t : _0(t) > p }; define _1(t) and _1^-1(p) analogously. The empirical ROC curve can be estimated nonparametrically as (p) = _1(_0^-1(p)). In a finite sample, the empirical ROC curve takes the shape of a staircase. Given a set of data, the empirical ROC curve can be generated using the R code presented in Code Block 1.

[language=R]
## Code Block 1: Nonparametric Estimation of the ROC curve

## values : string of values for the measure of interest
## group  : string of indicators for group membership
compute_roc <- function(values, group, plot = "TRUE") 
    sorted_values <- values[order(values)]
    sorted_group <- group[order(values)]
    cumsum_pos <- cumsum(sorted_group == 1)
    cumsum_neg <- cumsum(sorted_group == 0)
    n_pos <- sum(group == 1)
    n_neg <- sum(group == 0)
    TPR <- cumsum_pos/n_pos
    FPR <- cumsum_neg/n_neg
    TPR <- c(0, TPR, 1)
    FPR <- c(0, FPR, 1)
    if (plot == "TRUE") 
    plot(FPR, TPR, type = "l", xlim = c(0,1), ylim = c(0,1),
         frame.plot = FALSE, lwd = 2, ylab = "Sensitivity",
         xlab = expression(paste("1 ", "2̆013", " Specificity")),
         main = "Empirical ROC Curve")
    
    return(list(FPR = FPR, TPR = TPR))

## output : ordered pairs to be plotted on ROC curve


Constructing point-wise confidence bands is deceptively simple. We are oriented to think in terms of conditioning on the values marked on the x-axis and characterizing uncertainty in corresponding values on the y-axis (e.g., forming point-wise confidence bands in linear regression). However, the values marked on both the x- and y-axes are the result of estimation at fixed values of a cut-off point. The temptation may be, since the point-wise confidence bands involve vertical slices, to form a confidence interval for a one-sample proportion (e.g., a Wilson interval) for (Y_1 ≤ t)—where t is the estimated quantile of of the reference group associated with specificity p. However, this effectively treats the reference distribution as if fixed and known in advance, and fails to account for duality in sampling variation. Instead, a point-wise confidence band can be formed by way of the nonparametric bootstrap. If B denotes the number of bootstrap replicates (sufficiently large), the procedure is as follows in the case where n_0 and n_1 are fixed features of the study design:

for b in 1 to B:

    
  * Obtain a full-size resample (with replacement) of data from the reference group, Y_01^b, …, Y_0n_0^b. Let _0b^-1(p) denote the empirical quantile function of the bootstrapped observations.
    
  * Obtain a full-size resample (with replacement) of data from the comparator group Y_11^b, …, Y_1n_1^b. Let _1b(t) denote the empirical CDF of the bootstrapped observations.
    
  * Let _b(p) = _1b(_0b^-1(p)) denote the value of the empirical ROC curve at the value of p.

end


A 95% confidence band for (p) can be based, for instance, on the 2.5^th and 97.5^th percentiles of {_b(p) }_b = 1^B. When n_0 and n_1 are not fixed characteristics of the study design, a full (i.e., unconditional) bootstrap can be conducted in place of the above procedure ().

The estimated AUC associated with the empirical ROC curve can be computed using its connection to the Mann-Whitney U-statistic (). In particular, if R denotes the sum of the ranks (i.e., of the pooled data of size n, and accounting for ties via averaging ranks) in either group. The estimated empirical AUC is given as follows:

    AUC = 1 + |2(1 + (n_0 + 1)/(2n_1) - R/(n_0n_1)) - 1|/2.

This formula is expressed to automatically follow the convention 0.5 ≤AUC≤ 1. An R function is provided in Code Block 2 to compute the empirically estimated AUC.
[language=R]
## Code Block 2: Calculation of the Empirical AUC

## values : string of values for the measure of interest
## group  : string of indicators for group membership
compute_auc <- function(values, group) 
    n0 <- length(y0)
    n1 <- length(y1)
    R <- sum(rank(values, ties.method = "average")[group == 0])
    auc <- (1 + abs(2*(1 + (n0 + 1)/(2*n1) - R/(n0*n1)) - 1))/2
    return(auc)

## output : numeric value of AUC

A test of the weak null hypothesis, H_0 : AUC = 0.5, can be conducted using the Mann-Whitney test (e.g.,  in R). The strong null hypothesis, H_0 : (p) = p, can be tested nonparametrically using, for example, the Kolmogorov-Smirnov test (e.g.,  in R; ).



 §.§ Parametric estimation


Suppose now that we are willing to posit a parametric form for the data, so that Y_0 has quantile function F_0^-1(p; _0) and Y_1 has CDF F_1(t; _1). If _0 and _1 denote, say, maximum likelihood estimates (MLEs) of _0 and _1 (respectively), then the ROC curve can be consistently estimated as (p) = F_1(F_0^-1(p; _0); _1) by the invariance property of MLEs.



  §.§.§ Exponential data


Consider the parametric form of the biexponential ROC model described in Section 2.4, whereby λ_0 = 1/Y_0 and λ_1 = 1/Y_1 denote the MLEs of λ_0 and λ_1. To form a point-wise confidence interval for (p), we note the utility of the following variance-stabilizing transformation:

    √(n_0)(log(λ_0) - log(λ_0)) d⟶𝒩(0, 1),

so that, by independence of the observations, Var(log(λ_1) - log(λ_0)) ≈ n_0^-1 + n_1^-1 is an asymptotically valid approximation that lends itself to the following form for a point-wise confidence interval for (p) under the parametric biexponential model:

    1 - (1 - p)^exp((log(λ_1) - log(λ_0)) ± z_1 - α/2√(n_0^-1 + n_1^-1)).

Notably, the point estimates and confidence intervals do not depend upon the specific values of λ_1 and λ_0 except through the value of α = λ_1/λ_0, which is intuitive as the ROC curve is invariant to re-scaling the data. The biexponential ROC curve can reasonably be thought of as a one-parameter model. The AUC can be estimated as AUC = 1 - (α + 1)^-1. Under the exponential model, a test of both the weak and the strong null can be implemented using the likelihood ratio statistic, Λ = nlog(λ) - n_0log(λ_0) - n_1log(λ_1), where λ denotes the MLE of the rate parameter in the combined data. The likelihood ratio test statistic has an asymptotic χ_1^2 distribution under the strong null.



  §.§.§ Normal data


Now, consider the parametric form of the binormal model described in Section 2.5, whereby μ_0 = Y_0, μ_1 = Y_1, σ_0^2 = n_0^-1∑_i = 1^n_0 (Y_0i - μ_0)^2, and σ_1^2 = n_1^-1∑_i = 1^n_1 (Y_1i - μ_1)^2; in practice, we may use the sample variances (e.g., S_0^2 = n_0σ_0^2(n_0 - 1)^-1) in place of the MLEs. Following a similar line of logic as with the exponential model reveals the following form for a point-wise confidence interval for (p) under the binormal model:

    Φ( (μ_0 - μ_1) + S_0Φ^-1(p) ± t_1 - α/2, df√(S_0^2n_0^-1 + S_1^2n_1^-1)/S_1),

where df denotes the Welch-Satterthwaite approximation to degrees of freedom (). Similarly to the above example, the point estimates and confidence intervals do not depend upon the specific values of the estimates except through the values of δ = μ_0 - μ_1 and S_0/S_1; the binormal ROC curve can be conceptualized as a two-parameter model. The AUC can be estimated numerically given the MLEs using, for example, the  function in R. A test of the weak null can be based on an unequal-variance t-test of means. On the other hand, a test of the strong null can be based on the likelihood ratio test, with Λ = nlog(σ^2) - n_0log(σ_0^2) - n_1log(σ_1^2), where σ^2 denotes the MLE of the variance in the combined data. The likelihood ratio test statistic has an asymptotic χ_2^2 distribution under the strong null (note, however, that the MLEs for σ_0^2 and σ_1^2 may not be replaced by the corresponding sample variances in the test statistic for this to hold).



 §.§ Semiparametric estimation


Let U_ij = I(Y_1j≤ Y_0j) denote the indicator of the j^th observation from the comparator group having a value no larger than the i^th observation from the reference group (there are a total of n_0 × n_1 observations of the form U_ij). It has been previously shown () that [U_ij|F_0(Y_0i) = p] = (p). This lends itself to the use of generalized linear models (suitable for binomial data) to estimate an ROC curve directly, bypassing the need to impose an explicit parametric structure.



  §.§.§ The semiparametric biexponential model


Consider modeling the ROC curve using the biexponential form described in Section 2.4 (which is implied by, but does not itself imply, exponentially distributed data). Letting V_ij = 1 - U_ij, the corresponding semiparametric biexponential ROC model is given by log([V_ij|F_0(Y_0i) = p]) = αlog(1 - p). Note the absence of an intercept in this single-parameter family. In practice, the value of p is not known but for each V_ij is estimated as p_ij = _0(Y_0i). The associated estimating equations for α take the following form:

    𝔾(α) = ∑_i = 1^n_0∑_j = 1^n_1log(1 - p_ij)/1 - (1 - p_ij)^α(V_ij - (1 - p_ij)^α) = 0.

Observations with p_ij = 1 (so that log(1 - p_ij) = -∞) may be excluded for the estimating equation to be well defined; this will occur for the pairs that involve the maximum order statistic of the Y_0i's, so that the estimating equation relies on (n_0 - 1) × n_1 of the observations U_ij. An R function is provided in Code Block 3 to estimate and implement semiparametric biexponential estimation of the ROC curve.

[language=R]
## Code Block 3: Biexponential Semiparametric Estimation of the ROC curve

## values : string of values for the measure of interest
## group  : string of indicators for group membership
ROC.GLM.E <- function(group, values) 
     y0 <- values[group == 0]
     y1 <- values[group == 1]
     n0 <- length(y0)
     n1 <- length(y1)
     Pij <- matrix(colMeans(outer(y0, y0, "<=")), nrow = n0, ncol = n1)
     P <- matrix(Pij, ncol = 1)
     Q <- cbind(log(1 - P)[P != 1])
     V <- 1 - as.numeric(t(outer(y1, y0, "<=")))[P != 1]
     zz <- glm.fit(Q, V, family = binomial(link = "log"))
     zzcoefficients

## output : estimated coefficient from semiparametric biexponential ROC


This estimating equation possesses no closed-form solution, but can be solved numerically using a Gauss-Newton algorithm (). Due to the correlation of theU_ij's, however, a bootstrap approach is typically employed in order to obtain asymptotically valid confidence intervals for(p). As in Section 3.2.1, the AUC can be estimated asAUC = 1 - (α + 1)^-1. Both the weak and strong null can be tested using the bootstrap samples under an asymptotic normality assumption onα.



  §.§.§ The semiparametric binormal model


Now, consider modeling the ROC curve using the binormal form (which is implied by, but does not itself imply, normally distributed data). The corresponding semiparametric binormal ROC model is given byΦ^-1([U_ij|F_0(Y_0i) = p]) = β_0 + β_1Φ^-1(p). The associated estimating equations for= (β_0, β_1)^⊤take the following form:

    𝔾() = ∑_i = 1^n_0∑_j = 1^n_1_ijϕ(z_ij)/Φ(z_ij)(1 - Φ(z_ij))(U_ij - Φ(z_ij)) = 0,

wherez_ij = β_0 + β_1Φ^-1(p_ij), and_ij = (1, Φ^-1(p_ij))^⊤.
For reasons analogous to those described in Section 3.3.1, observations withp_ij = 1may be excluded for the estimating equations to be well defined. These estimating equations, too, possess no closed-form solutions, and a nonparametric bootstrap is typically employed to conduct inference. As in Section 3.2.2, the AUC can be estimated numerically given the MLEs using, for example, the  function in R. The weak and/or strong null can be tested by conducting the appropriate tests on, also previously described. An R function is provided in Code Block 4 to estimate implement semiparametric binormal estimation of the ROC curve.

[language=R]
## Code Block 4: Binormal Semiparametric Estimation of the ROC curve

## values : string of values for the measure of interest
## group  : string of indicators for group membership
ROC.GLM.N <- function(group, values) 
     y0 <- values[group == 0]
     y1 <- values[group == 1]
     n0 <- length(y0)
     n1 <- length(y1)
     Pij <- matrix(colMeans(outer(y0, y0, "<=")), nrow = n0, ncol = n1)
     P <- matrix(Pij, ncol = 1)
     Q <- cbind(1, qnorm(P)[P != 1])
     U <- as.numeric(t(outer(y1, y0, "<=")))[P != 1]
     zz <- glm.fit(Q, U, family = binomial(link = "probit"))
     zzcoefficients

## output : estimated coefficient from semiparametric binormal ROC


 §.§ Other considerations for characterizing uncertainty


We acknowledged at the beginning of Section 3 that our outlined estimation procedures assume that it is of interest to characterize uncertainty in estimating the TPR at specific values of the FPR. It could be of equal interest to characterize uncertainty of the FPR at specific values of the TPR. We do not consider this nuance in extraordinary depth because estimation of this form can be accomplished by a convention change as described in Section 2.2 and illustrated in Figure 2, so that the fundamental mechanics of the procedure are unchanged. We acknowledge that the setting in which there is no clear ordering of groups, nor a natural way to define higher values of the measure of interest as “better” or “worse” poses a unique challenge. Others have considered intuitively combining information from bi-directional confidence regions, including radial sweeps, though there is little theoretical or empirical justification for these methods ().



§ SIMULATION-BASED COMPARISON OF ESTIMATION METHODS


The goal of this section is to illustrate the bias-variance tradeoff associated with various approaches in finite samples. We utilize two general data generating mechanisms (DGMs): the first is an exponential DGM of the formY_0 ∼Exponential(λ_0 = 1)andY_1 ∼Exponential(λ_1 = 4), and the second is a normal DGMY_0 ∼𝒩(μ_0 = 5.5, σ_0^2 = 1)andY_1 ∼𝒩(μ_1 = 4.0, σ_1^2 = 1). We further consider two pairs of sample sizes:n_0 = n_1 = 30(low) andn_0 = n_1 = 60(high). In Section 4.1, we seek to confirm the inadequacy of the Wilson interval as compared to the nonparametric bootstrap for the empirical ROC curve. In Section 4.2, we generate data from the exponential DGM and compare the finite sample properties of the various methods described in Section 3. In Section 4.3, we do the same under the normal DGM.

To evaluate the finite sample properties of these methods in various scenarios, we compare the average estimated value of the ROC curve to the true value, and determine the average width and coverage associated with 95% confidence intervals. For each simulation scenario presented, we utilizeM = 1000simulation replicates and, where applicable,B=3000bootstrap replicates. All simulations were conducted in R ().




 §.§ The Wilson interval vs. the bootstrap for the empirical ROC


In this simulation study, we generate data from the normal DGM in order to compare the Wilson interval to the quantile-based nonparametric bootstrap-based interval associated with empirical ROC curve. We consider both the low and high sample size cases. The results of this simulation are shown in Table <ref>, which depicts the results at the specified values ofpsuch that(p) = 0.1, 0.3, 0.5, 0.7, and0.9; the justification for considering values ofpin this range that it encompasses a wide range of ordered pairs on the ROC curve that approach the boundary of the curve.

We observe a marked degree of finite-sample bias associated with the empirical ROC curve for very small values ofp. The Wilson intervals uniformly have lower width as compared to those associated with the empirical bootstrap, which is not surprising given that it treats the distribution of the reference group as if fixed. With the exception of the lower values ofp, the empirical bootstrap appears to possess adequate coverage. On the other hand, the Wilson interval generally does not possess proper coverage across the board. The intuition for the general pattern of under-coverage is that the Wilson interval fails to account for variation in estimatingp, the specificity associated with a particular cut-off point. We no longer consider the Wilson interval as a competing method for the remainder of this manuscript.



 §.§ Comparisons under the exponential DGM


In this section, we generated data under the exponential DGM, and considered four models: (1) a parametric binormal model, (2) a parametric exponential model, (3) a semiparametric biexponential model, and (4) a semiparametric binormal model. We consider both the low and high sample size cases. The results of this simulation are depicted in Figure 4. As expected, the parametric binormal ROC curve demonstrates a bias and a lack of coverage that are not overcome in a larger sample. The finite sample behavior is more favorable for the competing methods. The validity of the parametric and semiparametric biexponential models is not surprising (Figures 4(A) and 4(D)), nor is the apparent greater degree of efficiency associated with the parametric method (Figures 4(B) and 4(E)). It is noteworthy that the semiparametric binormal model exhibits so little bias given that it is not correctly specified, though we have pointed out that it offers greater flexibility as compared to the biexponential family. Given the additional model complexity of the binormal model, the finding that the confidence intervals are generally wider as compared to those of the respective biexponential models is not unexpected (Figures 4(B) and 4(E)).



 §.§ Comparisons under the normal DGM


We conduct a simulation study that mirrors the setup described in Section 4.2, but instead generating data under the normal DGM. The results of this simulation are depicted in Figure 5. The parametric biexponential ROC curve performs poorly with respect to bias and coverage. This can be heuristically explained by the fact that the parametric biexponential model uses information from the mean ratio, despite the fact that the data are parameterized by the (scaled) mean difference and the ratio of standard deviations. Since the mean ratio ofμ_1/μ_0 = 5.5/4.0is close to one, the parametric biexponential model does not “perceive,” so to speak, a meaningful difference in stochastic ordering under the (albeit incorrect) exponential assumption. The finite sample behavior is more favorable for the competing methods, although the parametric binormal model suffers a notable loss of coverage near the boundaries. The semiparametric biexponential model is not free of bias, although its relatively good performance is likely related to the fact that the variances were generated to be equal in this data generating mechanism (that is to say that the biexponential form is strictly concave, whereas unequal variances in a normal model introduces infection points into the ROC curve). This study is consistent with the relative efficiency associated with the correct parametric specification of the data (Figures 5(B) and 5(C)).





§ APPLICATION TO SOLID ORGAN TRANSPLANTS


In this section, we apply several estimation methods to data from a study that sought to investigate SARS-CoV-2 vaccination-induced immunogenicity.



 §.§ Study cohort and measures


Eligible individuals enrolled in this study included SOT recipients (n_1 = 54) and healthy controls (HCs;n_0 = 26) over 55 years old who were scheduled to receive a two-dose regimen of BNT162b2 as part of routine care at Vanderbilt University Medical Center (Nashville, TN). Specific transplantation groups included prior heart (11%), kidney (41%), liver (37%), lung (7%), and kidney/liver (4%) transplant recipients. Immunoglobulin G to SARS-CoV-2 spike extracellular domain (ECD) was evaluated by enzyme-linked immunosorbent assay. Three SOT recients were excluded from our analysis due to serological evidence of an ongoing or prior SARS-CoV-2 infection. Further details pertaining to recruitment, enrollment, and prior analysis are detailed by <cit.>.



 §.§ Empirical analysis


Our illustrative analysis specifically focuses on ECD response to the second vaccination dose (henceforth denoted). Higher values ofserve as a correlate of higher immunity (), and we anticipate a priori that SOT recipients, due to immunocompromised status, tend to have lowervalues as compared to HCs. Figure 6(A) presents histograms of, stratified by group, and Figure 6(B) presents an empirical ROC curve with a 95% confidence band as estimated by the nonparametric bootstrap procedure described in Section 3.1. The degree of separation between the distribution of groups is apparent from examining the histograms. The empirical ROC curve, particularly when considering the associated confidence bands, offers additional context regarding our certainty regarding how well-separated the groups are in theirresponse.



 §.§ Parametric estimation


The estimated ROC curve under the parametric biexponential model is depicted in Figure 6(C), and the estimated curve under the parametric binormal model is depicted in Figure 6(D). For the parametric biexponential model, we estimateα = 2.72. For the binormal model, we estimateβ_0 = 1.70andβ_1 = 0.52. Of note, the distribution ofamong SOT recipients exhibits right-skewness, whereas the distribution among HCs more closely resembles the symmetry of a normal distribution. That is to say that results from both parametric methods are likely biased, which is reflected in part by the discrepancy between the point estimates for the estimated ROC curves and the empirical curve; this discrepancy is particularly notable for the parametric biexponential model.



 §.§ Semiparametric estimation


We now estimate the ROC curve using the correspoinding semiparametric methods. The results of the semiparametric biexponential model are shown in Figure 6(E) and those of the semiparametric binormal model are shown in Figure 6(F). For the biexponential model, we estimateα = 4.48, a marked difference from the point estimate obtained using parametric methods. Recall thatαdoes not possess the interpretation of a mean ratio when the biexponential curve is estimated in a semiparametric fashion. We see from comparing Figure 6(E) to 6(C) that the semiparametric biexponential model—while still notably different from the empirical curve—is better calibrated to the empirical curve across the broader range ofp.

For the semiparametric binormal model, we estimateβ_0 = 1.35andβ_1 = 0.49. A particularly noteworthy result from the semiparametric binormal fit is that the empirical ROC curve is almost entirely captured within its confidence band; this is not true for any of the other three fits. The curve associated with the parametric model possesses a similar shape, but likely overstates certainty. This example is again consistent with the flexibility associated with the semiparametric binormal curve, especially as compared to the more parsimonious biexponential model.

One key comparison is also between the semiparametric biexponential model (Figure 6(E)) and the parametric binormal model (Figure 6(D)). The confidence bands are considerably wider under the former model, despite the fact that it has one fewer parameter. This might be a more surprising result if had the two models been nested, but this particular comparison is a stark example of the fact relative efficiency is driven by more than simply the relative number of parameters possessed by the model. One possible source of this discrepancy is a “ceiling effect” in the variability given the boundedness of the ROC function.



§ DISCUSSION


In this manuscript, we have described the ROC curve and highlighted aspects of its geometry that we believe may be underappreciated. We have outlined and offered software code for a number of competing nonparametric, parametric, and semiparametric estimation approaches. Simulation studies revealed, as expected, a notable trade-off between efficiency and bias associated with structure.

We acknowledge that the ROC curve may, for many, serve as a visual tool to complement a primary analysis and that explicitly conducting inference on the value of an ROC curve at a specified value ofpmay not always reflect the main scientific goal. However, the width of confidence bands offer visual clarity to uncertainty and speak in some respects to how trustworthiness of the findings regarding diagnostic utility or stochastic ordering. Our simulations and application revealed non-trivial differences in both bias and efficiency across approaches. To that end, knowledge regarding the bias and efficiency of the approach used to generate an ROC curve is relevant even if confidence intervals for the curve are not specifically generated in a given researcher's implementation.

We note that loss of coverage is a commonly encountered feature near the boundary of the ROC curve. This may not always be of major concern in the sense that the extreme ends of the curve signify either the setting of zero-specificity or zero-sensitivity, reflecting cut-off points that may not be considered viable candidates. However, further investigation may bring clarity to the downstream consequences on coverage associated with confidence intervals for other summary measures (say, the AUC).

Our finding that the binormal estimation techniques—parametric and semiparametric—appear to offer a certain degree of robustness, is consistent with previous findings (). Heuristically, the space of monotone increasing functions from[0, 1]to[0, 1]likely does not need to be indexed by more than two parameters in order to cover a wide range of real-world scenarios. Our work contributes to the literature by underscoring the inadequacy of the single-parameter biexponential model. The semiparametric variant of the biexponential model has previously been compared to other methods (), though in the context of estimatingαrather than(p)at various values ofp. Note that the findings of this work do not necessarily support the uptake of the semiparametric biexponential model in the real world.

In our application of ROC methods to vaccine-associated immunogenicity in Section 5, there is a notable degree of heterogeneity in the comparator group that is not captured by our illustrative analysis. For example, liver transplant recipients undergo the lowest degree of immunosuppression as compared to other organ transplant types (). Over the years, there has been research on regression of the ROC curve, whereby one seeks to evaluate how the nature of an ROC curve varies across population strata (). The techniques used to accomplish this are a natural extension of those presented in Section 3.3. This area has also been explored independent of the setting of diagnostic markers ().

We acknowledge that the suite of methods compared in this manuscript are not exhaustive. For example, <cit.>, <cit.>, and <cit.> consider alternative semiparametric methods for ROC estimation based on accelerated regression models and/or kernel density estimation. The goal of this work was not to serve as an exhaustive literature review of all existing ROC estimation techniques, but instead to provide insights into ROC analysis based on a selection of estimation approaches that are straightforward to implement in the real world. We hypothesize that the patterns of behavior for other semiparametric approaches would in many ways mirror those of the approaches we considered, although further study would be needed to confirm this.

An additional trade-off that we did not consider in this work is that of computational efficiency. The bootstrap is a computationally taxing procedure, particularly when one needs to employ enough replicates to justify quantile-based confidence intervals. Alonzo and Pepe consider coarsening the grid of FPR values at which to evaluate the TPR; more variability is introduced when failing to leverage the full distribution of the reference group (). In the modern era of improving computing resources, this trade-off may seem to wane in relevance; however, we are also in a modern era of access to increasing larger data sets, so considerations regarding the trade-off of computational burden should not be totally ignored.

We must offer our full acknowledgement that ROC methods are not without limitations. Some of the most meaningful criticisms of the methodology include that the ROC curve gives consideration (both visually and numerically via the AUC) to the non-viable region[0.5, 1] × [0, 0.5]in which both the specificity and sensitivity are seriously lacking, and that the ROC curve offers no meaningful insights into the positive predictive value and negative predictive value (). These challenges have not gone completely unaddressed in the literature. For example, weighted AUC methods have been proposed to more heavily consider viable regions ().

There are a number of important future directions for this work. Considerations regarding study design, matching, and repeated measures remain under-explored in this space. However, if the trends of interest in measures of stochastic ordering continue, we will soon identify unmet needs for continued expansions and extensions of ROC methodology to accommodate the nature of real-world data collection.

The Vanderbilt Institute for Clinical and Translational Research (VICTR) is funded by the National Center for Advancing Translational Sciences (NCATS) Clinical Translational Science Award (CTSA) Program, Award Number 5UL1TR002243-03. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

 Paper Title: Empirical Bayes Linked Matrix Decomposition.



Empirical Bayes Linked Matrix Decomposition]Empirical Bayes Linked Matrix Decomposition










[1]Eric F. Lockelock@umn.edu

*[1]Division of Biostatistics and Health Data Science, School of Public Health, University of Minnesota, Minneapolis, 55455, MN, USA







Data for several applications in diverse fields can be represented as multiple matrices that are linked across rows or columns.  This is particularly common in molecular biomedical research, in which multiple molecular “omics" technologies may capture different feature sets (e.g., corresponding to rows in a matrix) and/or different sample populations (corresponding to columns).  This has motivated a large body of work on integrative matrix factorization approaches that identify and decompose low-dimensional signal that is shared across multiple matrices or specific to a given matrix.  We propose an empirical variational Bayesian approach to this problem that has several advantages over existing techniques, including the flexibility to accommodate shared signal over any number of row or column sets (i.e., bidimensional integration), an intuitive model-based objective function that yields appropriate shrinkage for the inferred signals, and a relatively efficient estimation algorithm with no tuning parameters.  A general result establishes conditions for the uniqueness of the underlying decomposition for a broad family of methods that includes the proposed approach.  For scenarios with missing data, we describe an associated iterative imputation approach that is novel for the single-matrix context and a powerful approach for “blockwise" imputation (in which an entire row or column is missing) in various linked matrix contexts.  Extensive simulations show that the method performs very well under different scenarios with respect to recovering underlying low-rank signal, accurately decomposing shared and specific signals, and accurately imputing missing data. The approach is applied to gene expression and miRNA data from breast cancer tissue and normal breast tissue, for which it gives an informative decomposition of variation and outperforms alternative strategies for missing data imputation. 



















[
    *
    August 5, 2024
==================





§ INTRODUCTION



Low-rank matrix factorization techniques are widely used for many machine learning tasks such as data compression and dimension reduction, denoising to approximate underlying signal, and matrix completion.  Often, instead of a single matrix, the data for a given application takes the form of multiple matrices that are linked by rows and/or columns.  For example, in Section <ref> we consider data from The Cancer Genome Atlas, in which  gene expression (mRNA) and microRNA (miRNA) data are available from different platforms for breast cancer tumor samples and normal breast tissue samples from unrelated individuals.  Thus, the data can be represented as four matrices: (1) mRNA for cancer tissues, (2) mRNA for normal tissues, (3) miRNA for cancer tissues, and (4) miRNA for normal tissues; these matrices share both rows (here, molecular features) columns (here, samples), giving a bidimensionally linked structure. Furthermore, these data have column-wise missingness, in which molecular data for one of the platforms (mRNA or miRNA) is entirely missing for some samples in each of the cancer and control cohorts.  We are interested in imputing the missing data, as well as identifying patterns of variation that are shared or specific to the different data sources and sample cohorts; for these tasks, a low-rank factorization approach that integrates all of the matrices together while accounting for their differences is attractive.  

There is a growing and dynamic literature on multi-matrix low-rank factorization approaches.  The majority of existing methods assume that just one dimension, rows or columns, is shared across matrices, i.e., unidimensional integration. For example, the Joint and Individual Variation Explained (JIVE) approach <cit.> decomposes low-rank structure that is shared across matrices from variation that is specific to each matrix.  Several related approaches yield a similar decomposition using different strategies or assumptions <cit.>, while other approaches such as structural learning and integrative decomposition (SLIDE) <cit.> allow for structure that is partially shared across any subset of matrices.  Such an approach is efficient and useful for interpretation, as for many applications identifying shared, partially shared, and specific underlying structure is of interest.  

The bidimensional linked matrix factorization (BIDIFAC) approach <cit.> was developed to accommodate bidimensionally linked integration scenarios, by identifying structure that is shared or specific across both row sets and column sets.  The objective function for BIDIFAC involves a structured nuclear norm penalty, which approximates the data as a decomposition of low-rank modules that are shared across row or column subsets.  This approach is attractive in the bidimensionally linked context because it has favorable convergence properties and generally leads to a uniquely-defined decomposition <cit.>; methods for unidimensional integration without shrinkage penalties typically require orthogonality <cit.> or other constraints to assure uniqueness, which do not extend to the bidimensionally linked context.  Furthermore, the application of the structured nuclear norm penalty for unidimensional integration (termed  UNIFAC) has been shown to outperform alternative approaches with respect to accurately decomposing shared and specific structures <cit.> and imputing missing data <cit.> under certain scenarios. However, a drawback is that it tends to over-penalize the estimated structure, and is thus biased toward an over-conservative solution with respect to the underlying low-rank structure <cit.>.

For a single matrix the tendency of the nuclear norm to over-shrink low-rank signal has been well-documented, and in contrast low-rank approximations without any further shrinkage tend to overfit <cit.>.  Empirical Bayes approaches to low-rank factorization are an attractive compromise, in which the optimal levels of shrinkage are determined empirically within a model-based framework <cit.>. Under a variational Bayes approach, the true posterior distribution for Bayesian model is approximated by minimizing the free energy (reducing to Kullback-Leibler divergence) under simpler distributional assumptions <cit.>.  In particular, an empirical variational Bayes approach to low-rank matrix factorization under an assumption of normally distributed factor matrices has desirable theoretical properties, and the resulting approximation for a scenario without missing data can be efficiently computed via a closed-form shrinkage formula on the observed singular values of the matrix <cit.>.   

In this article, we describe an empirical variational Bayesian approach to identify and decompose shared, partially shared, or specific low-rank structure in the context of unidimensionally or bidimensionally linked matrices.  Moreover, we describe an accompanying iterative imputation approach analogous to an expectation-maximization (EM) algorithm that can handle entry-wise missing or row/column-wise missing values.  The proposed approach is free of any tuning parameters, and is shown to share desirable properties of the BIDIFAC approach while addressing the issue of over-shrinkage.     




§ NOTATION AND SETTING



Throughout this article bold uppercase characters (`') correspond to matrices and bold lowercase characters (`𝐱') correspond to vectors. Square brackets index the entries within a vector or matrix, e.g., `[m,n]'.  The singular value decomposition (SVD) of a matrix : M × N is given by ___^T, where the diagonal entries of _, _[r,r],  are the singular values of  ordered from largest to smallest for r=1,, min(M,N).  The Frobenius norm is given by ||·||_F, so that ||X||_F^2 is the sum of the squared entries in ; the nuclear norm is given by ||·||_*, which is the sum of the singular values in a matrix: 
    ||||_*=∑_r=1^min(M,N)_[r,r].


  We consider the setting of a collection of matrices  _ij: M_i × N_j for i=1,,I and j=1,,J. The column sets N_1,, N_J and row sets M_1,,M_L are consistent across the matrices.   The subscript `' defines concatenation across row or column sets, and the full data  can be represented as a single M × N  matrix where M = M_1++M_I and N = N_1++N_J:   

    =  [ [ _11 _12     _1J;   ⋮   ⋮   ⋮   ⋮; _I1 _I2     _IJ ] ]    where  _ij  are   M_i × N_j.

As for our motivating application in Section <ref>, the I column sets may correspond to different sample cohorts and the J row sets may correspond to features from different high-dimensional sources.   



§ SINGLE-MATRIX RESULTS


Here we review some established results for a single matrix that are critical to what follows. Propositions <ref> and <ref> describe solutions to the least squares matrix approximation problem under a rank-constrained or nuclear norm penalized criterion, respectively.

  For : M × N, the minimizer of the least squares objective ||X-||_F^2 under the constraint rank()=R ≤min (M,N) is given by =__S _^T where _ is diagonal with _[r,r]=_[r,r] for r=1,,R and _[r,r]=0 for r > R.  




  For : M × N, the minimizer of the nuclear norm penalized objective 1/2||X-||_F^2+λ ||||_* is given by =__S _^T where _S is diagonal with _[r,r]=max(_[r,r]-λ,0) for all r. 


Proposition <ref> is well-known, and a proof of Proposition <ref> can be found in <cit.>.  Because of how they operate on the singular values , Proposition <ref> represents a hard-thresholding and Proposition <ref> a soft-thresholding approach to low-rank approximation.  Proposition <ref> below establishes that the soft-thresholding operator also solves an objective with L_2-penalties on the factor components of , while Corollary <ref> further establishes that it is the posterior mode of a Bayesian model with normal priors on the factor components.  


   For : M × N,

    : M× H,: N× Hmin ||-^T||_F^2+λ (||||_F^2+||||_F^2) = : rank()≤ Hmin ||-||_F^2 +2λ ||||_*

   Moreover, =^T, where  solves the right-hand side of (<ref>) and {,  } solves the left-hand side of (<ref>).


   For : M × N,
if : M × H and : N × H have independent Normal(0, σ^2/λ) entries, and =^T+ where  has independent Normal(0,σ^2) entries, then the mode of the posterior distribution p(,|) satisfies =^T, where  solves the right-hand side of (<ref>).

A proof of Proposition <ref> can be found in <cit.>, and Corollary <ref> follows by noting that the log of the posterior density for the specified model is proportional to the left-hand side of (<ref>).  While the results hold for general H, in what follows we set H as the largest possible rank H= min (M,N); the actual estimated rank will tend to be smaller due to singular value thresholding (as in Proposition <ref>).     The distribution of singular values for the error matrix  has been well-studied, including the following results on the first singular value under general conditions from <cit.>:
    If : M × N has independent entries with mean 0, variance σ^2 and finite fourth moment, then [1,1] ≈σ (√(()M)+√(N)) as M,N →∞; further, if the entries are Gaussian, 𝔼([1,1]) ≤σ (√(()M)+√(N)) for any M,N.
    

These results can motivate choice of the nuclear norm penalty λ.  A conservative approach, if the error variance σ^2 is known, is to select λ = σ (√(M)+√(N)); per Propositions <ref> and <ref>, this will tend to shrink to 0 any singular values in  that result from error with no true low-rank signal.  Another approach is to adopt an empirical Bayes estimate for λ based on the model in Corollary <ref>. However, a general limitation of the nuclear norm approach to low-rank matrix reconstruction is the use of a universal threshold on all singular values; in practice larger singular values should be penalized less, as they capture a relatively lower ratio of error and higher signal  <cit.>.  An analogous interpretation is that the entries of  and  should not have identical variance across columns, and this can be relaxed by a model in which they have column-specific variances: 
[·,r] ∼Normal(,τ^2_,r) and [·,r] ∼Normal(,τ^2_,r) for r=1,…,H.[For most scenarios τ^2_,r and τ^2_,r are not independently identifiable, as the level of signal is determined by their product.  ] 

For given ={τ_,rτ_,r}_r=1^H, the approximation resulting from the posterior mode for p(,|, ) is given by thresholding the singular values of , analogous to Corollary <ref>.  To determine , directly maximizing the joint likelihood p(, , |) over , and  degenerates to → in the global solution, though it can have a reasonable local mode that is a singular value thresholding operator <cit.>.  Further, a marginal empirical Bayes estimator that optimizes p(|) (effectively integrating over  and ) is difficult to obtain computationally or analytically. As an alternative, one can use an empirical variational Bayes (EVB) approach.  In general, for a variational Bayes approach <cit.>, the true posterior of a Bayesian model with parameters Θ and data , p(Θ|) is approximated by another distribution q(Θ) under simplifying assumptions to minimize the free energy

    F(q) = E_q log q(Θ)/p(Θ,) = E_q log q(Θ)/p(Θ|)- log  p(),

where E_q(·) is the expected value with respect to the distribution defined by q(·). 
In the context of low-rank approximation, a useful simplifying assumption for q(·) is that the left and right factor matrices are independent, q(,)=q_u() q_v().  Thus, for hyperparameters  and σ the free energy to minimize is

    F(q |σ, ) = E_q log q_u()q_v()/p(|, , σ) p(|_) p(|_),

and a point estimate for low-rank structure  can be obtained via its expected value under q(·): 

    =E_q (^T).

Under an empirical variational Bayes approach,  and σ are also estimated by minimizing the free energy in (<ref>).  For fully-observed , <cit.> showed that the global solution to the empirical variation Bayes objective can be obtained in closed form as a singular value thresholding operator; their result for fixed σ is given in Theorem <ref>, and for estimation of σ in Theorem <ref>.
 
  The approximation  (<ref>) obtained by minimizing (<ref>) over q and  is ==__S _^T where _S is diagonal with 
 
    _[r,r]=0  if  D_[r,r]<σ√(M+N+√(MN) (κ + 1/κ))
 and 
 
    _[r,r]= (_[r,r]^2-(M+N) σ^2+√((_[r,r]^2-(M+N)σ^2)^2-4MNσ^4))/(2_[r,r])  otherwise.

The value κ is given as the zero-cross point of the function
 
    f(x)=x √(N/M)   log ( x√(M/N)+1 ) +x√(M/N)  log ( x√(N/M)+1 ) -1.

 
 
  
  Assume without loss of generality that M>N and α=N/M. The estimate σ̂ that minimizes (<ref>) is the global minimizer of  
 
    Ψ(σ)=∑_r=1^N ψ_1 (_X[r,r]^2/M σ^2),

 where 
    ψ_2(x) = x-log x+1_{x>c}ψ_2(x),

 
    ψ_2(x) = log  (√(α)ψ_3(x)+1)+αlog  (ψ_3(x)/√(α)+1)-αψ_3(x),

 and 
 
    ψ_3(x)=1/2 √(α)(x-1-α + √((x-(1+α))^2-4 α)).



Together, Theorems <ref> and <ref> provide a powerful and efficient way to approximate low-rank signal without any tuning parameters.  The only substantial computing requirement is in obtaining the SVD of , as other steps involve univariate functions that are trivial to solve.    

         



§ BIDIMENSIONAL LINKED MATRIX FACTORIZATION


BIDIFAC+ <cit.> was developed to decompose bidimensionally linked data as in (<ref>) into modules of low-rank structure that are shared across column sets and/or row sets.      That is,

    _≈∑_k=1^K ,
  
  where each  is a concatenation of blocks ^(k)_ij as in (<ref>) and 
      
    ^(k)_ij = _M_i × N_j   if [i,k] = 0  or [j,k] = 0
    _i^(k)_j^(k) T   if [i,k] = 1  and  [j,k] = 1 .

  Here,  and  are binary matrices that indicate the presence of module k () across the row and column sets; that is,  is low-rank structure specific to the submatrix  defined by the row sets identified in [,k] and the column sets identified in [,k].  This general framework subsumes several other integrative decompositions of structured variation. For example, in the unidimensionally linked case with shared columns J=1, JIVE <cit.> identifies components that are either shared across all matrices ([i,k]=1 for all i) or specific to a single matrix ([i,k]=1 for just one i), while SLIDE <cit.> allows for components that are shared across any number of matrices.  The BIDIFAC  method <cit.> is a special case in which modules are either globally shared ([i,k]=[j,k]=1 for all i,j), shared across all rows for a given column set ([i,k]=1 for all i and [j,k]=1 for one j), shared across all columns for a given row set ([j,k]=1 for all j and [i,k]=1 for one i), or specific to one matrix ([i,k]=1 and [j,k]=1 for just one i,j). 
  
 The BIDIFAC and BIDIFAC+ approaches solve a structured nuclear norm objective: 


    1/2||_-∑_k=1^K  ||^2_F + ∑_k=1^K λ_k ||||_*,

subject to ^(k)_ij = _M_i × N_j if  [i,k] = 0 or [j,k] = 0.  For fixed  and , objective (<ref>) may be solved with an iterative soft-singular value thresholding algorithm that cycles through the estimation of each   with applications of Proposition <ref>. For smaller I and J,  and  can enumerate all possible submatrices of the row and column sets; the solution may still be sparse, with = if the submatrix corresponding to module k has no particular shared structure.  Alternatively, if the number of row and column sets I and J are large, <cit.> describe an algorithm to adaptively estimate  and  to give the submatrices with the most significant low-rank structure during estimation.

The choice of the module-specific penalty parameters λ_k is critical.  By default, λ_k are selected by an approach motivated by Proposition <ref>; under the assumption that each matrix _ij has error variance 1, λ_k depends on the total number of non-zero rows and columns in its submatrix,

    λ_k = √(∑_i=1^I [i,k] M_i)+ √(∑_j=1^J [j,k] N_j),

for k=1,,K.  This approach will shrink to 0 any singular values in the submatrix that are due to error only.  In practice each data matrix _ij is scaled to have error variance 1 beforehand; if the error variance is unknown, then it can be taken as the total variance in _ij (a conservative choice) or via other approaches such as the median absolute deviation estimator in <cit.>.





§ EMPIRICAL VARIATIONAL BIDIFAC (EV-BIDIFAC)



While the BIDIFAC+ approach has several advantages, one weakness is the required standardization by the noise variance in each data matrix, which is estimated pre-hoc and not within a cohesive framework.  More critically, the use of nuclear norm penalties generally results in over-shrinkage of the signal, and the default theoretically-motivated choice of the λ_k's is particularly conservative.  Thus, we aim to develop a more flexible empirical variational Bayes approach to the decomposition of bidimensionally linked matrices, analogous to that for  a single matrix in Section <ref>.  Our psuedo-Bayesian model is as follows for i=1, I and j=1,,J: 


    _ij = ∑_k=1^K _i^(k)_j^(k) T + _i,j where _i,j[l,m] iid∼Normal(0, σ^2_ij), 
    _i^(k)[·,r]= if [i,k]=0 and  _i^(k)[·,r]∼Normal(, σ_ij^2_u[k,r] )  if [i,k]=1, 
    _j^(k)[·,r]= if [j,k]=0 and  _j^(k)[·,r] ∼Normal(, σ_ij^2_v[k,r]  )  if [j,k]=1,

where  is an identity matrix of appropriate dimension. 
 Using repeated applications of Corollary <ref>, the BIDIFAC+ objective can be shown to give the posterior mode of the model specified in (<ref>) for which ={σ_ij: i=1,,I, j=1,,J} and ={_u, _v} are fixed, and ^2_u[k,r]=^2_v[k,r]=1/λ_k for all r.  We extend this framework by allowing  and  to be estimated within a cohesive empirical variational Bayes framework.  We approximate the true posterior with a distribution q(·), for which the factor matrices are mutually independent: 
 
    q(,)=∏_k=1^K q_u,k(^(k)) q_v,k(^(k)).

The corresponding free energy is given by 

    F(q |, ) = E_q log∏_k=1^K q_u,k(^(k)) q_v,k(^(k))/p(|, , ) p({^(k)}_k=1^K |_) p({^(k)}_k=1^K |_),

with the densities p(·) in the denominator as specified in model (<ref>).  The resulting low-rank fit and decomposition is then given by 

    _= E_q ( ∑_k=1^K _·^(k)_·^(k) T) = ∑_k=1^K _^(k)
 
where for k=1,,K,

    _^(k) = E_q_u,q_v( _·^(k)_·^(k) T).


Algorithm <ref> describes an iterative approach to compute the empirical variational Bayes solution.  This algorithm converges to a blockwise minimum of the free energy (<ref>) over q and  with  fixed at the distinct minimizer of (<ref>) for each _ij separately.  Alternatively, one can update the variances  over the iterative algorithm to minimize free energy under the joint model.  In Algorithm <ref>, the number of modules K and the row- and column-set inclusions for the modules,  and , are assumed to be fixed.  For a smaller number of linked matrices, one can set  and  to enumerate all of the possible submatrices of shared, partially shared, or specific structure: K=(2^I-1)(2^J-1).  The estimated decomposition may still be sparse in that = if module k has no distinct low-rank structure, as demonstrated in Section <ref>.  If enumerating all submatrices of the row- and column-sets is not feasible due to a large number matrices, one can specify K,  and  based on prior knowledge. Alternatively,  we may approximate the fully enumerated decomposition with a large upper bound on the number distinct modules, K, and the row- and column-set inclusions  and  estimated empirically as in <cit.>.                
   
  
  



§ UNIQUENESS


  
  The uniqueness of decompositions into shared, partially shared, and specific low-rank structure requires careful consideration, as the terms in the model are generally not uniquely defined in general without further constraints.  For unidimensional decompositions of low-rank structure, conditions of orthogonality are commonly used to assure the different components are uniquely identified <cit.>; however, such conditions are not straightforward to extend to decompositions for bidimensionally linked data.  Alternatively, <cit.> showed that the terms in the BIDIFAC+ solution are uniquely defined under certain conditions, without orthogonality, as the minimizer of the structured nuclear norm objective in (<ref>).     Here we extend this result to a much broader class of penalties, that includes the EV-BIDIFAC decomposition.   
  
   For fixed row-set and column-set indicators for the modules,  and , let  be the set of possible decompositions that yield a given approximation _:

    = {|_ = ∑_k=1^K }.
  
In most setting the cardinality of || is infinite without further conditions.  Define P_ (·) for a vector  as a weighted sum of the singular values of a matrix:

    P_ () = ∑[r] _[r,r],

and let (·) give a structured penalty on the different modules of a decomposition,:

    () = ∑_k=1^K  P__k (),
 
with non-zero entries for each _k.  Theorem <ref> gives sufficient conditions for the minimizer of this penalty to be uniquely determined.   

  
  
Consider ∈ and let _^(k)^(k)_^(k) T give the SVD of  for k=1,,K. The following three properties uniquely identify .


  *  minimizes (·) over ,   

  * {_i^(k)[,r]: [i,k] = 1  and ^(k)[r,r]>0} are linearly independent for i=1, I,  

  * {_j^(k)[,r]: [j,k] = 1  and ^(k)[r,r]>0} are linearly independent for j=1,,J.

 
 
 The proof of Theorem <ref> is given in Appendix <ref>.  Note that the linear independence conditions are straightforward to verify, and are likely to hold in practice if the ranks of the terms in  are small relative  to the matrix dimensions.  As noted in Proposition <ref>, the terms in any accumulation point of Algorithm <ref> also give a coordinate-wise optimum of (·) for a certain set of penalties {_k}_k=1^K, and thus the EV-BIDIFAC solution is unique in this sense.     

  
  
If ∈ is an accumulation point of Algorithm <ref>, then it is a coordinate-wise optimum of (·) over  for some {_k}_k=1^K. 
 
 




§ MISSING DATA IMPUTATION


Now we consider the context in which the data are not fully observed and missing data must be imputed.  Let ℳ index observations in the full dataset  that are not observed: ℳ={(m,n): [m,n]  is missing}.  We develop an Expectation-Maximization (EM) approach for imputation that is analogous to previous approaches described for hard-thresholding <cit.>, soft-thresholding <cit.>, and BIDIFAC+ <cit.>.  The procedure for fixed estimates of the noise variance for each matrix, σ_ij, is described in Algorithm <ref>.  
     

Note that Step 4. in Algorithm <ref> is a partial maximization step,  as the free energy in (<ref>) is conditionally minimized for each module given the current imputed values, while Step 5. is an expectation step replacing the missing data with their expected value (<ref>) under the current model fit. Further, the algorithm can be shown to minimize the free energy over the missing data {[m,n] : (m,n) ∈ℳ}; this result is given as Theorem <ref>, and is appealing because all unknowns (model hyperpameters, parameters and missing entries) are then estimated via optimizing the same unified free energy objective.  

 
 
Algorithm <ref> iteratively minimizes the free energy (<ref>) over q(·), , and {[m,n] : (m,n) ∈ℳ}.    
 
 
  
 
 A proof of Theorem <ref> is provided in Appendix <ref>. For the choice of , we suggest two different approaches, depending on the type of missingness.  If entries are missing from an entire row or columns of a given data matrix _ij (i.e., row- or column-wise missing), then σ_ij can be estimated by the applying Theorem 2 to the complete submatrix {[m,n] : (m,n) ∉ℳ}.  If the matrix is missing entries without entire rows or columns, then we update σ_ij iteratively during Algorithm <ref> as follows at the end of each cycle: 1. apply Theorem 2 to the matrix _ij with currently imputed values to obtain σ'_ij, 2. update σ_ij=σ'_ij(M_iN_j/M_iN_j-|ℳ_ij|) where |ℳ_ij| is the number of missing entries in _ij.    










§ SIMULATIONS


Here we present a collection of simulations to illustrate and compare the proposed approach with respect to various criteria.  We explore several conditions covering different signal-to-noise ratios, missing data scenarios, and linked matrix patterns with shared and specific structure.  Further simulation results (e.g., exploring different matrix dimensions) are available at

<https://ericfrazerlock.com/ev-bidifac-sims.html>.     



 §.§ Single matrix



We first describe a simple simulation on a single matrix to illustrate the empirical variational Bayes approach to matrix factorization as a flexible compromise between a hard-thresholding and a conservative soft-thresholding approach.  We consider a fully observed matrix : 1000 × 100, with = c ^T + where the entries of , : M × 10 and : N × 10 are each independent standard normal, and c is a manipulated value controlling the signal-to-noise ratio.  We consider 10 values of c distributed on a log-uniform scale between c=0.05 and c=1, and generate 10 replications of the simulation for each value of c. For each generated dataset, we apply the following approaches:


  * OPT: The oracle operator on the singular values when the structure =c ^T is known: _OPT=_X _X^T where  is diagonal with entries that solve the OLS problem to minimize ||-_X _X^T||_F^2. 

  * EVB: the empirical variational Bayes approach applying Theorems <ref> and <ref>.

  * HT: The hard-thresholding approach described in Proposition <ref>, using the true rank R=10.

  * NN: The nuclear norm penalized soft-thresholding approach described in Proposition <ref>, with λ=√(M)+√(N) (this correspond to a threshold motivated by Proposition <ref> with true noise variance σ=1).      

For each setting and for each method we compute the relative squared error (RSE) and oracle-normalized squared error (ONSE)
 
    RSE=||-||_F^2/||||_F^2 and ONSE=||-||_F^2/||-_OPT||_F^2.

Figure <ref> shows the mean values for both RSE and ONSE under the different methods as a function of the signal-to-noise c. The hard-thresholding approach tends to overfit the signal, which can lead to particularly poor recovery when the signal is smaller relative to the noise. In contrast, the nuclear norm penalty is appropriately conservative in that it never overfits and performs better for lower signal; however, it tends to over-shrink for higher signal and can have several fold higher error than hard-thresholding.  The EVB approach is a flexible compromise between the two methods, performing comparably or better than both methods for all scenarios and substantially outperforming them for moderate levels of signal.  Moreover, performance of the EVB method is impressively close to the optimal in most scenarios.      



We consider another scenario with low-rank structure in which the underlying components have heterogeneous levels of signal.  First, we simulate '=' '^T where the entries of  : M × 10 and : 10 × N are independent standard normal. Then, we generate the signal as 
    = _S'_S _S'^T
 where _S is diagonal with entries sampled randomly from a log-uniform distribution between 0.05 √(MN) and √(MN), and =+ with entries of  independent standard normal.  Thus, some singular values of the true low-rank structure  may correspond to high signal, others may be of moderate signal, and others may be of weak or undetectable signal.  We generate 100 replications in this case, and for each we apply the EVB, HT, and NN methods described above, as well as the following additional approaches:

  
  * HT-OPT: the oracle hard-thresholding approach, in which the rank is selected that results in minimal squared error between the estimated signal and the true signal.  
  
  * NN-OPT: the oracle soft-thresholding approach, in which the nuclear norm penalty is selected that results in minimal squared error between the estimated signal and the true signal. 

  * RMT: the low-rank reconstruction approach proposed in <cit.>, based on minimizing squared error loss using asymptotic random matrix theory, with the true error variance σ^2=1. 
            
Boxplots showing distribution of RSE and ONSE for recovering the underlying structure across replications are shown in Figure <ref>.  The EVB and RMT methods have performance close to the oracle, and vastly outperform hard-thresholding and nuclear norm penalized approaches.  The relatively poor performance of HT-OPT suggests that singular values benefit from some shrinkage, while the poor performance of NN-OPT indicates that the level of shrinkage should not be uniform across singular values.  
 

 

We also compare missing data imputation performance of the EVB approach using Algorithm <ref>.  For each of the 100 simulated datasets generated, we consider scenarios in which 20%, 50%, or 80% of the entries are randomly held out as missing and imputed.  In addition to EVB imputation, we apply similar EM-imputation methods for the soft-thresholding (corresponding to softImpute of <cit.>) and hard thresholding <cit.> approaches. The RMT approach is not considered here, as it does not have an associated missing data imputation procedure.  For the NN-OPT and HT-OPT approaches, we select the value of λ or R, respectively, that minimizes squared imputation error for the signal of the missing data 

    RSE_miss=∑_(m,n) ∈ℳ ([m,n]-[m,n])^2/∑_(m,n) ∈ℳ[m,n]^2.

For oracle normalized imputation error ONSE_miss, we divide RSE_miss by the RSE for the OPT method in (<ref>).  The results are shown in Figure <ref>, and demonstrate that the EVB imputation approach is stable for different levels of missingness and substantially outperforms the other approaches.






 §.§ Two linked matrices


Here we generate two matrices, _1: 500 × 100 and _2: 500 × 100, that are linked by columns: _=[_1;_2]: 1000 × 100.  We consider this context primarily to allow comparison of the EV-BIDIFAC approach with several other methods that identify shared and unshared low-rank structure in this setting. 

First, we perform an illustrative simulation akin to that in Figure <ref> for a single matrix. We simulate shared structure _0=c _0 _0^T and individual structures  _1=c _1 _1^T _2=c _2 _2^T, where _0: 1000 × 5, _0: 100 × 5, _1: 500 × 5, _2: 500 × 5, _1: 100 × 5, and _2: 100 × 5 each have independent entries from a standard normal distribution. The observed data are then generated as _=_0+[_1;_2]+ where entries of  are independent standard normal. Thus c again controls the signal to noise ratio and we consider 10 values of c distributed on a log-uniform scale between c=0.05 and c=1, and generate 10 replications  for each value of c.  Here we consider estimates for the following methods:


  * EVB: the EV-BIDIFAC approach described in Algorithm <ref>

  * JIVE: the JIVE method implemented in <cit.> for true shared and unshared ranks R=5, R_1=5, R_2=5.

  * UNIFAC: the BIDIFAC method <cit.> for vertical integration using the true noise variance σ^2=1.	

Here JIVE is analogous to HT and UNIFAC to NN for a single matrix, as they involve iterative applications HT or NN, respectively, to the shared and unshared components.  For each method we compute RSE for the overall structure =_0+[_1;_2] as in (<ref>), and also compute the relative decomposition squared error (RDSE) as 

    RDSE=||_0-_0||_F^2+||_1-_1||_F^2+||_2-_2||_F^2/||_0||_F^2+||_1||_F^2+||_2||_F^2.

The results are shown in Figure <ref>. The results for overall RSE are similar to that for a single matrix, with UNIFAC appropriately conservative for lower signal but substantially underperforming JIVE for higher signal due to over-shrinkage.  However, per the RDSE results, UNIFAC can still identify the shared and unshared components more accurately with higher signal – this phenomenon was noted previously <cit.> and is in part due to orthogonality restrictions imposed by JIVE and related methods that are required for uniqueness of the decomposition without shrinkage.  The EVB approach again acts as a flexible compromise between the two methods, and substantially outperforms both for many s2n scenarios, particularly with respect to RDSE.  

 

Now, we compare RSE and RDSE under a scenario with heterogenous signal levels for the factors in each of the shared and unshared structures.  The rank-5 signal matrices _0, _1, and _2 are each generated as in (<ref>) with singular values again generated uniformly from a log-uniform distribution between 0.05√(MN)*0.05 and √(MN) for shared structure and between 0.05√(M_i N) and √(M_i N) for specific structure, and 100 replications are generated for this scenario.  In addition to the EVB, JIVE, UNIFAC methods we also consider the following approaches:


  * SLIDE: the structural learning and integrative decomposition (SLIDE) method <cit.>, with the true number of shared and unshared components (i.e., the ranks) specified.  

  * HNN.OPT: the hierarchical nuclear norm (HNN) approach <cit.>, with tuning parameters selected on a grid to minimize actual RSE for the true signal.   	

 Figure <ref> gives distribution of RSE and RDSE over the replications for the different methods. The EVB approach universally outperforms other methods in terms of both RSE and RDSE, and the benefit is large (by a factor of 2 or more) for both criteria in several cases.  While the HNN method is motivated by a model with shared and unshared low-rank structure, it does not provide an explicit decomposition of the structure and thus it is not shown for RDSE.    

 




 §.§ Bidimensionally linked matrices



Now, we consider the full bidimensionally linked framework in (<ref>), with I=J=2, N_1=N_2=50 and D_1=D_2=500.  Data were generated as 

    =∑_k=1^K  + ,

where the entries of  are independent standard normal.  There are K=9 possible distinct modules of low-rank structure in this setting: globally shared, shared across rows for either of the of the J=2 column sets, shared across columns for either of the I=2 row sets, or structure specific to any of the 4 matrices _ij.  We generate 100 datasets, where for each replication 5 of the 9 possible modules are randomly selected to have non-zero low-rank structure and the others are set to zero, =0.  All non-zero modules are generated to have rank 2 structure on the given submatrix as in (<ref>).  For each simulated dataset, we apply the following methods:


  * EB-BIDI: the approach in Algorithm <ref>, with  and  enumerating all K=9 possible modules.

  * BIDIFAC: the BIDIFAC algorithm described in <cit.> with  and  enumerating all possible modules and using the true noise variance σ^2=1.

  * EB-SEP: the EVB approach applied to each matrix _ij separately 

  * EB-JOINT: the EVB approach applied to the overall matrix .	

 For each method, we compute the overall RSE for _=∑_k=1^K as in (<ref>), and for EB-BIDI and BIDIFAC we compute the RDSE as 
 
    RDSE=∑_k=1^9 ||-||_F^2/∑_k=1^9 ||||_F^2.

 The results are summarized in Figure <ref>.  Overall RSE is substantially better for EB-BIDI vs. other methods; in comparison to EB-SEP and EB-JOINT, this illustrates the advantages of a parsimonious decomposition of the low-rank signal into the different shared and unshared components with respect recovering the overall signals.  Moreover, EB-BIDI has substantially better RDSE compared to BIDIFAC, demonstrating the advantages of the empirical variational Bayes objective with respect to accurately decomposing the structure into its different components.  Moreover, EB-BIDI correctly estimated modules with no true structure (=) as = for all of the 400 total instances (100 total datasets with 4 possible modules set to  in each case); on the other hand, it estimated = in only 19/500=3.8% of instances when ≠ 0, which occured when  was difficult to detect due to a small signal size.  This demonstrates how the approach can correctly recover the underlying rank-sparsity structure, only estimating non-zero structure for a module if it truly has distinct low-rank structure on its given rows and columns.               

 


We extend this simulation design to consider missing data imputation accuracy for the generated datasets. We consider two different missingness scenarios: an entry-wise missing scenario in which 20% of the entries are randomly set to missing in each matrix, and a blockwise missing scenario in which 10% of the columns and 10% of the rows are randomly selected to have all of their entries missing in each matrix.  In addition to imputation using the previous methods, we also consider the NN.OPT and HT.OPT methods as described in Section <ref>. For entrywise imputation the RSE_miss is computed as in (<ref>). For blockwise imputation, the true underlying structure is taken to be the sum of column-shared structure for missing rows and the sum of row-shared structure for missing columns, as this is the only estimable signal when an entire row or column is missing.  That is, letting _^C=∑_k:[·,k]=1 and _^R=∑_k:[·,k]=1, 

    RSE_miss=∑_(m,n) ∈ℳ^R (^C_[m,n]-_[m,n])^2+∑_(m,n) ∈ℳ^C (^R_[m,n]-_[m,n])^2/∑_(m,n) ∈ℳ^R^C_[m,n]^2+∑_(m,n) ∈ℳ^C^R_[m,n]^2,

where ℳ^R indexes row-missing entries and ℳ^C indexes column missing entries. 
The results are shown in Figure <ref>.  The EB-BIDI method substantially and consistently outperforms the other approaches with respect to missing data imputation, and the advantages are particularly large for blockwise missing imputation.  Note that the EB-SEP approach does not estimate any structure for blockwise missing data, as it is not estimable when considering each matrix separately.     
 




§ APPLICATION TO BRCA DATA



Here we describe an application to gene expression (mRNA) and microRNA (miRNA) data for breast cancer (BRCA) from the Cancer Genome Atlas  (TCGA) <cit.>. The data were processed as described in <cit.>, resulting in D_1=500 mRNAs and D_2=500 miRNAs for N_1=660 cancer tumor samples and N_2=86 non-cancerous breast tissue samples from unrelated individuals.  Thus, the data can be represented as bidimensionally linked matrices as in (<ref>), where _11 is mRNA data for cancer samples, _12 is mRNA data for normal samples, _21 is miRNA data for cancer samples and _22 is miRNA data for normal samples.  The decomposition of data into different modules of shared structured variation is very well-motivated in this context, as low-rank structure on any of the K=9 possible modules is plausible and interpretable.  For example, it is reasonable to expect shared structure across mRNA and miRNAs, as miRNAs can regulate mRNA. It is also reasonable to expect some shared structure across tumor and normal samples, as breast tumors arise from normal tissue and so they likely share some patterns of molecular variation.  However, it is also reasonable to expect specific structure in each case, e.g., patterns of molecular variability that are `cancer-specific' (i.e., not present in normal tissue) are of particular interest.  The BIDIFAC approach decomposes such shared and unique structure across both row sets (mRNA/miRNA) and column sets (tumor/normal), while other methods do not. Moreover, as demonstrated in our simulations, the empirical variational (EV) Bayes approach captures underlying low-rank structure more efficiently and accurately than other low-rank approximation methods. Thus, we expect our EV-BIDIFAC approach to decompose the underlying shared and unshared signals in an informative way, and accurately recover the underlying signal in the full dataset _·· better than alternative methods.    




We applied EV-BIDIFAC wherein  and  enumerate all K=9 possible modules, and significant distinct low-rank structure was identified for each module (≠ for all k).  Figure <ref> shows heatmaps of the original data (including missing data for miRNA and mRNA) and the resulting overall fit _=∑_k=1^K.  While all estimated structures are low-rank, it captures the original data well and imputation of the missing columns appears reasonable. Considering the relative amount of variation present in the different modules reveals the extent of shared variation across mRNA and miRNA, and across tumor and normal cells.  Interestingly, only 1.47% of the low-rank signal is in the module shared globally across all submatrices, while 59.8% of the signal is shared across mRNA and miRNA but specific to tumors. This suggests that the substantial coordinated activity between mRNA and miRNA in breast tumors is cancer-specific and not present in the normal cells they derive from.   

We further compare imputation accuracy for data under entry-wise, column-wise or row-wise missingness under a cross-validation scheme.  The accuracy of entry-wise imputation is of interest because it is a good proxy for the accuracy of the the overall fit of the estimated structure.  Column-wise imputation accuracy is of interest for two reasons: (1) the actual data have column-wise missingness, as not all samples have both mRNA and miRNA data in the full TCGA cohort, and (2) it provides a robust measure of the level of association between the two data sources, addressing how well can we predict gene expression with miRNA data, and vice versa.  Row-wise imputation accuracy is also of interest, as it provides a measure of similarity in molecular structure between cancer and normal samples (e.g., how well can we predict levels of a missing gene in cancer samples, given its associations in normal samples).

We create 20 folds in the data, where for each fold 5% of the columns, 5% of the rows, and 5% of the remaining entries in each matrix are randomly withheld as missing. For each fold, the missing data are imputed using the EV-BIDIFAC, BIDIFAC, EB-SEP or EB-JOINT approaches described in Section <ref>. We further consider nuclear norm penalized imputation (NN) and hard-thresholding (HT) using softImpute <cit.>, jointly for the full matrix (NN-JOINT and HT-JOINT) or separately for each submatrix (NN-SEP and HT-SEP).  In each case the nuclear norm penalty is selected via random matrix theory, and the rank for hard-thresholding is determined via cross-validation by optimizing MSE for an additional 5% of randomly withheld entries.  As an alternative to low-rank methods we consider k-nearest neighbor (KNN) imputation using the package impute <cit.> on the full matrix. Lastly, we consider a structured matrix completion approach <cit.> designed to impute a contiguous missing submatrix under a low-rank approximation using the StructureMC package <cit.>. As StructureMC imputes only one submatrix, it does not apply to randomly missing entries, and we hold out one set of rows or columns at a time to impute from the full matrix, with all other entries observed.    In each case, the relative squared error for the true observed data is computed and averaged across the four datasets yielding 

    MRSE_miss=1/4∑_i=1^2 ∑_j=1^2 ∑_(m,n) ∈ℳ_ij (_ij[m,n]-_ij[m,n])^2/∑_(m,n) ∈ℳ_ij_ij[m,n]^2.

The MRSE_miss under different imputation scenarios are summarized in Table <ref>.  The EV-BIDIFAC approach performs relatively well across all scenarios.  For entry-wise missing data EV-BIDIFAC performs much better than BIDIFAC (likely due to over-shrinkage of the signal for BIDIFAC) and similarly to EB-SEP, indicating that structure can be well-estimated by considering each dataset separately; however, separate estimation does not allow for row- or column imputation and does not estimate the shared variation that is of interest.  The MRSE_miss for row-missing data is relatively modest (0.90) compared to entry-wise missing (0.38), indicating that there is little shared information between tumor and normal cells.  The joint approach EB-JOINT on  performs relatively poorly across all scenarios as it tends to overfit shared variation when there  substantial specific variation across platforms (mRNA and miRNA) and cohorts (tumor and normal); this illustrates the benefits of a comprehensive decomposition of shared and specific variation across the constituent matrices.      
  



     

    



§ CONCLUSION AND DISCUSSION



The proposed EV-BIDIFAC approach can be considered as a flexible compromise between hard- and soft- thresholding techniques for the linked matrix decomposition problem, and has impressive performance with respect to 1. estimating underlying low-rank structure, 2. decomposing underlying structure into its shared, partially-shared and specific components, and 3. missing data imputation under a variety of scenarios.  An appealing aspect of the method is that it derives from a unified and intuitive model with a corresponding objective function that does not require any pre-hoc choices or parameters to tune.  In simulation comparisons the proposed approach tended to outperform other methods even when their parameters are set to the true values (e.g., true rank(s) or true error variance) or ideal values (e.g., to minimize error with knowledge of the true structure).  



While we have focused on obtaining a point estimate for underlying structure that minimizes free energy, this could also be used to empirically estimate hyperparameters and initialize a sampling algorithm for the associated fully Bayesian model.  This would capture uncertainty in the underlying factorization, which can be used for multiple imputation of missing values or to propagate uncertainty if the lower-dimensional components are used in subsequent statistical modeling.  A related Bayesian model is described in <cit.>, which also illustrates the importance of uncertainty propagation. 

Often, data take the form of higher-order tensors instead of matrices, and low-rank tensor factorizations are useful to uncover their underlying structure <cit.>. The models considered here may be extended to the tensor setting, e.g., by allowing  factors for each dimension of a tensor to have a distribution analogous to that for  and  in the matrix setting. However, the algorithms are not straightforward to extend, as there is likely not a closed-form solution for a tensor akin to that in Theorem <ref> for a matrix.  Extending this approach to the setting of a single tensor or linked tensors is an interesting    direction for future work.   

      



§ SUPPLEMENTARY MATERIALS
 

Reproducible R scripts for all results presented in this manuscript are available in a supplementary zipped folder.  Software to perform the EV-BIDIFAC method is available at <https://github.com/lockEF/bidifac>. Additional simulation study results are available at <https://ericfrazerlock.com/ev-bidifac-sims.html>.    



§ ACKNOWLEDGEMENT
   

This work was supported by the NIH National Institute of General Medical Sciences (NIGMS) grant R01-GM130622. 





§ PROOFS




 §.§ Proof of Theorem <ref>



The proof for Theorem <ref> follows a similar structure to that for Theorem 1 in <cit.>. We first establish four lemmas, Lemmas <ref>, <ref>, <ref>, and <ref>, which are useful to prove the main result.
 
  Take two decompositions  ∈ and ∈, and assume that both minimize the structured penalty for given {}_k=1^K: 

    ()=()=min   ().

Then, for any α∈ [0,1],

    ||α+(1-α)||_* = α ||||_*+(1-α) ||||_*

for k=1,,K.
 


Because  is a convex space and  is a convex function, the set of minimizers of   over  is also convex.  Thus, 

    ({α + (1-α) }_k=1^K )=min   ().

The result follows from the convexity of P_, which implies that for any two matrices of equal size  and , 

    P_(α + (1-α) ) ≤α P_() + (1-α)P_().
 
  Applying (<ref>) to each additive term  in  gives 

    ({α + (1-α) }_k=1^K )    ≤α() + (1-α) () 
       =  min   ().

Because {α + (1-α) }_k=1^K ∈, the inequality  in (<ref>) must be an equality,  and it follows that the inequality (<ref>) must be an equality for each penalized term in the decomposition.   




Take two matrices  and . If ||+||_*=||||_*+||||_*, and _+ ^T is the SVD of +, then = ^T where  is diagonal and ||||_*=||||_*, and = ^T where  is diagonal and ||||_*=||||_*. 


This result is proved in Lemma 2 of <cit.>.
 



Take two matrices  and . If P_ (+)=P_ () +P_(), and _+ ^T is the SVD of +, then = ^T where  is diagonal and P_()=P_(), and = ^T where  is diagonal and P_()=P_(). 


The result follows from repeated applications of Lemma <ref> to the rank-1 terms of  and .  Note that P_ (+)=P_ () +P_() implies 

    P_(∑_r _[·,r] _[r,r]_[·,r]^T+_[·,r] _[r,r]_[·,r]^T  ) = ∑_r [r] (_[r,r] + _[r,r]),

which requires that 
    || _[·,r] _[r,r]_[·,r]^T+_[·,r] _[r,r]_[·,r]^T ||_*
    =|| _[·,r] _[r,r]_[·,r]^T||_*+||_[·,r] _[r,r]_[·,r]^T ||_*
 
for each r.
 



Take two decompositions  ∈ and ∈, and assume that both satisfy 

    ()=()=min   ().
 Then, = _^(k)_^(k) T and = _^(k)^(k)_^(k) T where 
_^(k): M × R_k and _^(k): N × R_k have orthonormal columns, and
^(k) and ^(k) are diagonal.


The result follows as a direct corollary of Lemmas <ref> and <ref>,  as Lemma <ref> implies P_(+)=P_(|)+P_() for each k, and then Lemma <ref> 
implies the result. 	


Theorem <ref> is then established as follows:










Take two decomposition  and  that satisfy properties 1., 2., and 3. of Theorem 1; we will show that =. For each k=1,,K, write = _^(k)_^(k) T and = _^(k)^(k)_^(k) T as in Lemma <ref>.   Then, it suffices to show that ^(k)[r,r]=^(k)[r,r] for all k,r.  

First, consider module k=1 with [,1]=[1  0  ⋯  0]^T and [,1]=[1   0  ⋯ 0]^T.    By way of contradiction, assume ^(1)[1,1]>0 and ^(1)[1,1]=0.  The linear independence of {_j^(k)[,r]: ^(k)[r,r]>0} and {_j^(k)[,r]: ^(k)[r,r]>0} implies that

    (_) = {_^(k)[,r]: ^(k)[r,r]>0}={{_^(k)[,r]: ^(k)[r,r]>0}.

Thus, ^(1)[,1]] ∈{{_^(k)[,r]: ^(k)[r,r]>0}, and it follows from the orthogonality of ^(1)[,1] and {^(1)[,r], r>1} that 
    _^(1)[,1] ∈{{_^(k)[,r]: ^(k)[r,r]>0  and   k>1}.

Moreover, because _i^(1)= for any i>1 and {_i^(k)[,r]: ^(k)[r,r]>0} are linearly independent it follows that
   
    _^(1)[,1] ∈{_^(k)[,r]: ^(k)[r,r]>0,   k>1,   and [i,k]=0  for any  i>1}.

Note that (<ref>) implies _1^(1)[,1] ∈(_12 + ⋯ + (_1J),
however, this is contradicted by the linear independence of _1^(1)[,1] and {_i^(k)[,r]: ^(k)[r,r]>0, k>1}.  Thus, we conclude that ^(1)[1,1]>0 implies ^(1)[1,1]>0.  Analogous arguments show that ^(k)[r,r]>0 if and only if ^(k)[r,r]>0 for any pair (r,k).  It follows that  {_i^(k)[,r]: ^(k)[r,r]>0  or ^(k)[r,r]>0} are linearly independent for i=1, I, and {_j^(k)[,r]: ^(k)[r,r]>0  or ^(k)[r,r]>0} are linearly independent for j=1,,J.  Thus,   

    ∑_k=1^K _^(k) (^(k)-^(k)) _^(k) T = ∑_k=1^K  -  = _ - _=

implies that ^(k)[r,r]=^(k)[r,r] for all k,r.  


 




 §.§ Proof of Theorem <ref>


It is clear that the partial `M-steps' in Algorithm <ref> each partially optimize the free energy.  It remains to be shown that the missing entries minimize free energy when they are replaced with their conditional expectation under the model. Note that the free energy (<ref>) can be expressed as 

    E_q log(∏_k=1^K q_u,k(_k) q_v,k (_k) ) - E_q log(p({}_k=1^K |_) p({}_k=1^K |_))-E_q log(p({|, ,  ) ).

Consider the last term in the expression, decomposing the missing and non-missing entries of :

    E_q log(p(|, ,  ) )    = E_q log(p({[m,n]: (m,n) ∉ℳ}|, ,  ) p({[m,n]: (m,n) ∈ℳ}|, ,  ) ) 
       = E_q log(p({[m,n]: (m,n) ∉ℳ}|, ,  ) ) + 
                        E_q log(p({[m,n]: (m,n) ∈ℳ}|, , )).

Missing data {[m,n]: (m,n) ∈ℳ} are random variables that are independent and normally distributed given {, } as in (<ref>).  Thus, the last term in (<ref>) can be expressed as follows:

    E_q log(p({[m,n]: (m,n) ∈ℳ}|, , ))    = c_1 + ∑_(m,n) ∈ℳ1/2 ^2[m,n]  E_q ([m,n]-𝐒_[m,n])^2 
       ≥ c_1 + ∑_(m,n) ∈ℳ1/2 ^2[m,n]  E_q(𝐒_[m,n])-𝐒_[m,n])^2 .

Thus, for any given q, the free energy is minimized at _[m,n]=E_q S_[m,n]=_[m,n] for (m,n) ∈ℳ. 
















.








29

#1ISBN #1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<https://doi.org/#1>et al.#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<><#>1#1#1#1#1#1#1#1#1#1#1#1#1#1PreBibitemsHook


[Attias1999]attias1999

Attias, H.:
Inferring parameters and structure of latent variable models by
  variational Bayes.
In: Proceedings of the Fifteenth Conference on Uncertainty in
  Artificial Intelligence,
pp. 21–30.
Morgan Kaufmann Publishers Inc.,
San Francisco, CA, USA
(1999)




[Cai et al.2016]cai2016structured

Cai, T.,
Cai, T.T.,
Zhang, A.:
Structured matrix completion with applications to genomic data
  integration.
Journal of the American Statistical Association
111(514),
621–633
(2016)




[Feng et al.2018]feng2018angle

Feng, Q.,
Jiang, M.,
Hannig, J.,
Marron, J.:
Angle-based joint and individual variation explained.
Journal of multivariate analysis
166,
241–265
(2018)




[Fox and Roberts2012]fox2012tutorial

Fox, C.W.,
Roberts, S.J.:
A tutorial on variational Bayesian inference.
Artificial intelligence review
38,
85–95
(2012)




[Gavish and Donoho2017]gavish2017optimal

Gavish, M.,
Donoho, D.L.:
Optimal shrinkage of singular values.
IEEE Transactions on Information Theory
63(4),
2137–2152
(2017)




[Gaynanova and
  Li2019]gaynanova2019structural

Gaynanova, I.,
Li, G.:
Structural learning and integrative decomposition of multi-view data.
Biometrics
75(4),
1121–1132
(2019)




[Hastie and Mazumder2021]softImpute

Hastie, T.,
Mazumder, R.:
softImpute: Matrix Completion Via Iterative Soft-Thresholded SVD.
(2021).
R package version 1.4-1.
<https://CRAN.R-project.org/package=softImpute>




[Hastie et al.2021]KNN

Hastie, T.,
Tibshirani, R.,
Narasimhan, B.,
Chu, G.:
Impute: Impute: Imputation for Microarray Data.
(2021).
R package version 1.68.0




[Josse and Sardy2016]josse2016adaptive

Josse, J.,
Sardy, S.:
Adaptive shrinkage of singular values.
Statistics and Computing
26,
715–724
(2016)




[Kolda and Bader2009]kolda2009tensor

Kolda, T.G.,
Bader, B.W.:
Tensor decompositions and applications.
SIAM review
51(3),
455–500
(2009)




[Kurucz et al.2007]kurucz2007methods

Kurucz, M.,
Benczúr, A.A.,
Csalogány, K.:
Methods for large scale SVD with missing values.
In: Proceedings of KDD Cup and Workshop,
vol. 12,
pp. 31–38
(2007)




[Lock et al.2013]lock2013joint

Lock, E.F.,
Hoadley, K.,
Marron, J.S.,
Nobel, A.B.:
Joint and Individual Variation Explained (JIVE) for integrated
  analysis of multiple data types.
The Annals of Applied Statistics
7(1),
523
(2013)




[Lock et al.2022]lock2020bidimensional

Lock, E.F.,
Park, J.Y.,
Hoadley, K.:
Bidimensional linked matrix factorization for pan-omics pan-cancer
  analysis.
Annals of Applied Statistics
16(1),
193–215
(2022)




[Liu and Zhang2019]StructureMC

Liu, Y.,
Zhang, A.:
StructureMC: Structured Matrix Completion.
(2019).
R package version 1.0.
<https://CRAN.R-project.org/package=StructureMC>




[Mazumder et al.2010]mazumder2010spectral

Mazumder, R.,
Hastie, T.,
Tibshirani, R.:
Spectral regularization algorithms for learning large incomplete
  matrices.
The Journal of Machine Learning Research
11,
2287–2322
(2010)




[Nakajima and
  Sugiyama2014]nakajima2014analysis

Nakajima, S.,
Sugiyama, M.:
Analysis of empirical map and empirical partially bayes: Can they be
  alternatives to variational Bayes?
In: Artificial Intelligence and Statistics,
pp. 20–28
(2014).
PMLR




[Nakajima et al.2013]nakajima2013global

Nakajima, S.,
Sugiyama, M.,
Babacan, S.D.,
Tomioka, R.:
Global analytic solution of fully-observed variational Bayesian
  matrix factorization.
Journal of Machine Learning Research
14(1),
1–37
(2013)




[Nakajima et al.2012]nakajima2012perfect

Nakajima, S.,
Tomioka, R.,
Sugiyama, M.,
Babacan, S.:
Perfect dimensionality recovery by variational Bayesian pca.
In: Pereira, F.,
Burges, C.J.,
Bottou, L.,
Weinberger, K.Q. (eds.)
Advances in Neural Information Processing Systems,
vol. 25
(2012)




[O'Connell and Lock2016]oconnell2016

O'Connell, M.J.,
Lock, E.F.:
R.JIVE for exploration of multi-source molecular data.
Bioinformatics
32(18),
2877–2879
(2016)




[Park and Lock2020]park2020integrative

Park, J.Y.,
Lock, E.F.:
Integrative factorization of bidimensionally linked matrices.
Biometrics
76(1),
61–74
(2020)




[Rudelson and
  Vershynin2010]rudelson2010non

Rudelson, M.,
Vershynin, R.:
Non-asymptotic theory of random matrices: extreme singular values.
In: Proceedings of the International Congress of Mathematicians 2010
  (ICM 2010) (In 4 Volumes) Vol. I: Plenary Lectures and Ceremonies Vols.
  II–IV: Invited Lectures,
pp. 1576–1602
(2010).
World Scientific




[Shabalin and
  Nobel2013]shabalin2013reconstruction

Shabalin, A.A.,
Nobel, A.B.:
Reconstruction of a low-rank matrix in the presence of Gaussian
  noise.
Journal of Multivariate Analysis
118,
67–76
(2013)




[Schouteden
  et al.2014]schouteden2014performing

Schouteden, M.,
Van Deun, K.,
Wilderjans, T.F.,
Van Mechelen, I.:
Performing disco-sca to search for distinctive and common information
  in linked data.
Behavior research methods
46(2),
576–587
(2014)




[Samorodnitsky
  et al.2024]samorodnitsky2022bayesian

Samorodnitsky, S.,
Wendt, C.H.,
Lock, E.F.:
Bayesian simultaneous factorization and prediction using multi-omic data.
Computational Statistics & Data Analysis
197, 107974 (2024) 




[TCGA Research Network
  et al.2012]cancer2012comprehensive

TCGA Research Network, :
Comprehensive molecular portraits of human breast tumors.
Nature
490(7418),
61
(2012)




[Tang and Allen2021]tang2021integrated

Tang, T.M.,
Allen, G.I.:
Integrated principal components analysis.
Journal of Machine Learning Research
22(198),
1–71
(2021)




[Wang and Stephens2021]wang2021empirical

Wang, W.,
Stephens, M.:
Empirical Bayes matrix factorization.
Journal of Machine Learning Research
22(120),
1–40
(2021)




[Yang and Michailidis2016]yang2015non

Yang, Z.,
Michailidis, G.:
A non-negative matrix factorization method for detecting modules in
  heterogeneous omics multi-modal data.
Bioinformatics
32(1),
1–8
(2016)




[Yi et al.2023]yi2023hierarchical

Yi, S.,
Wong, R.K.W.,
Gaynanova, I.:
Hierarchical nuclear norm penalization for multi-view data integration.
Biometrics
(2023)









 Paper Title: Deep Fréchet Regression.


*These authors contributed equally to this work


#1 1




1
  
  
  
  
  
  
  
  
  
    
  Deep Regression
    Su I Iao^*, Yidong Zhou^*, Hans-Georg Müller

  Department of Statistics, University of California, Davis
    June 11, 2024
=============================================================================================================

 

0
  

    Deep Regression


   

§ ABSTRACT 

Advancements in modern science have led to the increasing availability of non-Euclidean data in metric spaces. This paper addresses the challenge of modeling relationships between non-Euclidean responses and multivariate Euclidean predictors. We propose a flexible regression model capable of handling high-dimensional predictors without imposing parametric assumptions. Two primary challenges are addressed: the curse of dimensionality in nonparametric regression and the absence of linear structure in general metric spaces. The former is tackled using deep neural networks, while for the latter we demonstrate the feasibility of mapping the metric space where responses reside to a low-dimensional Euclidean space using manifold learning. We introduce a reverse mapping approach, employing local regression, to map the low-dimensional manifold representations back to objects in the original metric space. We develop a theoretical framework, investigating the convergence rate of deep neural networks under dependent sub-Gaussian noise with bias. The convergence rate of the proposed regression model is then obtained by expanding the scope of local regression to accommodate multivariate predictors in the presence of errors in predictors. Simulations and case studies show that the proposed model outperforms existing methods for non-Euclidean responses, focusing on the special cases of probability measures and networks.



Keywords:  curse of dimensionality, deep learning, regression, non-Euclidean data, Wasserstein space. 


1.9 


§ INTRODUCTION





Regression analysis serves as the cornerstone of statistical methodology, providing a fundamental framework for modeling complex relationships between response and predictor variables. The demand for comprehensive and versatile regression techniques has soared, driven by the prevalence of non-Euclidean responses. These responses encompass a broad spectrum of data types, ranging from networks <cit.>, covariance matrices <cit.> and trees <cit.> to distributions <cit.> and other complex structures.

Real-world applications across various domains underscore the need for advanced regression methods. For instance, in transportation science, understanding the evolution of transportation networks in response to factors like weather conditions and public health data is of paramount interest <cit.>. These and related scenarios are covered by a general approach where one views the responses as random objects situated in metric spaces <cit.>.  Challenges that we address in this paper include the absence of linear structure and the potential high dimensionality of multivariate predictors.

Regression with metric-space valued responses is thus of paramount interest. Early explorations include regression for non-Euclidean data using Euclidean embedding with distance matrices <cit.> and a Nadaraya-Watson kernel regression approach <cit.>. A regression approach was introduced more recently <cit.>. It extends linear and nonparametric regression techniques to accommodate metric-space valued responses through conditional means. However, these regression models either rely on strict assumptions akin to classical linear models for scalar responses or suffer from the curse of dimensionality in analogy to traditional nonparametric regression.

In practical applications, the need often arises to flexibly model metric-space valued responses with potentially high-dimensional predictors, so that these limitations prevent application of these methods for real-world challenges. Several recent works considered extending sufficient dimension reduction approaches <cit.>, single index models <cit.>, and principal component regression <cit.> to regression to address high-dimensionality of the predictors. These methods, while capable of reducing predictor dimensionality, are confined to the construction of a universal kernel or strong assumptions similar to those for classical single index and linear models.

To address these limitations, we harness deep learning <cit.>, taking advantage of its capability of handling high-dimensional predictors without compromising model flexibility. Deep learning is known for its capacity to automatically extract features from complex data, but has not been developed so far for non-Euclidean responses. The proposed regression model utilizes manifold learning <cit.> for the observed metric-space valued responses, generating low-dimensional representations. Manifold learning is a nonlinear dimensionality reduction technique that has proven useful for dimensionality reduction of infinite-dimensional functional data <cit.>, where data are assumed to lie on a low-dimensional manifold that is unknown. Indeed, many data residing in metric spaces, including parametric families of probability distributions or stochastic block models for networks, are expected to have such a low-dimensional manifold structure; we exploit such structure for the proposed statistical modeling. 

We deploy a deep neural network that takes the observed predictors as input and produces the low-dimensional manifold representation as output. One key innovation of the proposed approach is that we demonstrate that local regression can be utilized to map the low-dimensional manifold representation to the original metric space so that the final output resides in the metric space where the random objects are situated.  This map, which has not yet been considered in the manifold learning literature for metric-space valued data, is crucial for the implementation of a regression model in statistics, as it is mandatory for interpretability to predict the random objects residing in metric spaces, rather than their low-dimensional representations. This comprehensive framework makes it possible to handle high-dimensional predictors while concurrently accommodating the complexities intrinsic to metric-space valued responses.

Our key contributions are as follows. First, we utilize deep learning for regression, highlighting its efficacy as a powerful tool for the regression of metric-space valued responses. Second, our model can handle high-dimensional predictors for metric-space valued data that allow a manifold representation. Third, we investigate the convergence rate of deep neural networks in the presence of dependent sub-Gaussian noise accompanied by bias. This analysis is conducted under a well-established general composition assumption on the regression function <cit.>, which necessitates an extension of the existing theoretical framework due to the presence of dependence and bias. Although <cit.> introduced the convergence rate of deep neural networks with dependent and non-central response variables, their findings are limited to local dependence, while  
the dependence considered here does not rely on any specific form and is not restricted to local dependence. Fourth, we demonstrate how local regression can be used to map the low-dimensional representation back to the original metric space, for which we establish convergence in the presence of errors in predictors. Fifth, we examine the asymptotic properties of the proposed regression model, including its rate of convergence.  Lastly, we showcase results from various simulation settings and two real-data applications, demonstrating the superior performance of the proposed regression model compared to existing approaches.

The structure of the paper is as follows. In Section <ref>, we introduce notation and provide background on deep neural networks and local regression. The proposed regression model is introduced in Section <ref>, with the asymptotic convergence rate established in Section <ref>. Simulation results for distributional data are presented in Section <ref>. The proposed framework is illustrated in Section <ref> using networks arising from the New York yellow taxi records. Finally, we conclude with a brief discussion in Section <ref>. Auxiliary results and proofs as well as additional simulations for networks and a second data application for age-at-death distributions of human mortality are provided in the Supplementary Material.



§ PRELIMINARIES




 §.§ Notation
 
Let ℝ represent the set of real numbers and ℝ_+ denote the set of all positive real numbers. The set of all positive natural numbers is denoted as ℕ_+. Throughout this work, plain letters refer to (random) scalars and bold letters refer to (random) vectors. For a vector v, we define v^⊕0=1, v^⊕1=v, v^⊕2=vv^. We let · denote a norm and set ·=·_2 (the Euclidean norm for vectors) by default unless stated otherwise. Additionally, ·_0 denotes the number of nonzero entries of a matrix or vector, and ·_∞ represents the sup-norm for vectors, matrices, or functions. The notation a ≲ b implies the existence of a positive constant C such that a ≤ Cb. Moreover, a≍ b if both a≲ b and b≲ a. Finally, a ∧ b=min{a, b}, a ∨ b=max{a, b}.




 §.§ Deep neural networks


We provide an overview of deep neural networks (DNNs) as function approximations. The focus lies on the nonparametric regression model with p-dimensional covariates in the unit hypercube, represented by observations (X_i, U_i)∈ [0,1]^p×ℝ. Consider the following regression model


    U_i = g_0(X_i) + ϵ_i,    i=1,…,n,

where g_0 is an unknown function and ϵ_i is independent and identically distributed (i.i.d.) noise with zero mean and finite variance.


A DNN with L hidden layers and layer width 𝐩=(p_0, …, p_L, p_L+1)^ is defined recursively, forming a composite function g: ℝ^p_0↦ℝ^p_L+1,

    g(x)=W_L g_L(x)+b_L ,
       g_L(x)=σ{W_L-1 g_L-1(x)+b_L-1}, …, g_1(x)=σ(W_0 x+b_0),

where W_l∈ℝ^p_l+1× p_l is a weight matrix and b_l∈ℝ^p_l+1 is a shift vector for l=0,…,L. The rectifier linear unit (ReLU) activation function σ=max{0, x} <cit.> operates component-wise, that is, σ{(x_1,…,x_p_l)^}=(σ(x_1),…,σ(x_p_l))^. We focus on the ReLU activation function for its widespread use, empirical success, and theoretical support <cit.>. Observe that p_0=p and p_L+1=1 in the case of nonparametric regression with p-dimensional covariates. 







Given L ∈ℕ_+, 𝐩∈ℕ_+^L+2, s ∈ℕ_+ and a constant D>0, in analogy to <cit.>, we consider a class of sparse neural networks

    𝒢(L, s, 𝐩, D)={   g  of the form (<ref>): max_l=0,…,L{W_l_∞,b_l_∞}≤ 1, 
       ∑_l=1^LW_l_0+b_l_0 ≤ s,g_∞≤ D}.

We focus on DNNs with bounded weight matrices and shift vectors. This choice is motivated by empirical observations where the learned matrices and vectors tend to remain relatively small, especially when initialized with modest-sized matrices and vectors. Additionally, the consideration of sparse neural networks arises from the practical issue of overfitting deep feedforward neural networks with fully connected layers. This concern is addressed by pruning weights, reducing the total number of nonzero parameters, and sparsely connecting layers <cit.>.







To estimate the regression function g_0, the basic paradigm is to minimize the empirical risk

    ĝ = _g∈𝒢1/n∑_i=1^n{U_i - g(X_i)}^2.

Certain restrictions on the regression function g_0 are necessary to study the asymptotic properties of the estimator ĝ. Consider the Hölder class of smooth functions, 
where the ball of β-Hölder functions with radius M>0 is

    ℋ_p^β(𝔻, M)=   {g: 𝔻⊂ℝ^p ↦ℝ: 
       ∑_α:|α|<β∂^α g_∞+∑_α:|α|=⌊β⌋sup _x, y∈𝔻, x≠y|∂^α g(x)-∂^α g(y)|/x-y_∞^β-⌊β⌋≤ M},

where ⌊β⌋ denotes the largest integer strictly smaller than β, ∂^α=∂^α_1⋯∂^α_p with α=(α_1, …, α_p)^, and |α|=∑_k=1^p α_k. Let q ∈ℕ, β=(β_0, …, β_q)^∈ℝ_+^q+1 and 𝐝= (d_0, …, d_q+1)^∈ℕ_+^q+2, 𝐝̃=(d̃_0, …, d̃_q)^∈ℕ_+^q+1 with d̃_j ≤ d_j, j=0, …, q. We assume that the regression function g_0 belongs to a composite smoothness function class:

    ℋ(q, β, 𝐝, 𝐝̃, M)=    {g=g_q ∘⋯∘ g_0: g_j=(g_j 1, …, g_j d_j+1)^ and 
        g_jk∈ℋ_d̃_j^β_j([a_j, b_j]^d̃_j, M),  for some |a_j|,|b_j| ≤ M} .

Functions in this class are characterized by two kinds of dimensions, 𝐝 and 𝐝̃, where the latter represents the intrinsic dimension of the function. For example, if

    g(x)= g_31(   g_21[g_11{g_01(x_1) + g_02(x_2)} + g_12{g_03(x_3) + g_04(x_4)}] 
    
        +    g_22[g_13{g_05(x_5)+ g_06(x_6)} + g_14{g_07(x_7) + g_08(x_8)}]),  x∈[0,1]^8

and g_i j are twice continuously differentiable, then smoothness β=(2,2,2,2)^, dimensions d= (8,8,4,2,1)^ and d̃=(1,1,1,1)^. The composite smoothness class ℋ(q, β, 𝐝, 𝐝̃, M) contains a rich set of classical smoothness classes and has been widely adopted <cit.>.

Writing 

    κ_n=max _j=0, …, q n^-β̃_j /(2 β̃_j+d̃_j) with β̃_j=β_j ∏_k=j+1^q(β_k ∧ 1),
 we require the following common assumption on the structure of the DNN model <cit.>.


    
  * D≥max{M,1}, L=O(log n), s=O(n κ_n^2 log n) and n κ_n^2 ≲min_l=1, …, Lp_l ≤max_l=1, …, Lp_l ≲ n.

Assumption <ref> characterizes the flexibility of the neural network family 𝒢(L, s, p, D) as per (<ref>). While more flexible neural networks (with increased depth and width) are capable of achieving smaller approximation errors <cit.>, they often introduce larger estimation errors. Thus, Assumption <ref> provides a balance between approximation and estimation errors.













 §.§ Local regression

Let (Ω, d) be a totally bounded metric space with distance d: Ω×Ω↦ [0,∞) and 𝒯 be a closed interval in ℝ. 
Consider a random pair (Z^0, Y) with a joint distribution F supported on the product space 𝒯^r×Ω. We denote the marginal distributions of Z^0 and Y as F_Z^0 and F_Y, respectively, and the conditional distribution of Y given Z^0 as F_Y|Z^0. The mean and variance of random objects in metric spaces <cit.>, as generalizations of usual notions of mean and variance, are defined as

    y_⊕=_y∈ΩE{d^2(Y, y)},   V_⊕=E{d^2(Y, y_⊕)},

where the existence and uniqueness of the minimizer depend on the structural properties of the underlying metric space and are guaranteed for Hadamard spaces.

The conditional mean of Y given Z^0=z, corresponding to the regression function, is

    v(z)=_y∈Ω Q(y, z),   Q(·, z)=E{d^2(Y, ·)|Z^0=z}.

The lack of an algebraic structure in general metric spaces poses challenges in modeling and estimating the regression function v(z). To address this challenge, previous work <cit.> suggested leveraging the algebraic structure within the predictor space 𝒯^r. Let K be an r-dimensional kernel corresponding to a symmetric r-dimensional probability density function and write K_h(z)=h^-rK(H^-1z) with H=hI_r for a suitably chosen bandwidth h. Extending local linear regression to metric-space valued responses <cit.>, local regression (LFR) models conditional mean v(z) as a weighted mean,

    v_h(z)=_y∈ΩQ_h(y, z),   Q_h(·, z)=E{w(Z^0, z, h)d^2(Y, ·)}.

Here 

    w(Z^0, z, h)=1/μ_0-μ_1^μ_2^-1μ_1K_h(Z^0-z){1-μ_1^μ_2^-1(Z^0-z)},

where μ_j=E{K_h(Z^0-z)(Z^0-z)^⊕ j} for j=0, 1, 2. For an i.i.d. sample {(Z^0_i, Y_i)}_i=1^n, the corresponding empirical version is

    ṽ_h(z)=_y∈ΩQ̃_h(y, z),  Q̃_h(·, z)=1/n∑_i=1^nw̃(Z^0_i, z, h)d^2(Y_i, ·).

Here 

    w̃(Z^0_i, z, h)=1/μ̃_0-μ̃_1^μ̃_2^-1μ̃_1K_h(Z^0_i-z){1-μ̃_1^μ̃_2^-1(Z^0_i-z)},

where μ̃_j=n^-1∑_i=1^nK_h(Z^0_i-z)(Z^0_i-z)^⊕ j for j=0, 1, 2.



§ METHODOLOGY



Consider a totally bounded metric space (Ω, d) with distance d:Ω×Ω↦ [0,∞), and let ℳ⊂Ω be a manifold isomorphic to a subspace of the Euclidean space ℝ^r. Let (X, Y) be a random pair in ℝ^p×ℳ, where X, without loss of generality, is assumed to reside in the unit hypercube [0, 1]^p. Suppose {(X_i, Y_i)}_i=1^n are n independent realizations of (X, Y). Let r be the intrinsic dimension of the manifold ℳ. There exists a bijective representation map of the manifold ℳ:

    ψ:ℳ↦ℝ^r.
 
Algorithms such as ISOMAP <cit.>, locally linear embedding <cit.>, Laplacian eigenmaps <cit.>, or diffusion maps <cit.> are commonly employed to estimate the representation map ψ. Here we choose the ISOMAP algorithm; see <cit.> for more details. The manifold representation is useful across many scenarios, for example when one has latent parametric families of probability distributions or stochastic block models for networks, where the low-dimensional manifold closely aligns with the parameters of the underlying model <cit.>.


We propose a unified framework, the deep regression (DFR), to model the relationship between the metric-space valued response Y and a possibly high-dimensional multivariate predictor X, as summarized in Figure <ref>.






The regression function

    m=v∘g_0:[0, 1]^p↦Ω

exhibits a composite structure, where g_0=(g_01, …, g_0r)^ and v correspond to nonparametric regression using DNNs and local regression, respectively. When the intrinsic dimension is r>1, 
we fit one DNN for each coordinate. 


The proposed framework includes the ISOMAP representation ψ which however is not known and must be estimated from the data, yielding ψ̂. 
We postulate that the manifold can be well identified at the sample points through the ISOMAP algorithm. 

    
  *  Consider Z^0_i=(Z_i1^0, …, Z_ir^0)^=ψ(Y_i) and Z_i=(Z_i1, …, Z_ir)^=ψ̂(Y_i). For each j=1, …, r, there exists a function π_j:[0,1]^p× n↦ℝ such that
    
    Z_ij-Z_ij^0=π_j(X_1,…,X_n)(ϵ_ij+u_nj),   i=1,…,n,

    where the bias u_nj→ 0 as n→∞, and {ϵ_ij}_i=1^n are i.i.d. (mean zero) sub-Gaussian random variables with parameter 1 and independent of {X_i}_i=1^n, i.e., P(|ϵ_ij|≥ t)≤ 2exp{-t^2}. Furthermore, there exists a constant C_π_j such that 
    
    C_π_j = sup_{x_1,…,x_n}|π_j (x_1,…,x_n) | < ∞.


Assumption <ref> is not overly restrictive 
as it allows for dependence between the estimated low-dimensional representations Z_ij and includes potential bias u_nj that may be induced by the ISOMAP algorithm. The dependence among Z_ij across different i is quantified by π_j with an upper bound C_π_j that is specific to each coordinate. This assumption aligns with similar considerations made in <cit.> when employing DNNs to analyze time series data. The sub-Gaussian noise ϵ_ij reflects randomness associated with the manifold estimation process. 


We employ DNNs that take the observed predictors X_i as input and produce the estimated low-dimensional representations Z_i as output. For each j=1, …, r, by Assumption <ref> and the fact that Z_ij^0=g_0j(X_i) we may model the relationship between Z_ij and X_i as

    Z_ij = g_0j(X_i) + π_j(X_1,…,X_n)(ϵ_ij+u_nj).


Unlike the prevalent focus on independent Gaussian/sub-Gaussian noise without bias in the existing literature on DNNs <cit.>, the noise π_j(X_1,…,X_n)(ϵ_ij+u_nj) considered in (<ref>) exhibits dependence across different i and is accompanied by a vanishing bias u_nj, characterized by the randomness of a sub-Gaussian variable. Such generalizations necessitate a nontrivial expansion of the current theoretical framework for DNNs <cit.>; this extension is provided below as  Theorem <ref>.  
The estimation of g_0j is performed by minimizing the empirical risk 

    ĝ_j = _g∈𝒢1/n∑_i=1^n {Z_ij - g(X_i)}^2.




The proposed framework also deploys local regression using the  fitted value of DNNs Ẑ_i = ĝ(X_i)={ĝ_1(X_i),…,ĝ_r(X_i)}^ in lieu of the unobservable Z^0_i as predictors. This leads to an errors-in-variables version of  local regression,

    v̂_h(z)=_y∈ΩQ̂_h(y, z),  Q̂_h(·, z)=1/n∑_i=1^nŵ(Ẑ_i, z, h)d^2(Y_i, ·),

where 

    ŵ(Ẑ_i, z, h)=1/μ̂_0-μ̂_1^μ̂_2^-1μ̂_1K_h(Ẑ_i-z){1-μ̂_1^μ̂_2^-1(Ẑ_i-z)},

with μ̂_j=n^-1∑_i=1^nK_h(Ẑ_i-z)(Ẑ_i-z)^⊕ j for j=0, 1, 2.

Consequently, the regression function m as per (<ref>), relating the predictor X to the metric-space valued response Y, is estimated as

    m̂=v̂_h∘ĝ

with v̂_h and ĝ=(ĝ_1, …, ĝ_r)^ defined in (<ref>) and (<ref>). This comprehensive framework empowers us to effectively handle potentially high-dimensional predictors while simultaneously addressing the inherent complexities associated with metric-space valued responses. Further details are provided in Algorithm <ref>.



§ ASYMPTOTIC PROPERTIES
















We first derive the convergence rate of DNN estimates (<ref>), extending the existing theoretical framework for DNNs to accommodate dependent noise and bias.  
We then delve into the convergence rate of the local regression estimate (<ref>), considering multivariate predictors contaminated by errors induced by DNNs. Finally, the convergence rate of the deep regression estimate as per (<ref>) is established by combining the convergence rates of DNNs and local regression. For DNN, we require <ref> and the following common assumption <cit.>.

    
    
  * The target function g_0j belongs to the Hölder class ℋ (q,β,𝐝,𝐝̃,M) defined in (<ref>) for j = 1,…, r.
    

Assumption <ref> states that the spaces of g_0j is the Hölder class ℋ (q,β,𝐝,𝐝̃,M) which encompasses a rich class of smooth functions <cit.>; see Section <ref> for more details. The following result formalizes the convergence rate of DNN estimates (<ref>).

 





    Consider the nonparametric regression model (<ref>).
    
    Under <ref>, <ref> and <ref>, for any i=1,…,n and j = 1,…,r, there exists an estimator ĝ_j in (<ref>) such that
    
    E[{ĝ_j(X_i) - g_0j(X_i)}^2] = O(κ_n^2 log^3 n+|u_nj|)

    and
    
    E[{ĝ_j(X) - g_0j(X)}^2] = O(κ_n^2 log^3 n+|u_nj|)

    where  X is a new realization of the predictor that is independent of the sample {(X_i, Y_i)}_i=1^n and κ_n is defined in (<ref>). It immediately follows that
    
    E(Ẑ_i - Z_i^0^2)= O(κ_n^2 log^3 n+u^2_n)

    and 
    
    E(Ẑ - Z^0^2)= O(κ_n^2 log^3 n+u^2_n),

    where Ẑ=(ĝ_1(X),…,ĝ_r(X))^, Z^0 = (g_01(X),…,g_0r(X))^, u_n^2=max_j=1, …, r|u_nj| and g_0j are the components of g_0 as defined in (<ref>).

All proofs are provided in the Supplementary Material. In contrast to the prevailing emphasis on DNNs with independent Gaussian/sub-Gaussian noise, Theorem <ref> establishes the convergence rate of DNNs in the presence of dependent sub-Gaussian noise coupled with vanishing bias. This extension may also be useful for other work in the area of  DNN. Note that the convergence rates in Theorem <ref> depend on the bias term u_nj, smoothness β, and intrinsic dimension 𝐝̃ of functions g_0j as per (<ref>), rather than the predictor dimension p. As a result, the proposed method can circumvent the curse of dimensionality, exhibiting a faster convergence rate particularly when the intrinsic dimension 𝐝̃ remains relatively low. For instance, disregarding the bias term momentarily, consider the true function g_0j structured in (<ref>). For such a target function traditional smoothing methods are fraught with a slow convergence rate of order n^-2/p+4 = n^-1/6, whereas the proposed DNN approach has a convergence rate of order n^-2/5log^3/2 n, reaching the optimal one-dimensional nonparametric regression convergence rate up to a polylogarithmic factor. This accelerated convergence rate effectively mitigates the curse of dimensionality inherent in nonparametric regression.








































Next, we show the convergence rate for errors-in-variables multivariate local regression, for which we require the following assumptions.

    
  *  The minimizers v(z), v_h(z), ṽ_h(z) and v̂_h(z) for local regression as per (<ref>), (<ref>), (<ref>), and (<ref>) exist and are unique, the last two almost surely. Additionally, for any ε>0,
    
        
  * inf_d{v(z), y}>ε[Q(y, z)-Q{v(z), z}]>0,
        
  * lim inf_h→0inf_d{v_h(z), y}>ε[Q_h(y, z)-Q_h{v_h(z), z}]>0,
        
  * P(inf_d{ṽ_h(z), y}>ε[Q̃_h(y, z)-Q̃_h{ṽ_h(z), z}]>0)→1,
        
  * P(inf_d{v̂_h(z), y}>ε[Q̂_h(y, z)-Q̂_h{v̂_h(z), z}]>0)→1.
    
    
    
  *  Let B_δ(y)⊂Ω be the ball of radius δ centered at y and N{ε, B_δ(y), d} be its covering number using balls of size ε (see Definition <ref> in Supplementary Material <ref>). Then for any y∈Ω, 
    
    ∫_0^1[1+log N{δε, B_δ(y), d}]^1/2dε=O(1)  as δ→0.

    
    
  * There exists η_1, η_2, η_3>0, C_1, C_2, C_3>0 and γ_1, γ_2, γ_3>1 such that
    
    
  * inf_d{v(z), y}<η_1[Q(y, z)-Q{v(z), z}-C_1d{v(z), y}^γ_1]≥0,
    
  * lim inf_h→0inf_d{v_h(z), y}<η_2[Q_h(y, z)-Q_h{v_h(z), z}-C_2d{v_h(z), y}^γ_2]≥0,
    
  *  P(inf_d{ṽ_h(z), y}<η_3[Q̃_h(y, z)-Q̃_h{ṽ_h(z), z}-C_3d{ṽ_h(z), y}^γ_3]≥0)→1,
    
    
    
    
    
    



Assumption <ref> is a standard requirement to ensure the consistency of M-estimators, as outlined by <cit.>. We note that the existence and uniqueness of the minimizers in <ref> is guaranteed for the case of Hadamard spaces, the curvature of which is bounded above by 0 <cit.>. Assumption <ref> entails conditions on the metric entropy of Ω to control its size. 
In Proposition <ref>, it constrains the bracketing integral used to derive the tail bound, following Theorem 2.14.16 in <cit.>. 
The curvature assumption in <ref> regulates the behavior of (Q_h - Q), (Q̃_h - Q_h), and (Q̂_h - Q̃_h) near their minima, which makes it possible to derive convergence rates. 

These assumptions stem from empirical process theory and are frequently employed in the literature concerning statistical analysis for metric-space valued data <cit.>. 

We now provide some statistically relevant examples of metric spaces, for which we demonstrate in Section <ref> of the Supplementary Material that these metric spaces satisfy  Assumptions <ref>–<ref>; see Propositions <ref>, <ref> and <ref>.
















	Let Ω be the space of probability distributions on a closed interval of ℝ with finite second moments, equipped with the Wasserstein metric d_𝒲 where
	
    d_𝒲^2(μ, ν)=∫_0^1{F_μ^-1(p)-F_ν^-1(p)}^2dp

	for any two probability distributions μ, ν∈Ω. The Wasserstein space (Ω, d_𝒲) satisfies Assumptions <ref>–<ref> with γ_1=γ_2=γ_3=2.



	Let Ω be the space of graph Laplacians of undirected weighted networks with a fixed number of nodes m and bounded edge weights equipped with the Frobenius metric d_F. The space (Ω, d_F) satisfies Assumptions <ref>–<ref> with γ_1=γ_2=γ_3=2.



	Let Ω be the space of m-dimensional covariance matrices with bounded variances or correlation matrices equipped with the Frobenius metric d_F. The space (Ω, d_F) satisfies Assumptions <ref>–<ref> with γ_1=γ_2=γ_3=2.














We next derive the convergence rate of multivariate local regression, extending previous results for local regression (<ref>) with one-dimensional predictors. The proof relies on M-estimation techniques arising from empirical process theory. The kernel and distributional assumptions <ref> and <ref> listed in the Appendix are standard for local regression. In the following, r is as in (<ref>) the intrinsic dimension of the manifold representation (<ref>).


If <ref>, <ref>, <ref> (i)–(ii) and <ref> (i)–(ii) hold, then

    d{v_h(z), v(z)}=O(h^2/(γ_1-1))

as h→0. If furthermore nh^r→∞, then

    d{ṽ_h(z), v_h(z)}=O_p{(nh^r)^-1/{2(γ_2-1)}}.


In general, the convergence rates are determined by the local geometry near the minimum as quantified in <ref>. For metric spaces in Examples <ref>, <ref> and <ref>, γ_1=γ_2=2 and convergence rates of the bias and stochastic deviation are d{v_h(z), v(z)}=O(h^2) and d{ṽ_h(z), v_h(z)}=O_p{(nh^r)^-1/2}, respectively. With h=n^-1/(4+r), the multivariate local regression estimate achieves the rate d{ṽ_h(z), v(z)}=O_p(n^-2/(4+r)), corresponding to the well-known optimal rate for standard local linear regression.

The low-dimensional representations Z^0_i in (<ref>) are unobservable, necessitating the use of (<ref>) in practice, where the estimated representations Ẑ_i obtained through DNNs are substituted. Consequently, errors are introduced in the predictors of local regression, leading to the need to address the errors-in-variables problem. The following theorem characterizes the impact of these errors on the estimation.



Suppose <ref>, <ref>, <ref> hold. If Ẑ_i-Z^0_i=O_p(ζ_n) for i=1, …, n, Ẑ-Z^0=O_p(ζ_n), nh^r→∞, and h^-r-2ζ_n→0, then it holds for ṽ_h(·) and v̂_h(·) as per (<ref>) and (<ref>) that

    d{v̂_h(Ẑ), ṽ_h(Z^0)}=O_p{(h^-r-1ζ_n)^1/(γ_3-1)}.



For metric spaces in Examples <ref>, <ref> and <ref>, γ_3=2 and the convergence rate provided in Proposition <ref> reduces to O_p{(h^-r-1ζ_n). Combining Theorems <ref> and Proposition <ref> and <ref>, we have the following Theorem.


If <ref>, <ref>–<ref>, <ref>, <ref>, <ref>–<ref> hold and furthermore nh^r→∞, h^-r-2ζ_n→0, one has

    d{m̂(X), m(X)}=O_p{h^2/(γ_1-1)+(nh^r)^-1/{2(γ_2-1)}+(h^-r-1ζ_n)^1/(γ_3-1)},

where X is a new predictor independent of the sample {(X_i, Y_i)}_i=1^n and ζ_n=κ_nlog^3/2n+u_n.



The initial two terms h^2/(γ_1-1), (nh^r)^-1/{2(γ_2-1)} in Theorem <ref> give  the convergence rate of multivariate local regression, as outlined in Proposition <ref>, representing the optimal rate when Z_i^0, ψ, and g_0 are known. The final term, (h^-r-1ζ_n)^1/(γ_3-1), originates from Theorem <ref> and Proposition <ref>, reflecting errors introduced during the estimation of low-dimensional representations of metric-space valued responses (ISOMAP), DNNs, and the subsequent errors-in-variables problem.



If one assumes E{sup_i=1,…,n(Ẑ_i - Z_i^0)^2}=O(τ_n^2), the dependence of the convergence rate presented in Proposition <ref> on the intrinsic dimension of metric-space valued responses r would vanish. Specifically, the convergence rate in Proposition <ref> would be further improved to (h^-1τ_n)^1/(γ_3-1), leading to an enhanced convergence rate of the proposed model as h^2/(γ_1-1)+(nh^r)^-1/2(γ_2-1)+(h^-1τ_n)^1/(γ_3-1). The proof is provided in the Supplementary Material <ref>. While the primary focus of the current work is to devise a general framework for handling multivariate predictors and metric-space valued responses, it would be intriguing for future work to derive such a supremum rate for DNNs when dealing with dependent sub-Gaussian noise coupled with bias.  



§ IMPLEMENTATION AND SIMULATIONS




 §.§ Implementation details

The optimization problem in (<ref>) was addressed within Pytorch framework, where we opted for the Adam optimizer <cit.>, a widely used choice in deep learning known for its robust performance across diverse models and datasets. The weight matrices W_l and shift vectors b_l of the function g∈𝒢 as per (<ref>) were initialized using the default random initialization provided by Pytorch.

Manifold learning (ISOMAP) and DNNs involve tuning parameters such as the number of nearest neighbors k in the Dijkstra algorithm <cit.>,  the number of hidden layers L, the number of neurons p_l in each hidden layer, dropout rate, and learning rate. For simplicity, we maintained the same number of neurons in each hidden layer. The dropout rate <cit.> is the rate of randomly excluding neurons during training. The learning rate <cit.> determines the step size for gradient descent in the Adam algorithm. Tuning these parameters can be accomplished through a grid search, assessing empirical risk over a held-out validation set after each training iteration. We set aside 20% of the training set as the validation set in each run. The training process stops when the empirical risk on the validation set ceases to reliably improve.




















 §.§ Simulations for probability distributions

To assess the performance of the proposed model, we report the results of simulations for various settings. The random objects we consider first are univariate probability distributions equipped with the Wasserstein metric; see Example <ref>. The Wasserstein space is a geodesic metric space related to optimal transport <cit.> and is increasingly applied in various research domains, including population pyramids <cit.>, financial returns <cit.>, and multi-cohort studies <cit.>, among others <cit.>. Additional simulations for networks can be found in Supplementary Material <ref>. Despite the simulated networks not residing on a 2-dimensional manifold, the proposed model exhibits remarkable robustness as the sample size increases.

Consider the true regression function, represented as a quantile function, to be

    m(X) = E(η|X)+E(σ|X) Φ^-1(·),

which corresponds to a Gaussian distribution with mean and standard deviation depending on X. The distribution parameters of m(X) are generated conditionally on X, where the mean η and the standard deviation σ are assumed to follow a normal distribution and a Gamma distribution, respectively.

We consider sample sizes n=100,200,500,1000, with Q=500 Monte Carlo runs. In each Monte Carlo run, predictors X_i ∈ℝ^9, i=1, …, n, are independently sampled as

    X_i1∼ U(-1,0), X_i2∼ U(0,1), X_i3∼ U(1,2)
       
            X_i4∼ N(0,1), X_i5∼ N(-10,3), X_i6∼ N(10,3)
       
            X_i7∼Bernoulli(0.6), X_i8∼Bernoulli(0.7), X_i9∼Bernoulli(0.3).






For each i, the mean and standard deviation of the Gaussian distribution are generated conditionally on X_i as

    η_i|X_i∼ N(μ, 0.5^2), σ_i|X_i ∼Gamma(θ^2, θ^-1) where 
       μ=3 X_i8{sin (π X_i1)+cos(π X_i2)}  + X_i7(5X_i4^2+X_i5) ,
       θ=[3+0.5 X_i8{sin (π X_i1) +cos(π X_i2)} + X_i7|5X_i4^2 + X_i5|].






The corresponding distributional response is then constructed as Y_i = η_i + σ_i Φ^-1. 

To simulate practical scenarios where direct observations of probability distributions are not available, but rather independent data samples generated by the corresponding distribution are obtained, we independently sample 100 observations {y_ij}_j=1^100 from each distributional response Y_i. Consequently, one must initially estimate the distributional response Y_i from the random sample {y_ij}_j=1^100, introducing a bias to the regression model. This practical consideration aligns with prior research <cit.>, which addresses the issue by adopting the empirical measure in place of the unobservable distributional response Y_i. For the implementation of local regression using empirical measures, we follow the algorithm outlined in <cit.>. The bandwidths for the local regression in each Monte Carlo run are chosen as 10% of the range of the intermediate estimates Ẑ_i = ĝ(X_i).

For the qth Monte Carlo run out of Q=500 runs, with m̂_q(·) denoting the fitted regression function, the quality of the estimation is quantified by the mean squared prediction error (MSPE),

    MSPE_q = 1/100∑_i=1^100d_𝒲^2{m̂_q(X_i^test), m(X_i^test)},

where {X_i^test}_i=1^100 denote out-of-sample predictors and the metric d_𝒲 is the Wasserstein metric for probability distributions. The average quality of the estimation over the Q=500 Monte Carlo runs is quantified  by the average mean squared prediction error (AMSPE)

    AMSPE=1/Q∑_q=1^QMSPE_q.




























To assess the validity of the manifold assumption, Figure <ref>(a) presents a scatter plot depicting the two-dimensional representations of distributional responses {Y_i}_i=1^n alongside their true parameters (mean and standard deviation) for a sample size of n=500. The plot reveals a distinctive horseshoe shape in the two-dimensional representations, implying that the distributional responses adhere to a lower-dimensional manifold. Intriguingly, the two-dimensional representations closely align with the two-dimensional parameters of the distributional responses. This observation suggests that for distributional responses ISOMAP effectively identifies suitable low-dimensional representations corresponding to the latent parameter space. 


We conduct comparisons with global regression (GFR) <cit.> and local regression with sufficient dimension reduction (SDR) <cit.>. Figure <ref>(b) summarizes MSPEs for all Monte Carlo runs and various sample sizes using the proposed model, GFR and SDR. The MSPE decreases with increasing sample size, indicating the convergence of the proposed model to the target. Notably, the proposed model exhibits superior performance over GFR and SDR across all sample sizes, including the case of n=100. In Table <ref>, we present the AMSPE for different sample sizes. The proposed model (DFR) is seen to fare best across different methods. Despite the conventional need for large sample sizes in the performance of deep neural networks, the proposed model surprisingly demonstrates robustness for small sample sizes, potentially owing to its flexibility in accommodating complex regression relationships.




















































































§ DATA APPLICATION





    
    
    
    
    
    
    

















































Yellow and green taxi trip records in New York City (NYC), including pick-up and drop-off dates, locations, trip distances, payment methods, and passenger counts, are available at <https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page>. Additionally, we collect NYC weather history, including daily average temperature, humidity, wind speed, pressure, and total precipitation from <https://www.wunderground.com/history/daily/us/ny/new-york-city/KLGA/date>. The objective is to predict transport networks constructed from taxi trip records using relevant predictors. Given the potential influence of weather conditions on travel plans, the variability in travel patterns across different days of the week, and the impact of daily trip features on the taxi system, we construct a fifteen-dimensional predictor set including daily weather information, indicators for days of the week or holiday, and daily trip features averaged over each day; see Table <ref>.












We analyze yellow taxi trip records in Manhattan, excluding islands, and divide the 66 taxi zones into 13 regions based on preprocessing procedures outlined in <cit.>. While <cit.> focused on the effect of COVID-19 on the transport network, our study investigates the pre-COVID transport network. Therefore, we limit our analysis to a period of 1092 days from January 1, 2017, to December 31, 2019, excluding 3 outlier days.





The main interest regarding these traffic records is the transport network that represents daily passenger movement between the 13 regions. To this end, we build daily undirected networks with nodes standing for the 13 regions and edge weights representing the number of passengers traveling between the regions. Each of these networks is uniquely associated with a 13 × 13 graph Laplacian. The two-dimensional representations of 1092 graph Laplacians using MDS are shown in Figure <ref> which suggest a clear separation among Monday to Thursday, Friday or Saturday, and Sunday or holiday in the second dimension. Additionally, as the first dimension increases, the total ridership also increases. The scatter plot exhibits a horseshoe shape, indicating that the manifold assumption is plausible within this context.



The proposed model was applied to model the relationship between daily graph Laplacians and the fifteen-dimensional predictors. The MSPE was calculated using 10-fold cross-validation, averaged over 100 runs. The proposed method achieves better prediction performance, resulting in a 46% and 55% improvement in prediction accuracy compared to GFR <cit.> and SDR <cit.>, respectively.

To further investigate the effect of daily total precipitation on taxi traffic, we predicted networks with varying total precipitation levels and fixed all other covariates at their median levels. Figure <ref> shows the predicted networks at total precipitation levels of 0, 2, and 6 inches on Monday to Thursday, Friday or Saturday, and Sunday or holiday. We observed that edge weights decrease on Sunday or holiday compared with those on Monday to Saturday, regardless of total precipitation levels. This suggests that more people tend to stay at home on Sunday and holiday. Across different precipitation levels, the taxi traffic from Monday to Thursday is more concentrated in areas 105, 106, 107, and 108, which are primarily residential areas with popular destinations such as Penn Station and Grand Central Terminal. Conversely, taxi traffic on Friday and Saturday is more diffuse, spreading into areas 102, 103, and 104, which are home to popular bars, restaurants, chain stores, and high-end art galleries and museums. As the total precipitation increases, taxi traffic decreases, irrespective of the day of the week. When the total precipitation reaches 6 inches, corresponding to a rainstorm, the remaining taxi traffic for all three categories of days is primarily concentrated in areas 105, 106, 107, and 108. This may be related to unavoidable activities such as work-related events. However, discretionary travel plans, primarily observed in areas 102, 103, and 104, are more likely to be canceled due to adverse weather conditions.






































































































































































































§ DISCUSSION









The proposed deep regression model introduces a novel approach for analyzing metric-space responses using multivariate predictors. Extensive simulations and applications across diverse scenarios, including networks and probability distributions, highlight the versatility of the model. 
The proposed approach is flexible and requires no specific model assumptions. 
Key contributions include an in-depth exploration of the convergence rate of deep neural networks under dependent sub-Gaussian noise with bias and an extension of the scope of local regression to handle multivariate predictors, including the convergence rate in an errors-in-variables regression framework. The proposed approach can be easily adapted to various manifold learning methods, such as locally linear embedding, Laplacian eigenmaps and diffusion maps.

A limitation lies in the assumption of the existence of a low-dimensional representation of random objects. However, our simulations and applications demonstrate the broad applicability of the proposed method, showcasing robust performance even when the intrinsic dimension of metric-space valued responses is not low.
Open problems for future research include the supremum rate of DNN convergence when dealing with dependent sub-Gaussian noise with bias and the case of high-dimensional predictors.   















§ ACKNOWLEDGMENTS


This research was supported in part by NSF grant DMS-2310450.



§ APPENDIX A. KERNEL AND DISTRIBUTIONAL ASSUMPTIONS


    
  *  The r-dimensional kernel K(·) satisfies ∫_ℝ^rK(u)u_j^4du<∞ and ∫_ℝ^rK^2(u)u_j^6du<∞ for j=1, …, r and u=(u_1, …, u_r)^, and is Lipschitz continuous with compact support [-1, 1]^r. 


    
  *  Both the marginal density f_Z^0(·) of Z^0 and the conditional densities f_Z^0| Y(·, y) of Z^0 given Y=y exist and are twice continuously differentiable, the latter for all y∈Ω, and sup_z, y|(∂^2 f_Z^0| Y/∂z^2)(z, y)|<∞. In addition, for any open set U⊂Ω, ∫_UdF_Y| Z^0(z, y) is continuous as a function of z.



chicago







Supplementary Material
















 §.§ Additional Background

[Covering number]
    Let (Ω, d) be a metric space. Let S ⊆Ω be a subset and ε>0. The ε-covering number of S, denoted by N(ε, S, d), is the minimal number of balls of radius ε needed to cover the set S, i.e.,

    N(ε, S, d):=min{k: There exist y_1, …, y_k ∈Ω such that S ⊆⋃_i=1^k B_ε(y_i)},

where B_ε(y)={y' ∈Ω: d(y, y') ≤ε} denotes the closed ε-ball centered at y ∈Ω.
































 §.§ Verification of Model Assumptions






	
    The Wasserstein space (Ω, d_𝒲) defined in Example <ref> satisfies Assumptions <ref>–<ref>.


    As discussed in Example <ref> of <cit.>, the Wasserstein space (Ω, d_𝒲) satisfies the first three parts of <ref>, <ref>, and the first two parts of <ref> with γ_1=γ_2=2. It suffices to show <ref> (iv) and <ref> (iii). For any probability distribution y∈Ω, let F_y^-1 be the corresponding quantile function. Let ⟨·, ·⟩, ·_L^2, and d_L^2(·, ·) be the inner product, norm, and distance on the Hilbert space L^2(0, 1). For any y∈Ω, the map from y to F_y^-1 is an isometry from Ω to the subset of L^2(0, 1) formed by equivalence classes of left-continuous nondecreasing functions on (0, 1). The Wasserstein space Ω can thus be viewed as a subset of L^2(0, 1), which has been shown to be convex and closed <cit.>.
    
    Let
    
    B̂(z)=1/n∑_i=1^nŵ(Ẑ_i, z, h)F_Y_i^-1.

    Since 
    
    Q̂_h(y, z)   =1/n∑_i=1^nŵ(Ẑ_i, z, h)d_𝒲^2(Y_i, y)
       =1/n∑_i=1^nŵ(Ẑ_i, z, h)[d_L^2^2{F_Y_i^-1, B̂(z)}+d_L^2^2{B̂(z), F_y^-1}+
       2⟨ F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2]
       =Q̂_h(B̂(z), z)+d_L^2^2{B̂(z), F_y^-1}+
       2/n∑_i=1^nŵ(Ẑ_i, z, h)⟨ F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2

    and
    
    1/n∑_i=1^nŵ(Ẑ_i, z, h)⟨ F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2
       =⟨1/n∑_i=1^nŵ(Ẑ_i, z, h)F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2
       =⟨B̂(z)-B̂(z), B̂(z)-F_y^-1⟩_L^2
       =0,

    One has
    
    Q̂_h(y, z)=Q̂_h(B̂(z), z)+d_L^2^2{B̂(z), F_y^-1},

    whence
    
    v̂_h(z)=_y∈ΩQ̂_h(y, z)=_y∈Ωd_L^2^2{B̂(z), F_y^-1}.

    Then by the convexity and closedness of the Wasserstein space, the minimizer v̂_h(z) exists and is unique for any z∈ℝ^r. Hence <ref> (iv) is satisfied.
    
    In view of 
    
    1/n∑_i=1^nw̃(Z_i^0, z, h)=1,

    one can similarly show that 
    
    ṽ_h(z)=_y∈ΩQ̃_h(y, z)=_y∈Ωd_L^2^2{B̃(z), F_y^-1},

    where
    
    B̃(z)=1/n∑_i=1^nw̃(Z_i^0, z, h)F_Y_i^-1.

    Observe that ṽ_h(z), viewed as the best approximation of B̃(z) in Ω, is characterized by 
    
    ⟨B̃(z)-F_ṽ_h(z)^-1, F_y^-1-F_ṽ_h(z)^-1⟩_L^2≤0,  for all y∈Ω.

    It follows that
    
    Q̃_h(y, z)   =1/n∑_i=1^nw̃(Z_i^0, z, h)d_𝒲^2(Y_i, y)
       =Q̃_h{ṽ_h(z), z}+d_𝒲^2{ṽ_h(z), y}+
       2/n∑_i=1^nw̃(Z_i^0, z, h)⟨ F_Y_i^-1-F_ṽ_h(z)^-1, F_ṽ_h(z)^-1-F_y^-1⟩_L^2
       =Q̃_h{ṽ_h(z), z}+d_𝒲^2{ṽ_h(z), y}+2⟨B̃(z)-F_ṽ_h(z)^-1, F_ṽ_h(z)^-1-F_y^-1⟩_L^2
       ≥Q̃_h{ṽ_h(z), z}+d_𝒲^2{ṽ_h(z), y}

    for all y∈Ω. Therefore, <ref> (iii) holds for the Wasserstein space (Ω, d_𝒲) for any η_3>0, C_3=1 and γ_3=2.

    
    
    
    
    




    The metric space (Ω, d_F) defined in Example <ref> satisfies Assumptions <ref>–<ref>, where Ω is the space of graph Laplacians of undirected weighted networks with a fixed number of nodes m and bounded edge weights.


    
    As discussed in Theorem 3 of <cit.>, the space of graph Laplacians (Ω, d_F) satisfies the first three parts of <ref>, <ref>, and the first two parts of <ref> with γ_1=γ_2=2. It suffices to show <ref> (iv) and <ref> (iii). Let ⟨·, ·⟩_F and ·_F be the Frobenius inner product and norm on Ω. Define
    
    B̂(z)=1/n∑_i=1^nŵ(Ẑ_i, z, h)Y_i,  B̃(z)=1/n∑_i=1^nw̃(Z_i^0, z, h)Y_i.

    Similar to the proof of Proposition <ref>, one can show that
    
    v̂_h(z)=_y∈Ωd_F^2{B̂(z), y},  ṽ_h(z)=_y∈Ωd_F^2{B̃(z), y}.

    Then by the convexity and closedness of the space of graph Laplacians, the minimizer v̂_h(z) exists and is unique for any z∈ℝ^r. Hence <ref> (iv) is satisfied.
    
    Observe that ṽ_h(z), viewed as the best approximation of B̃(z) in Ω, is characterized by
    
    ⟨B̃(z)-ṽ_h(z), y-ṽ_h(z)⟩_F≤0,  for all y∈Ω.

    It follows that
    
    Q̃_h(y, z)   =1/n∑_i=1^nw̃(Z_i^0, z, h)d_F^2(Y_i, y)
       =Q̃_h{ṽ_h(z), z}+d_F^2{ṽ_h(z), y}+
       2/n∑_i=1^nw̃(Z_i^0, z, h)⟨ Y_i-ṽ_h(z), ṽ_h(z)-y⟩_F
       =Q̃_h{ṽ_h(z), z}+d_F^2{ṽ_h(z), y}+2⟨B̃(z)-ṽ_h(z), ṽ_h(z)-y⟩_F
       ≥Q̃_h{ṽ_h(z), z}+d_F^2{ṽ_h(z), y}

    for all y∈Ω. Therefore, <ref> (iii) holds for the space of graph Laplacians (Ω, d_F) for any η_3>0, C_3=1 and γ_3=2.




    The metric space (Ω, d_F) defined in Example <ref> satisfies Assumptions <ref>–<ref>, where Ω is the space of m-dimensional covariance matrices with bounded variances or correlation matrices.

The proof of Proposition <ref> is similar to that of Proposition <ref> and is thus omitted.




























 §.§ Proof of Theorem <ref>







To establish the convergence of estimation error, <cit.> applied an oracle-type inequality in his Theorem 2, supported by Lemma 4. In contrast to <cit.> where the prediction error was studied assuming independent Gaussian noise with mean 0, here we consider the case of dependent sub-Gaussian noise with possible bias that vanishes asymptotically as in (<ref>). Additionally, we investigate both out-of-sample and in-sample errors, given that the output of deep neural networks serves as the input for local regression. Understanding the in-sample error is crucial for examining the convergence rate of local regression. Revisiting Lemma 4 of <cit.>, we provide here a parallel result, Lemma <ref>, that is suitable for the type of dependent sub-Gaussian noise with bias that we assume for our data.  Theorem <ref> then follows along similar arguments as in Theorem 1 and Corollary 1 of <cit.>. 




    Let ξ_1, …, ξ_M be sub-Gaussian random variables with mean 0 and a common parameter C, then 
    E(max_l=1,…,M ξ_l^2)≲log M.




By the sub-Gaussian assumption, we have P(|ξ_l|≥√(t))≤ 2 e^-t/C^2. For any u>0, we can apply the union bound and observe the following:

    E(max_l=1,…,Mξ_l^2)    ≤ u + ∫_u^∞ P(max_l=1,…,Mξ_l^2 ≥ t) dt
       ≤ u + ∑_l=1^M∫_u^∞ P(|ξ_l| ≥√(t)) dt
       ≤ u + 2M∫_u^∞ e^-t/C^2dt
        = u + 2MC^2e^-u/C^2.

By selecting u≍log M, we conclude that 

    E(max_l=1,…,Mξ_l^2)≲log M.




Consider the p-variate nonparametric regression model (<ref>) with unknown regression function g_0j. Let g̃_j be any estimator taking values in 𝒢=𝒢(L, s, 𝐩, D) as per (<ref>) and N_δ=N(δ,𝒢,·_∞) be the covering number of 𝒢 using balls of size δ. Define

    Δ_n= E[1/n∑_i=1^n {Z_ij - g̃_j(X_i)}^2 - inf_g∈𝒢1/n∑_i=1^n{Z_ij-g(X_i)}^2]

and assume g_0j_∞≤ D. Then for any i=1, …, n and j=1, …, r, we have

    E[{g̃_j(X_i) - g_0j(X_i)}^2]≲inf_g∈𝒢E[{g(X)-g_0j(X)}^2] + log N_δ/n + Δ_n + δ + |u_nj|

and

    E[{g̃_j(X) - g_0j(X)}^2]≲inf_g∈𝒢E[{g(X)-g_0j(X)}^2] + log N_δ/n + Δ_n + δ + |u_nj|,

with X being an independent copy of X_1.








Define g^2_n = 1/n∑_k=1^n g(X_k)^2. Since E[{g̃_j(X_i) - g_0j(X_i)}^2]≤ 4 D^2 and E[{g̃_j(X) - g_0j(X)}^2]≤ 4 D^2, the inequality in (<ref>) and (<ref>) hold trivially if log N_δ≥ n. In the following, we focus on the case log N_δ < n.

Similar to Lemma 4 of <cit.>, the proof is divided into three parts which are denoted by (I)–(III).

    
  * Relate the in-sample error E[{g̃_j(X_i) - g_0j(X_i)}^2] and predicted error E[{g̃_j(X) - g_0j(X)}^2] to E(g̃_j-g_0j_n^2) via the inequality

    E[{g̃_j(X_i) - g_0j(X_i)}^2] ≲ E(g̃_j-g_0j_n^2) + log N_δ/n + δ .


    E[{g̃_j(X) - g_0j(X)}^2] ≲ E(g̃_j-g_0j_n^2) + log N_δ/n + δ .


    
  * For any estimator g̃_j taking values in 𝒢,

    | E{1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) g̃_j(X_k)}| ≲√(E(g̃_j-g_0j_n^2)  log N_δ/n) + δ


    
  * Show that

    E(g̃_j-g_0j_n^2) ≲inf _g ∈𝒢 E[{g(X)-g_0j(X)}^2]+log N_δ/n+Δ_n + δ + |u_nj|.



The inequality in (<ref>) and (<ref>) follow from (I) and (III).

(I): Since the inequality (<ref>) has been shown by Lemma 4(I) in <cit.>, we only need to show the inequality (<ref>). Given a minimal δ-covering of 𝒢, denote the centers of the balls by g_l. By construction, there exists a l^* such that g̃-g_l^*_∞≤δ. Without loss of generality, one can assume that g_l_∞≤ D by the definition of 𝒢. It follows that 

    |E[{g̃_j(X_i)-g_0j(X_i)}^2] - E{g̃_j-g_0j_n^2}|
       = |E[1/n∑_k=1^n{g̃_j(X_i)-g_0j(X_i)}^2 - {g̃_j(X_k)-g_0j(X_k)}^2]|
        = |1/n∑_k=1^n E[{g̃_j(X_i)-g_l^*(X_i)}^2 + {g_l^*(X_i) - g_0j(X_i)}^2 
        + 2{g̃_j(X_i)-g_l^*(X_i)}{g_l^*(X_i) - g_0j(X_i)} - {g̃_j(X_k)-g_l^*(X_k)}^2 
        - {g_l^*(X_k) - g_0j(X_k)}^2 - 2{g̃_j(X_k)-g_l^*(X_k)}{g_l^*(X_k) - g_0j(X_k)}]|
       ≲ E{|1/n∑_k=1^n f_l^*(X_k, X_i)| } + δ

where f_l^*(X_k, X_i)={g_l^*(X_i)-g_0j(X_i)}^2-{g_l^*(X_k)-g_0j(X_k)}^2. Furthermore, define R_l^*=D√(n^-1log N_δ)∨√(E[{g_l^*(X_i)-g_0j(X_i)}^2]). Using g_l^*-g̃_j_∞≤δ and the triangle inequality, we have R_l^*≤ D√(n^-1log N_δ)+√(E[{g̃_j(X_i)-g_0j(X_i)}^2]) +δ. Similarly, we can define f_l and R_l in the same way with l^* being replaced by l.

Set U=√(E[{g̃_j(X_i)-g_0j(X_i)}^2]) and T=max_l=1,…,N_δ|∑_i=1^n f_l(X_k, X_i) /(R_l D)|, we have

    |E[{g̃_j(X_i)-g_0j(X_i)}^2] - E[g̃_j-g_0j_n^2]| 
       ≲ E{|1/n∑_k=1^n f_l^*(X_k, X_i)/(R_l^* D)| R_l^*} + δ
       ≲1/nE{T (U + √(n^-1log N_δ) + δ) } + δ
       ≲1/n√(E(T^2))√(E(U^2)) + 1/n(√(n^-1log N_δ) + δ)E(T) + δ,

where the last inequality follows from the Cauchy-Schwarz inequality. Observe that E[f_l(X_k, X_i)]=0,|f_l(X_k, X_i)| ≤ 4 D^2 and

    {f_l(X_k, X_i)}    =2  [{g_l(X_i)-g_0j(X_i)}^2] ·1_k  i
       ≤ 2 E[{g_l(X_i)-g_0j(X_i)}^4] 
       ≤ 8 D^2 R_l^2 .

Up to now, all the settings match Lemma 4(I) of <cit.>. Therefore, using the same technique, we can show that

    E[{g̃_j(X_i) - g_0j(X_i)}^2] ≲ E(g̃_j-g_0j_n^2) + log N_δ/n + δ .










































(II): For any estimator g̃_j taking values in 𝒢, using Jensen's inequality and Assumption <ref>, we have

    |E{∑_k=1^n ϵ_kjπ_j(X_1,…,X_n){g̃_j(X_k)-g_l^*(X_k)}}| 
       ≤ C_π_jδ∑_k=1^n E(|ϵ_kj|) 
       ≤ n δ C_π_j√(E(ϵ_1j^2))
       ≤ n δ C_π_j.


Since 

    E{ϵ_kjπ_j(X_1,…,X_n) g_0j(X_k)}
       =E[E{ϵ_kjπ_j(X_1,…,X_n) g_0j(X_k) | X_1, …, X_n}]
       =0,

we also find

    |E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) g̃_j(X_k)]| 
        =|E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g̃_j(X_k)-g_0j(X_k)}]| 
       ≤|E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g̃_j(X_k)-g_l^*(X_k) } ]|
       + |E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g_l^*(X_k) - g_0j(X_k)}]| 
       ≤δ C_π_j+|1/√(n)E[∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g_l^*(X_k) - g_0j(X_k)}/√(n)g_l^*-g_0j_ng_l^*-g_0j_n ]| 
       ≤δ C_π_j+1/√(n)E{|ξ_l^*|(g̃_j-g_0j_n+δ)},

with

    ξ_l=∑_k=1^n ϵ_kjπ_j(X_1,…,X_n){g_l(X_k)-g_0j(X_k)}/√(n)g_l-g_0j_n .

Given X_1, …, X_n, ξ_l are sub-Gaussian with common parameter C_π_j. It follows from Lemma <ref> that E(ξ_l^*^2) ≤ E(max_l=1, …, N_δξ_l^2) ≲log N_δ. Using Cauchy-Schwarz inequality,

    E{|ξ_l^*|(g̃_j-g_0j_n+δ)}≲√(log N_δ){√(E(g̃_j-g_0j_n^2))+δ}.

Together with (<ref>) and (<ref>), the inequality in (II) follows.

(III): For any fixed g ∈𝒢, it follows from the definition of Δ_n that

    E[1/n∑_k=1^n{Z_kj-g̃_j(X_k)}^2] ≤ E[1/n∑_k=1^n{Z_kj-g(X_k)}^2]+Δ_n.
 
Using the inequality in (II), we have

    E(g̃_j-g_0j_n^2)     = E[1/n∑_k=1^n {g̃_j(X_k) - Z_kj + (ϵ_kj+u_nj)π_j(X_1,…,X_n)}^2]
       = E[1/n∑_k=1^n {g̃_j(X_k) - Z_kj}^2 + 1/n∑_k=1^n (ϵ_kj+u_nj)^2π_j^2(X_1,…,X_n) 
        + 2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - Z_kj}]
       ≤Δ_n + E[1/n∑_k=1^n {g(X_k) - Z_kj}^2 + 1/n∑_k=1^n (ϵ_kj+u_nj)^2π_j^2(X_1,…,X_n) 
        + 2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - Z_kj}]
       ≤Δ_n + E(g-g_0j_n^2) + E{2/n∑_k=1^n (ϵ_kj+u_nj)^2π_j^2(X_1,…,X_n)}
        + E[2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - Z_kj - g(X_k) + g_0j(X_k)}]
        = Δ_n +  E[{g(X)-g_0j(X)}^2] 
        + E[2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - g(X_k)}]
       ≲Δ_n + E[{g(X)-g_0j(X)}^2]
        + E{1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) g̃_j(X_k)} + |u_nj| 
       ≲Δ_n +  E[{g(X)-g_0j(X)}^2]
        +√(E{g̃_j-g_0j_n^2}log N_δ/n) + δ + |u_nj|.


Finally, applying (C.4) in <cit.>, the result follows.



Using Lemma <ref> to replace Lemma 4 of <cit.>, 




the remaining proof follows the proof of Theorem 1 and Corollary 1 in <cit.>. Let g̃_j = ĝ_j be the minimizer of empirical risk (<ref>), one has Δ_n = 0. Then for any i=1,…,n and j=1,…,r, we have

    E[{ĝ_j(X_i)-g_0j(X_i)}^2] ≲ |u_nj| + κ_n^2 log^3 n

and

    E[{ĝ_j(X)-g_0j(X)}^2] ≲ |u_nj| + κ_n^2 log^3 n.

The result follows.




































































 §.§ Proof of Proposition <ref>


We present the elementary results of an auxiliary Lemma <ref> and its proof, extending the well-known results from <cit.>. The key quantities of interest are μ_j = E{K_h(Z^0-z)(Z^0-z)^⊕ j}, τ_j(y) = E{K_h(Z^0-z)(Z^0-z)^⊕ j|Y=y} and the estimators μ̃_j=1/n∑_i=1^n K_h(Z^0_i-z)(Z^0_i-z)^⊕ j, for j=0,1,2. These quantities are fundamental in local linear smoother, as they relate to the bias and variance properties of local regression.


    Suppose <ref> and <ref> hold. Then,

    μ_0     = f_Z^0(z) K_10 + O(h^2),
    μ_1     = h^2  K_12∂ f_Z^0(z)/∂z + O(h^3 1),
    μ_2     = h^2 f_Z^0(z) K_12 + O(h^4 11^),

where K_1j = ∫_ℝ^r K(u) u^⊕ j du for j=0,2. Furthermore, we have

    μ̃_j=μ_j+O_p{(h^2 j-r n^-1)^1 / 21^⊕ j}
 and

    τ_0(y)     = f_Z^0|Y(z,y) K_10 + O(h^2),
    τ_1(y)     = h^2 K_12∂ f_Z^0|Y(z,y)/∂z + O(h^31),
    τ_2(y)     = h^2 f_Z^0|Y(z,y) K_12 + O(h^4 11^),

where the order term is uniform over y ∈Ω.



By definition, we have

    μ_j     = E{K_h(Z^0-z)(Z^0-z)^⊕ j}
        = ∫ h^-rK{H^-1(x-z)}  (x-z)^⊕ j f_Z^0(x) dx,


    τ_j(y)     = E{K_h(Z^0-z)(Z^0-z)^⊕ j|Y=y}
        = ∫ h^-rK{H^-1(x-z)}  (x-z)^⊕ j f_Z^0|Y(x,y) dx.

The statements regarding μ_j and τ_j(y) follow from Assumptions <ref> and <ref> using a second-order Taylor expansion of the densities f_Z^0 and f_Z^0|Y.

    μ_0     = ∫ h^-rK{H^-1(x-z)}f_Z^0(x) dx
        = ∫ K(u)f_Z^0(z + Hu) du
        = f_Z^0(z) K_10 + h (∂ f_Z^0(z)/∂z)^ K_11 + O(h^2)
        = f_Z^0(z) K_10 + O(h^2).


    μ_1     = ∫ h^-rK{H^-1(x-z)}  (x - z) f_Z^0(x) dx
        = h ∫u K(u) f_Z^0(z + Hu) du
        = h { f_Z^0(z) K_11 + h  K_12∂ f_Z^0(z)/∂z + O(h^2)}
        = h^2  K_12∂ f_Z^0(z)/∂z + O(h^3 1)).


    μ_2     = ∫ h^-rK{H^-1(x-z)}  (x - z)(x - z)^ f_Z^0(x) dx
        = h^2 ∫uu^ K(u) f_Z^0(z + Hu) du
        = h^2{f_Z^0(z) K_12 + h  ∫uu^u^∂ f_Z^0(z)/∂z K(u)du + O(h^2 11^)}
        = h^2 f_Z^0(z) K_12 + O(h^4 11^).


    τ_0(y)     = ∫ h^-rK{H^-1(x-z)} f_Z^0|Y(x,y) dx
        = ∫ K(u)f_Z^0|Y(z + Hu,y) du
        = f_Z^0|Y(z,y) K_10 + h (∂ f_Z^0|Y(z,y)/∂z)^ K_11 + O(h^2)
        = f_Z^0|Y(z,y) K_10 + O(h^2).


    τ_1(y)     = ∫ h^-rK{H^-1(x-z)}  (x-z) f_Z^0|Y(x,y) dx
        = h ∫u K(u)f_Z^0|Y(z + Hu,y) du
        = h{f_Z^0|Y(z,y) K_11 + h K_12∂ f_Z^0|Y(z,y)/∂z  + O(h^21)}
        = h^2 K_12∂ f_Z^0|Y(z,y)/∂z + O(h^31).


    τ_2(y)     = ∫ h^-rK{H^-1(x-z)}  (x - z)(x - z)^ f_Z^0|Y(x,y) dx
        = h^2 ∫uu^ K(u) f_Z^0|Y(z+Hu,y) du
        = h^2{f_Z^0|Y(z,y) K_12 + h  ∫uu^u^∂ f_Z^0|Y(z,y)/∂z K(u)du + O(h^2 11^)}
        = h^2 f_Z^0|Y(z,y) K_12 + O(h^4 11^).

Next, by definition

    μ̃_j=1/n∑_i=1^n K_h(Z^0_i-z)(Z^0_i-z)^⊕ j.

Note that E(μ̃_j)=μ_j and

    E{K_h^2(Z^0_i-z)(Z_il-z_l)^2 j}=h^2 j-r∫ K(u)^2 u_l^2 j f_Z^0(z+Hu) d u=O(h^2 j-r),

where Z_il is the lth element of the random vector Z^0_i and z_l is the lth element of the vector z. It follows that (μ̃_j)=O(h^2 j-r n^-11^⊕ j), proving the result for the μ̃_j.



    Suppose <ref> and <ref> hold and furthermore h → 0, n h^r →∞. Then d{ṽ_h(z), v_h(z)}=o_p(1).



The symbol ⇝ denotes weak convergence and the notation l^∞(Ω) denotes the space of bounded functions on Ω. According to Corollary 3.2.3 in <cit.> and Assumption <ref> (i), it is sufficient to 

demonstrate the convergence of sup_y∈Ω|Q̃_h(y,z)-Q_h(y,z)| to zero in probability. Achieving this requires showing Q_h(·,z)-Q̃_h(·,z) ⇝ 0 in l^∞(Ω) and then applying Theorem 1.3.6 of <cit.>. Finally, by Theorem 1.5.4 and 1.5.7 in <cit.>, we can establish weak convergence by showing the following:


  
  * Q̃_h(y,z)-Q_h(y,z)=o_p(1) for all y∈Ω and
  
  * Q̃_h(·,z)-Q_h(·,z) is asymptotically equicontinuous in probability, i.e., for all ϵ,η>0, there exists δ>0 such that
  
    lim sup_n P(sup _d(y_1, y_2)<δ|{Q̃_h(y_1,z)-Q_h(y_1,z)} - {Q̃_h(y_2,z)-Q_h(y_2,z)}|>ϵ)<η.


To begin with <ref>, recall that 

    w(Z^0_i, z, h)=1/σ_0^2K_h(Z^0_i-z)[1-μ_1^μ_2^-1(Z^0_i-z)]

and 

    w̃(Z^0_i, z, h)=1/σ̃_0^2 K_h(Z^0_i-z)[1-μ̃_1^μ̃_2^-1(Z^0_i-z)],
 
where σ_0^2 = μ_0 - μ_1^μ_2^-1μ_1 and σ̃_0^2 = μ̃_0 - μ̃_1^μ̃_2^-1μ̃_1. Then, one observes that

    Q̃_h(y,z)-Q_h(y,z) =    1/n∑_i=1^n{w̃(Z^0_i, z, h)-w(Z^0_i, z, h)} d^2(Y_i, y) + 
       1/n∑_i=1^n[w(Z^0_i, z, h) d^2(Y_i, y)-E{w(Z^0_i, z, h) d^2(Y_i, y)}].

Observe that w̃(Z^0_i, z, h)-w(Z^0_i, z, h)=W_0 n K_h(Z^0_i-z)+K_h(Z^0_i-z)W_1 n(Z^0_i-z), where

    W_0 n=1/σ̃_0^2-1/σ_0^2,    W_1 n=μ̃_1^μ̃_2^-1/σ̃_0^2-μ_1^μ_2^-1/σ_0^2.

Using the results of Lemma <ref> and Sherman-Morrison formula, it follows that 

    μ̃_2^-1μ̃_1 - μ_2^-1μ_1 
    
            =    [μ_2+O_p{(h^4-r n^-1)^1/211^}]^-1[μ_1+O_p{(h^2-r n^-1)^1/21}] - μ_2^-1μ_1 
    
            =    [μ_2^-1 - μ_2^-1O_p{(h^4-r n^-1)^1/211^}μ_2^-1/1 + 1^μ_2^-11]
            [μ_1+O_p{(h^2-r n^-1)^1/21}]- μ_2^-1μ_1 
     
            =    O_p{(nh^2+r)^-1 / 21}.

Furthermore, it is easy to get σ̃^2_0 - σ_0^2=O_p{(n h^r)^-1 / 2}. Therefore, W_0 n=O_p{(n h^r)^-1 / 2} and W_1 n=O_p{(nh^2+r)^-1 / 21}. Since for j =0, 1, we have

    E{K_h(Z^0_i-z)(Z_il-z_l)^j d^2(Y_i, y)}=O(h^j), 
        E{K_h^2(Z^0_i-z)(Z_il-z_l)^2 j d^4(Y_i, y)}=O(h^2 j-r),


it follows that the first term in (<ref>) is O_p((n h^r)^-1 / 2). We find that E{w^2(Z^0_i,z, h)}=O(h^-r), so the second term in (<ref>) is O_p{(n h^r)^-1 / 2} as well. Then, we show that Q̃_h(y,z)-Q_h(y,z)=o_p(1) for any y ∈Ω and any z∈ℝ^r, since n h^r →∞. 

Moving on to <ref>, for any y_1,y_2∈Ω,

    |{Q̃_h(y_1,z)-Q_h(y_1,z)} - {Q̃_h(y_2,z)-Q_h(y_2,z)}|
    ≤    |Q̃_h(y_1,z)-Q̃_h(y_2,z)| + |Q_h(y_1,z)-Q_h(y_2,z)|
    ≤   1/n∑_i=1^n|w̃(Z^0_i, z, h)||d(Y_i, y_1)-d(Y_i, y_2)||d(Y_i, y_1)+d(Y_i, y_2)| 
        + |E[w(Z^0_i, z, h){d(Y_i, y_1)+d(Y_i, y_2)}{d(Y_i, y_1)-d(Y_i, y_2)}]|
    ≤    2 diam(Ω) d(y_1,y_2) 1/n∑_i=1^n {|w̃(Z^0_i, z, h)| + E|w(Z^0_i, z, h)|}
    
    =    O_p{d(y_1,y_2)},

since E{|w(Z^0_i, z, h)|}=O(1), E{w^2(Z^0_i, z, h)}=O(h^-1), and n^-1∑_i=1^n|w̃(Z^0_i, z, h)|=O_p(1). Therefore,

    sup_d(y_1,y_2)<δ|{Q̃_h(y_1,z)-Q_h(y_1,z)} - {Q̃_h(y_2,z)-Q_h(y_2,z)}| = O_p(δ),

which implies <ref>. These show that d{ṽ_h(z),v_h(z)} = o_p(1).



Firstly, we prove (<ref>). Similar to the proof of Theorem 3 in <cit.>, we can show 

    d F_Y|Z^0(z,y)/d F_Y(y) = f_Z^0|Y(z,y)/f_Z^0(z)

for all z given that f_Z^0(z)>0. Then by Lemma <ref>,

    ∫ w(x,z,h)d F_Z^0|Y(x|y)    = τ_0(y) -μ_1^μ_2^-1τ_1(y)/μ_0 -μ_1^μ_2^-1μ_1 = f_Z^0|Y(z,y)/f_Z^0(z) + O(h^2)

where the error term is uniform over y∈Ω. Hence,

    Q_h(ω,z)     = ∫ d^2(y,ω)w(x,z,h)dF_Z^0,Y(x,y) = ∫ w(x,z,h)dF_Z^0|Y(x,y) d^2(y, ω) dF_Y(y)
       = ∫f_Z^0|Y(z,y)/f_Z^0(z)  d^2(y, ω) dF_Y(y) + O(h^2) = ∫ d^2(y, ω) d F_Y|Z^0(z,y) + O(h^2)
       = Q(ω, z) + O(h^2),

where the error term is now uniform over ω∈Ω. By the first two parts of Assumption <ref> and employing proof by contradiction, we then have d{v_h(z), v(z)}=o(1) as h=h_n→ 0.

Next, define r_h=h^-γ_1/γ_1-1 and set 

    S_j,h(z)={y: 2^j-1 < r_h d{y, v(z)}^γ_1/2≤ 2^j}.

Let I denote the indicator function. Then, for any M>0, there exists a>0 such that, for large n,

    I[r_h d{v_h(z), v(z)}^γ_1/2>2^M]     = ∑_j ≥ M I{v_h(z)∈ S_j,h(z)}
       ≤∑_j ≥ M I(sup_y∈ S_j,h(z) [Q_h{v(z),z} - Q_h(y,z)] ≥ 0).

By the first two parts of Assumption <ref>, for every j involved in the sum, we have, for every y∈ S_j,h(z),

    Q{v(z),z} - Q(y,z) ≲ -d{v(z),y}^γ_1≲ -2^2(j-1)/r_h^2.

Define V(y,z) = Q_h(y,z) - Q(y,z) and we have

    sup_y∈ S_j,h(z)| V(y,z) - V{v(z),z}| 
    ≥   sup_y∈ S_j,h(z)[ Q_h{v(z),z} - Q_h(y,z)] -sup_y∈ S_j,h(z)[ Q{v(z),z} - Q(y,z)].


Hence,

    I[r_h d{v_h(z), v(z)}^γ_1/2>2^M]    ≤∑_j ≥ M I[sup_y∈ S_j,h(z) |V(y,z) - V{v(z),z}| ≥2^2(j-1)/r_h^2]
       ≤∑_j ≥ M I{a h^2 (2^j/r_h)^2/γ_1≥2^2(j-1)/r_h^2}
       ≤∑_j ≥ M I{4a h^2 (2^2j/γ_1-2j/r_h^2/γ_1-2)≥ 1}
       ≤ 4a ∑_j ≥ M2^2 j(1-γ_1) / γ_1/r_h^2(1-γ_1) / γ_1 h^-2≤ 4a ∑_j ≥ M(1/4^(γ_1-1) / γ_1)^j,

which converges since γ_1>1. Thus, for some M>0, we have

    d{v_h(z), v(z)}≤ 2^2M/γ_1  h^2/(γ_1-1)

for large n and hence

    d{v_h(z), v(z)}=O(h^2/(γ_1-1))

for large n, so we showed (<ref>).

Secondly, we prove (<ref>). Define T_n(y,z)=Q̃_h(y,z)-Q_h(y, z). Letting

    D_i(y, z)=d^2(Y_i, y)-d^2{Y_i, v_h(z)},

we have

    |T_n(y,z)-T_n{v_h(z),z}| ≤   |1/n∑_i=1^n{w̃(Z^0_i, z, h)-w(Z^0_i, z, h)} D_i(y, z)|  
        +|1/n∑_i=1^n[w(Z^0_i, z, h) D_i(y, z)-E{w(Z^0_i, z, h) D_i(y, z)}]|.


Since W_0 n and W_1 n from (<ref>) are O_p{(n h^r)^-1/2} and O_p{(n h^2+r)^-1 / 21}, respectively, and using the fact that |D_i(y, z)| ≤ 2 diam(Ω) d(y, v_h(z)), the first term on the right-hand side of (<ref>) is O_p[(n h^r)^-1/2 d{y, v_h(z)}], where the O_p term is independent of Ω and v_h(z). Thus, we can define

    B_R={sup _d(y, v_h(z))<δ|1/n∑_i=1^n{w̃(Z^0_i, z, h)-w(Z^0_i, z, h)} D_i(y, z)| ≤ R δ(n h^r)^-1 / 2}

for R>0, so that P(B_R^c) → 0.

Next, to control the second term on the right-hand side of (<ref>), define the functions g_y: ℝ^r ×Ω↦ℝ by

    g_y(x, w)=1/σ_0^2 K_h(x-z){1-μ_1^μ_2^-1(x-z)} d^2(y, w),

and the corresponding function class

    ℱ_n δ={g_y-g_v_h(z): d{y, v_h(z)}<δ}.

An envelope function for ℱ_n δ is

    F_n δ(z)=2 diam(Ω) δ/σ_0^2 K_h(z-x)|1-μ_1^μ_2^-1(z-x)|,

and E{F_n δ^2(Z^0)}=O(δ^2 h^-r). According to Theorem 2.7.17 of <cit.> and Assumption <ref>, we know the bracketing integral of the class ℱ_nδ

    J_[](ϵ, ℱ_nδ,·_Z^0,Y,2)=∫_0^ϵ√(1+log N_[](t{E(F_nδ^2)}^1/2, ℱ_nδ,·_Z^0,Y,2)) d t

is O(1), where N_[](t{E(F_nδ^2)}^1/2, ℱ_nδ,·_Z^0,Y,2) denotes the minimum number of t{E(F_nδ^2)}^1/2-brackets needed to cover the function class ℱ_nδ (see Definition 2.1.6 of <cit.>) and ·_Z^0,Y,2 denotes the norm that for any f∈ℱ_nδ, f_Z^0,Y,2=[E{f(Z^0,Y)^2}]^1/2. Using these facts together with Theorems 2.14.16 of <cit.> and Assumption <ref>, for small δ,

    E(sup _d{y, v_h(z)}<δ|1/n∑_i=1^n w(Z^0_i, z, h) D_i(y, z)-E{w(Z^0_i, z, h) D_i(y, z)}|)
       ≤J_[](ϵ, ℱ_nδ,·_Z^0,Y,2)  [E{F_n δ^2(Z^0)}]^1/2/√(n)
       =O(δ(n h^r)^-1 / 2).

Combining this with (<ref>) and the definition of B_R,

    E(I_B_Rsup _d{y, v_h(z)}<δ|T_n(y,z)-T_n{v_h(z),z}|) ≤a δ/(n h^r)^1 / 2,

where I_B_R is the indicator function for the set B_R and a is a constant depending on R and the entropy integral in Assumption <ref>.

To finish, set r_n=(n h^r)^γ_2/4(γ_2-1) and define

    S_j, n(z)={y: 2^j-1< r_n d{y, v_h(z)}^γ_2/2≤ 2^j} .

Choose η_2 satisfying Assumption <ref> (ii) and such that Assumption <ref> is satisfied for any δ<η_2. Set η=(η_2 / 2)^γ_2 / 2. Denote events A_η_2 = {d{ṽ_h(z),v_h(z)}>η_2/2} and C_M = {r_n d{ṽ_h(z),v_h(z)}^γ_2/2>2^M)}. If r_n d{ṽ_h(z),v_h(z)}^γ_2/2 is larger than 2^M for a given integer M, then ṽ_h(z) is in one of the shells S_j,n(z) with j>M such that the supremum of the map y↦Q̃_h{v_h(z),z} - Q̃_h(y,z) over this shell is nonnegative by the property of ṽ_h(z).

    P(C_M) ≤    P(B_R^c) + P(A_η_2) + P(B_R∩ A_η_2^c∩ C_M)
    ≤    P(B_R^c) + P(A_η_2)  + ∑_j≥ M
     2^j≤ r_n ηP({sup_y∈ S_j,n(z)| Q̃_h{v_h(z), z} - Q̃_h(y, z)≥ 0 |}∩ B_R),

where the first term goes to zero clearly and the second term goes to zero by Lemma <ref>. Choose η_2>0 small enough that Assumption <ref> (ii) holds for every d{y,v_h(z)}≤η and (<ref>) holds for every δ≤η. Then for every j involved in the sum, we have, for every y∈ S_j,n(z),

    Q_h(v_h(z),z) - Q_h(y,z)≲ - d{v_h(z),y}^γ_2≲ - 2^2(j-1)/r_n^2.

In terms of T_n(y,z) = Q̃_h(y,z) - Q_h(y,z), by Markov's inequality, we have

    P(C_M) ≤    P(B_R^c) + P(A_η_2) 
        + ∑_j≥ M
     2^j≤ r_n ηP({sup_y∈ S_j,n| T_n(y,z) - T_n{v_h(z),z}|≳2^2(j-1)/r_n^2}∩ B_R)
    ≲   ∑_j≥ M
     2^j≤ r_n η(2^j/r_n)^2/γ_2 (nh^r)^-1/2/2^2j/r_n^2
    ≲   ∑_j≥ M2^2j(1-γ_2)/γ_2/r_n^2(1-γ_2)/γ_2√(nh^r)
    
         =    ∑_j≥ M (1/4^(γ_2-1)/γ_2)^j,

which converges since γ_2>1. Hence, as M→∞,

    P(d{ṽ_h(z),v_h(z)}>(2^M/r_n)^2/γ_2)→ 0.

Therefore,

    d{ṽ_h(z),v_h(z)} = O_p{(n h^r)^-1/{2(γ_2-1)}}.





 §.§ Proof of Proposition <ref>















































Suppose <ref> holds. If Ẑ_i-Z^0_i=O_p(ζ_n) for i=1, …, n, Ẑ-Z^0=O_p(ζ_n), nh^r→∞, and h^-r-2ζ_n→0, then the weight functions as defined in (<ref>) and (<ref>) satisfy

    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-r-1ζ_n).



Recall that 

    ŵ(Ẑ_i, Ẑ, h)=1/σ̂^2K_h(Ẑ_i-Ẑ)[1-μ̂_1^μ̂_2^-1(Ẑ_i-Ẑ)],

where 

    σ̂^2=μ̂_0-μ̂_1^μ̂_2^-1μ̂_1,  μ̂_j=1/n∑_i=1^nK_h(Ẑ_i-Ẑ)(Ẑ_i-Ẑ)^⊕ j, j=0, 1, 2

and 

    w̃(Z_i^0, Z^0, h)=1/σ̃^2K_h(Z^0_i-Z^0)[1-μ̃_1^μ̃_2^-1(Z^0_i-Z^0)],

where 

    σ̃^2=μ̃_0-μ̃_1^μ̃_2^-1μ̃_1,  μ̃_j=1/n∑_i=1^nK_h(Z^0_i-Z^0)(Z^0_i-Z^0)^⊕ j, j=0, 1, 2

with K_h(z)=h^-rK{H^-1z}. 

Define

    ρ_j(x, z)=K_h(x-z)(x-z)^⊕ j,  x, z∈𝒯^r.

By Assumption <ref>, K(·) is Lipschitz continuous. For any x_1, x_2, z_1, z_2∈𝒯^r, one has

    |ρ_0(x_1, z_1)-ρ_0(x_2, z_2)|   =|K_h(x_1-z_1)-K_h(x_2-z_2)|
       ≲ h^-r-1x_1-z_1-(x_2-z_2)
       ≤ h^-r-1(x_1-x_2+z_1-z_2).

For j=1, 2, it hold that

    ρ_j(x_1, z_1)-ρ_j(x_2, z_2)   =K_h(x_1-z_1)(x_1-z_1)^⊕ j-K_h(x_2-z_2)(x_2-z_2)^⊕ j
       ≤K_h(x_1-z_1)(x_1-z_1)^⊕ j-K_h(x_2-z_2)(x_1-z_1)^⊕ j+
       K_h(x_2-z_2)(x_1-z_1)^⊕ j-K_h(x_2-z_2)(x_2-z_2)^⊕ j
       ≤(x_1-z_1)^⊕ j·|K_h(x_1-z_1)-K_h(x_2-z_2)|+
       K_h(x_2-z_2)·(x_1-z_1)^⊕ j-(x_2-z_2)^⊕ j.

Using the Lipschitz continuity of K(·) by Assumption <ref> and the fact that K_h(x-z)=0 if any component of |x-z| exceeds h, for j=1, 2 one has

    ρ_j(x_1, z_1)-ρ_j(x_2, z_2)   ≲ h^j-r-1x_1-z_1-(x_2-z_2)+
       h^j-1x_1-z_1-(x_2-z_2)
       ≲ h^-r-1+jx_1-z_1-(x_2-z_2)
       ≤ h^-r-1+j(x_1-x_2+z_1-z_2).

One thus has

    μ̂_j-μ̃_j   =1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-1/n∑_i=1^nρ_j(Z^0_i, Z^0)
       ≤1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-ρ_j(Z^0_i, Z^0)
       ≲1/n∑_i=1^nh^-r-1+j(Ẑ_i-Z^0_i+Ẑ-Z^0).

We conclude that μ̂_j-μ̃_j=O_p(h^-r-1+jζ_n) for j=0, 1, 2. Since nh^r→∞, by Lemma <ref> one has μ̃_0=O_p(1), μ̃_1=O_p(h^2), and μ̃_2=O_p(h^2). It follows that 

    |σ̂^2-σ̃^2|   =|(μ̂_0-μ̂_1^μ̂_2^-1μ̂_1)-(μ̃_0-μ̃_1^μ̃_2^-1μ̃_1)|
       =O_p(h^-r-1ζ_n)

and 

    μ̂_1^μ̂_2^-1/σ̂^2-μ̃_1^μ̃_2^-1/σ̃^2=O_p(h^-r-2ζ_n),

Observe that 

    ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)   ={σ̂^-2-σ̃^-2}ρ_0(Z^0_i, Z^0)+σ̂^-2{ρ_0(Ẑ_i, Ẑ)-ρ_0(Z^0_i, Z^0)}-
       {μ̂_1^μ̂_2^-1/σ̂^2-μ̃_1^μ̃_2^-1/σ̃^2}ρ_1(Z^0_i, Z^0)-
       μ̂_1^μ̂_2^-1/σ̂^2{ρ_1(Ẑ_i, Ẑ)-ρ_1(Z^0_i, Z^0)}

and E{ρ_j(Z^0_i, Z^0)}=O(h^j1^⊕ j), E[{ρ_j(Z^0_i, Z^0)}^⊕ 2]=O(h^2j-r1^⊕ 2j) for j=0, 1. 
We have

    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-r-1ζ_n).













    
    Suppose <ref> and the last two parts of <ref>. If Ẑ_i-Z^0_i=O_p(ζ_n) for i=1, …, n, Ẑ-Z^0=O_p(ζ_n), nh^r→∞, and h^-r-2ζ_n→0, then the following holds,
    
    d{v̂_h(Ẑ), ṽ_h(Z^0)}=o_p(1).







For any y∈Ω, observe that

    Q̂_h(y, Ẑ)-Q̃_h(y, Z^0)   =1/n∑_i=1^nŵ(Ẑ_i, Ẑ, h)d^2(Y_i, y)-1/n∑_i=1^nw̃(Z_i^0, Z^0, h)d^2(Y_i, y)
       =1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}d^2(Y_i, y)
       =O_p(h^-r-1ζ_n)

by Lemma <ref> and the fact that Ω is totally bounded. By the last two parts of Assumption <ref> and employing proof by contradiction, one concludes that d{v̂_h(Ẑ), ṽ_h(Z^0)}=o_p(1).






























Define V_n(y)=Q̂_h(y, Ẑ)-Q̃_h(y, Z^0). Letting D_i(y, Z^0)=d^2(Y_i, y)-d^2{Y_i, ṽ_h(Z^0)}, we have

    |V_n(y)-V_n{ṽ_h(Z^0)}|=|1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}D_i(y, Z^0)|.

By Lemma <ref>, and using the fact that |D_i(y, Z^0)|≤2diam(Ω)d{y, ṽ_h(Z^0)}, the right-hand side of (<ref>) is O_p[d{y, ṽ_h(Z^0)} h^-r-1ζ_n]. Thus, we can define

    B_R={sup_d{y, ṽ_h(Z^0)}<δ|V_n(y)-V_n{ṽ_h(Z^0)}|≤ Rδ h^-r-1ζ_n}

for R>0, so that P(B_R^C)→0.

Set t_n=(h^-r-1ζ_n)^-γ_3/2(γ_3-1). For each n, the metric space Ω (minus the point ṽ_h(Z^0)) can be partitioned into the “shells”

    S_j, n(Z^0)={y: 2^j-1<t_nd{y, ṽ_h(Z^0)}^γ_3/2≤2^j},

with j varying over the integers. Choose η=(η_3/2)^γ_3/2>0 small enough that Assumption <ref> (iii) holds and δ<η_3. If t_nd{v̂_h(Ẑ), ṽ_h(Z^0)}^γ_3/2 is larger than 2^M for a given integer M, then v̂_h(Ẑ) is in one of the shells S_j, n(Z^0) with j>M. In that case the supremum of the map y↦Q̂_h(ṽ_h(Z^0), Ẑ)-Q̂_h(y, Ẑ) over this shell is nonnegative by the optimality of v̂_h(Ẑ). Conclude that, for every η>0,

    P(t_nd{v̂_h(Ẑ), ṽ_h(Z^0)}^γ_3/2>2^M)
    ≤   P(B_R^C)+P(2d{v̂_h(Ẑ), ṽ_h(Z^0)}>η_3)
        + ∑_j>M
    2^j≤η t_nP({sup_y∈ S_j, n(Z^0)[Q̂_h{ṽ_h(Z^0), Ẑ}-Q̂_h(y, Ẑ)]≥0}∩ B_R),

where P(B_R^C)→0 as discussed previously and the second term goes to zero by Lemma <ref>. Then for every j involved in the sum, by Assumption <ref> (iii) we have, for every y∈ S_j, n(Z^0),

    Q̃_h{ṽ_h(Z^0), Z^0}-Q̃_h(y, Z^0)≲-2^2(j-1)/t_n^2.

In terms of the centered process V_n, by Markov's inequality the right-hand side of (<ref>) may be bounded by

    ∑_j>M
    2^j≤η t_nP({sup_y∈ S_j, n(Z^0)[V_n(y)-V_n{ṽ_h(Z^0)}]≳2^2(j-1)/t_n^2}∩ B_R)
       ≲∑_j>Mt_n^2/2^2j(2^j/t_n)^2/γ_3 h^-r-1ζ_n
       =∑_j>M(1/4^(γ_3-1)/γ_3)^j,

which converges to zero for every M=M_n→∞ since γ_3>1. Hence

    d{v̂_h(Ẑ), ṽ_h(Z^0)}=O_p(t_n^-2/γ_3)=O_p{(h^-r-1ζ_n)^1/(γ_3-1)}.



    P(inf_d{ṽ_h(Z^0), y}≥η_3[Q̃_h(y, Z^0)-Q̃_h{ṽ_h(Z^0), z}-C_3η_3^γ_3]≥0)→0.

For any y∈Ω, observe that

    |Q̂_h(y, Ẑ)-Q̃_h(y, Z^0)|   =|1/n∑_i=1^nŵ(Ẑ_i, Ẑ, h)d^2(Y_i, y)-1/n∑_i=1^nw̃(Z_i^0, Z^0, h)d^2(Y_i, y)|
       =|1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}d^2(Y_i, y)|
       =O_p(h^-r-1ξ_n)

by Lemma <ref> and the fact that Ω is totally bounded. It follows that

    Q̃_h{v̂_h(Ẑ), z}-Q̃_h{ṽ_h(Z^0), z}   =Q̃_h{v̂_h(Ẑ), z}-Q̂_h{v̂_h(Ẑ), z}+Q̂_h{v̂_h(Ẑ), z}-Q̃_h{ṽ_h(Z^0), z}
       ≤Q̃_h{v̂_h(Ẑ), z}-Q̂_h{v̂_h(Ẑ), z}+Q̂_h{ṽ_h(Z^0), z}-Q̃_h{ṽ_h(Z^0), z}
       =O_p(h^-r-1ξ_n).

where the inequality follows from the optimality of v̂_h(Ẑ), i.e., Q̂_h{ṽ_h(Z^0), z}≤Q̂_h{v̂_h(Ẑ), z}. Now Assumption <ref> indicates that

    d{v̂_h(Ẑ), ṽ_h(Z^0)}=O_p{(h^-r-1ξ_n)^1/γ_3},

the result follows.




 §.§ Proof of Theorem <ref>


Combining Theorems, <ref> and Proposition <ref>, and <ref>, it follows from the triangle inequality that

    d{m̂(X), m(X)}    = d{v̂_h(Ẑ), v(Z^0)}
       ≤ d{ṽ_h(Z^0), v(Z^0)}+d{v̂_h(Ẑ), ṽ_h(Z^0)}
        = O_p{h^2/(γ_1-1)+(nh^r)^-1/{2(γ_2-1)}+(h^-r-1ζ_n)^1/(γ_3-1)},

where Z^0 = g_0(X) and Ẑ = ĝ(X).






 §.§ Additional Proof

If we have the supremum rate, E(sup_i=1,…,nẐ_i-Z^0_i)=O(τ_n), we can substitute Lemma <ref> with Lemma <ref> below. This substitution would lead to an improvement in the convergence rate outlined in Proposition <ref>, resulting in (h^-1τ_n)^1/(γ_3-1). Furthermore, an enhanced convergence rate can be achieved in Theorem <ref>, expressed as h^2/(γ_1-1)+(nh^r)^-1/2(γ_2-1)+(h^-1τ_n)^1/(γ_3-1).



Suppose <ref> and <ref> hold. If E(sup_i=1,…,nẐ_i-Z^0_i)=O(τ_n), nh^r→∞ and h^-r-1τ_n→0, then the weight functions as defined in (<ref>) and (<ref>) satisfy


    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-1τ_n).
















Recall that in Lemma <ref>, for any x_1, x_2, z_1, z_2∈𝒯^r, one has






    ρ_j(x_1, z_1)-ρ_j(x_2, z_2)≲ h^-r-1+j(x_1-x_2+z_1-z_2).

One thus has

    μ̂_j-μ̃_j   =1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-1/n∑_i=1^nρ_j(Z^0_i, Z^0)
       ≤1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-ρ_j(Z^0_i, Z^0)·[𝐈{ρ_j(Ẑ_i, Ẑ) 0} + 𝐈{ρ_j(Z^0_i, Z^0) 0}] 
       ≤1/n h^r+1-j ( sup_i=1,…,nẐ_i-Z_i^0 + Ẑ-Z^0)
       ∑_i=1^n[𝐈(Ẑ_i-Ẑ≤ h) + 𝐈(Z_i^0-Z^0≤ h)]
       ≲τ_n/n h^r+1-j∑_i=1^n [𝐈(Ẑ_i-Ẑ≤ h) + 𝐈(Z_i^0-Z^0≤ h) ].

Given Z^0, observe 

    1/nh^r∑_i=1^n 𝐈(Z_i^0-Z^0≤ h)P→ c_1 f_Z^0(Z^0)

for some constant c_1<∞, and that

    1/nh^r∑_i=1^n𝐈(Ẑ_i-Ẑ≤ h)
    ≤   1/nh^r∑_i=1^n𝐈(Z^0_i-Z^0≤ 3h) + 𝐈(Ẑ_i-Z_i^0≥ h) + 𝐈(Ẑ -Z^0≥ h)

converges to c_2 f(Z_i^0) in probability for some constant c_2<∞. Here, 

    1/n h^r∑_i=1^n𝐈(Ẑ_i-Z_i^0≥ h) and 1/nh^r∑_i=1^n𝐈(Ẑ - Z^0≥ h)
 are both o_p(1) since by the Markov inequality and Theorem <ref>,

    E{1/n h^r∑_i=1^n𝐈(Ẑ_i-Z_i^0≥ h)}≤1/n h^r+1∑_i=1^n EẐ_i-Z_i^0≤1/n∑_i=1^n h^-r-1τ_n → 0,
       E{1/n h^r∑_i=1^n𝐈(Ẑ - Z^0≥ h)}≤1/n h^r+1∑_i=1^n EẐ - Z^0≤1/n∑_i=1^n h^-r-1τ_n → 0.

This implies that μ̂_j-μ̃_j=O_p(h^j-1τ_n) for j=0, 1, 2. It follows that 

    |σ̂^2-σ̃^2|=O_p(h^-1τ_n)   and  μ̂_1^μ̂_2^-1/σ̂^2-μ̃_1^μ̃_2^-1/σ̃^2=O_p(h^-2τ_n).

Thus, due to E{ρ_j(Z^0_i, Z^0)}=O(h^j1^⊕ j) and E[{ρ_j(Z^0_i, Z^0)}^⊕ 2]=O(h^2j-r1^⊕ 2j) for j=0, 1, we have

    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-1τ_n).














  * Framework:

    ℝ^pℝ^r[ISOMAP: ψ]LFR: ϕΩ


  * Data: {(X_i, Y_i)}_i=1^n, where X_i∈ℝ^p, Y_i∈Ω.


  * Proof outlines:

    
  * Deep learning: <cit.>.
    
  * Manifold learning: <cit.> and possibly Alvaro's manuscript.
    
  * Errors-in-variables: examine the difference in the weights; see Appendix F of <cit.>, and D.2, D.3 of <cit.>.
    
  * Multivariate local regression: <cit.>, <cit.>, and <cit.>.


  * Difficulties:

    
  * ISOMAP manifold coordinates are not directly observed: α_n = sup_1≤ i≤ nψ̂(Y_i)-ψ(Y_i). Any influence on the asymptotic property of DNN?
    
  * i.i.d. assumption is in fact not satisfied for DNN. Ẑ_i = ψ(Y_i) is not i.i.d.
    
    Ẑ_i = g_0(X_i) + ϵ_i

    
  * error term, ϵ_i, need to be i.i.d. standard normal.
    
  * The theory in <cit.> is only applicable for scalar responses. Need to think about the extension to vector responses. Use sequential learning.
    
  * Uniform rate of convergence for local regression seems not directly extendable to multivariate cases, as the rate in <cit.> is suboptimal compared to usual local linear regression.






 §.§ Additional Simulations for Networks


We consider graph Laplacians of undirected weighted networks with a fixed number of nodes m and bounded edge weights as a second type of metric-space valued responses; see Example <ref>. The space of graph Laplacians <cit.> is

    Ω={Y=(y_i j): Y=Y^T ; Y 1_m=0_m ;-W ≤ y_i j≤ 0  for i ≠ j},

where 1_m and 0_m are the m-vectors of ones and zeroes, respectively. We aim to construct a generative model that produces random graph Laplacians Y along with an Euclidean predictor X∈ℝ^9. 

Denote the half vectorization excluding the diagonal of a symmetric and centered matrix by vech, with inverse operation vech^-1. By the symmetry and centrality of graph Laplacians as per (<ref>), every graph Laplacian Y is fully determined by its upper (or lower) triangular part, which can be further vectorized into vech(Y), a vector of length d=m(m-1) / 2. The true regression function can thus be defined as

    m(X)=vech^-1[{-E(l_1|X), …, -E(l_d|X)}^].


To generate random response Y, each entry of the random vector (l_1, …, l_d)^ is first generated using beta distributions whose parameters depend on the predictor X. The random response Y is then generated conditional on X through an inverse half vectorization vech^-1 applied to (-l_1, …,-l_d)^. The space of graph Laplacians Ω is not a vector space. Instead, it is a bounded, closed, and convex subset in ℝ^m^2 of dimension m(m-1)/2. To ensure that the random response Y generated in simulations resides in L_m, the off-diagonal entries l_j, j = 1, …, d, need to be nonpositive and bounded below as per (<ref>). To this end, we consider

    Y=vech^-1(-l_1, …,-l_d),  where  l_j i.i.d.∼Beta(α_1, α_2),
       α_1 = X_8sin(π X_1) +(1-X_8)cos(π X_2),
       α_2 = X_4^2 X_7+X_5^2(1-X_7),

where components of the predictor X∈ℝ^9 are distributed as

    X_1 ∼ U(0,1), X_2 ∼ U(-1/2,1/2), X_3 ∼ U(1,2)
       X_4 ∼ N(0,1), X_5 ∼ N(0,1), X_6 ∼ N(5,5)
       X_7 ∼Ber(0.4), X_8 ∼Ber(0.3), X_9 ∼Ber(0.6).

The true regression function is thus

    m(X)=vech^-1(-α_1/(α_1 + α_2), …, -α_1/(α_1 + α_2)).




Similar to simulations for the distributional data, we investigated sample sizes n = 100, 200, 500, 1000, with Q=500 Monte Carlo runs and set r = 2 as the dimension of low-dimensional representations of random objects. We compare the proposed model with global regression (GFR) and sufficient dimension reduction (SDR). Figure <ref> displays the MSPE for various sample sizes across all Monte Carlo runs, while Table <ref> summarizes their averages. We observe that MSPE decreases with increasing sample size, highlighting the convergence of the proposed methods. Both the boxplot and the table illustrate the superior performance of the proposed model over GFR and SDR, even with small sample sizes. Despite the simulated network data not residing on a 2-dimensional manifold,

the proposed model exhibits remarkable robustness as the sample size increases. Indeed, as the sample size grows, the proposed model proves to be a superior method for handling multivariate predictors and metric-space valued responses.





 §.§ Data Application for Human Mortality Data


We illustrate the proposed method using the age-at-death distributions of 162 countries in 2015, obtained from United Nations databases (<http://data.un.org/>) and UN World Population Prospects 2022 Databases (<https://population.un.org/wpp/Download>). For each country and age, the life table contains the number of deaths aggregated every five years, which can be treated as histograms of the number of deaths at age, with bin widths equal to 5 years. We use the  package <cit.> to create the density by local linear smoother on the histograms and finally standardize the function by trapezoid integration. Mortality distribution in each country is influenced by various factors, such as economics, healthcare systems, and social and environmental factors. To investigate these potential factors, we consider nine predictors, as used in <cit.>, including demographic, economic, and environmental factors in 2015; see Table <ref>.




The MSPE was calculated through leave-one-out cross-validation. From Table <ref>, the proposed method achieves the most accurate prediction results, with a 16% and 3% improvement compared to GFR and SDR, respectively. These results demonstrate that despite the data not residing on a two-dimensional manifold and with a small sample size, the proposed method is superior and robust. 











Additionally, our proposed method also allows for the investigation of the effects of predictors. Here, we further investigate the effect of GDP and health expenditure on mortality distributions while holding other predictors constant at their median values. Figure <ref> (left) shows that the fitted age-at-death densities shift towards the right as GDP increases, indicating an increase in life expectancy. Additionally, the probability of death before age 5 declines with increasing GDP, suggesting a lower infant mortality rate. Similarly, as the percentage of health expenditure in GDP increases, the age-at-death densities also shift rightward, indicating improved life expectancy. Conversely, as the percentage of agricultural GVA increases, the age-at-death densities shift leftward, indicating a decrease in life expectancy. This trend is consistent with the observation that as a country develops, GDP increases and the agricultural share of GVA commonly decreases.



























































































































































































































 Paper Title: Potential weights and implicit causal designs in linear regression.





Data-driven Modeling for Grid Edge IBRs: A Digital Twin Perspective of User-Defined Models 





    Kaveri Mahapatra*, Bhaskar Mitra, and Soumya Kundu
Pacific Northwest National Laboratory, United States 



{kaveri.mahapatra, bhaskar.mitra, soumya.kundu}@pnnl.gov































    July 30, 2024. I thank Isaiah Andrews, Ambarish Chattopadhyay, Matthew Gentzkow, 
Guido Imbens, Lihua Lei, Ashesh Rambachan,
David Ritzwoller, Brad Ross, Jonathan Roth, Jesse Shapiro, Jann Spiess, Davide Viviano,
Jose Zubizarreta, and participants at the Harvard graduate student workshop on
econometrics for helpful comments. 
===========================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 

When do linear regressions estimate causal effects in quasi-experiments? This paper
provides a generic diagnostic that assesses whether a given linear regression
specification on a given dataset admits a design-based interpretation. To do so, we define
a notion of potential weights, which encode counterfactual decisions a given
regression makes to unobserved potential outcomes. If the specification does admit such an
interpretation, this diagnostic can find a vector of unit-level treatment assignment
probabilities—which we call an implicit design—under which the regression
estimates a causal effect. This diagnostic also finds the implicit causal effect estimand.
Knowing the implicit design and estimand adds transparency, leads to further sanity
checks, and opens the door to design-based statistical inference. When applied to
regression specifications studied in the causal inference literature, our framework
recovers and extends existing theoretical results. When applied to widely-used
specifications not covered by existing causal inference literature, our framework
generates new theoretical insights.







§ INTRODUCTION


Much of economic research, particularly in applied microeconomics, has put a focus on
research design and causal identification.[By May 2024, over 50% of papers in
the NBER working paper series mentions experimental or quasi-experimental methods
<cit.>.] The key distinguishing feature of this school of
thought—which we will refer to as “design-based”—is an experimentalist focus on
reasoning with plausible randomization of treatment, so as to mimic some randomized
control trial.[We refer to “design-based” as modeling treatment assignment, in
contrast to modeling potential outcomes (which is often called “model-based”). This is
consistent with the terminology adopted in <cit.> and
<cit.>.] In observational studies, researchers often argue that the
treatment is plausibly as-if random, so that causal effects are identified as they are in
randomized experiments. An advantage of this mode of identification is validity without
requiring functional form or homogeneity assumptions on potential outcomes.

Despite the popularity of the design-based perspective in terms of identification, the
same is not true for estimation. Economic research continues to overwhelmingly prefer
linear regression methods to other methods that are arguably more “natively”
design-based <cit.>. These other methdos—including inverse
propensity-score weighting, matching <cit.>,
doubly robust <cit.>, or balancing approaches <cit.>—explicitly use the
probability of treatment assignment to target some explicit treatment effect. In contrast,
linear regressions are historically motivated by treating the linear model
literally as a structural model of potential outcomes (e.g., linear simultaneous equation
models). When read in this way, regression methods usually impose some form of treatment
effect homogeneity—and are thus more natively
model-based. Indeed, the influential article by <cit.> criticizes
regressions in experimental settings precisely because randomization does not imply the
linear model.

To address this tension, a modern perspective views linear regressions as simply
estimators to be analyzed under certain designs—placing restrictions on treatment
assignment but little to no restrictions on potential outcomes. In this view, linear
regression is essentially a machine that compares different groups of units. Under the
right treatment randomization assumption, these comparisons recover causal effects.
Indeed, much has been written on the properties of various linear specifications under
known restrictions on treatment assignment <cit.>.
This line of research analyzes particular regression specifications under a
known set of designs and examines whether the specification makes sensible comparisons
under the design. A limitation is that this literature results in a hodge-podge of
results: Certain specifications are design-based under certain restrictions, but other
configurations of specification and design may produce nonsensical estimands (e.g.,
negatively weighted causal effects).


Complementary to this literature, this paper instead asks whether, given a dataset and a
regression specification, there exists a design—that is, a distribution over treatment
assignment—that rationalizes the specification, without restrictive outcome models.
Suppose we are presented with a regression for which a design-based interpretation is
ascribed: An empirical researcher hopes that this regression estimates a causal effect
under some design—some process of random treatment assignment—without
restrictions on potential outcomes. This paper provides generic tools and diagnostics that
answer the following three questions:

    
  *  First, are the hopes of the researcher obviously misplaced and
    can we reject design-based interpretation altogether?
    
  *  Second, if a design-based interpretation is not obviously
    rejected, can we estimate or infer what the design is when it is not explicitly
    specified?
    
  *  Third, given such a design, what is the estimand that the
    regression specification targets?


By a causal effect, we mean the following: An estimand is a causal contrast if it
satisfies level irrelevance <cit.> under the true treatment
assignment process. That is, shifting every potential outcome of some unit by the same
number should not alter the estimand. This is a minimal requirement for a causal
comparison when we do not impose restrictions on potential outcomes. For instance, if the
treatment is binary, then a quantity satisfies level irrelevance if and only if it is a
function of individual treatment effects.

We answer the questions <ref>–<ref> by connecting the algebra of linear
regression estimators to their behavior under random treatment assignment. Every linear
regression estimator is a weighted average of observed outcomes, where weights
depend on observed treatment status and other covariates. Analogous to potential
outcomes, we define a notion of potential weights for the linear specification. The
potential weight for unit i, evaluated at treatment , is the weight put on i's
potential outcome by the regression under the counterfactual treatment . The
observed weights can then be viewed as the potential weights evaluated at the realized
treatment assignment.

In other words, the potential weights encode  what the linear estimator would have done to
unit i's outcomes, had unit i received different treatment value. Thus, potential
weights encode the precise comparisons regression estimators make under treatment
randomization. These potential weights are readily computable in practice, and estimators
for them are consistent under no stronger assumptions than typically assumed for the
consistency of the OLS estimator. We show that, given the potential weights, the answers
to all three questions are immediate.

If a regression estimand satisfies level irrelevance, then the potential weights for every
unit must be mean zero under the true design. Otherwise, shifting all potential outcomes
by one unit would shift the estimand by the expected value of the potential weights.
Fixing the potential weights, this then yields linear equations that the true design must
satisfy, assuming that the regression estimand is a causal contrast. We call the solution
to these equations implicit designs—vectors that are orthogonal to the potential
weights. In many cases, the implicit designs, if they exist at all, are often unique.

A key observation is that a regression estimand is a causal contrast if and only if the
true design is an implicit design; therefore, computing the set of implicit designs gives
information on the true design, under a presumption of innocence for the regression
specification. This diagnostic then helps answer the three questions
<ref>–<ref>.

For <ref>, we can immediately reject a causal interpretation altogether if no
implicit designs exist, or if the only solutions to the level irrelevance equations are
not valid probability measures. If that happens, then we know that the regression
specification could not have been a causal contrast under the true design, since otherwise
the true design would have solved the equations. For <ref>, when the implicit
design is unique, we immediately have a single plausible candidate for the true design. We
can then inspect whether such a design accords with the researcher's economic intuition,
and whether it is well-calibrated for the observed treatment statuses.[A set of
probability forecasts are said to be well-calibrated if, among the events predicted to
occur with probability p, a proportion of approximately p of them actually occur.]
Finally, for <ref>, given a design, we can compute the estimand under that
design. The estimand is a weighted average of potential outcomes, where the weight put on
unit i's potential outcome under treatment w is simply the product of the
corresponding treatment assignment probability and the potential weight. Thus, for such an
implicit estimand, we can inspect whether it is sensible—e.g., whether it is a
positively weighted average of individual treatment effects, whether it is contaminated by
potential outcomes that should be irrelevant <cit.>, or
whether it is of policy interest.




Our diagnostic provides a useful and universal sanity check for both empirical researchers
and their critics. If a plausible implicit design is found and the corresponding implicit
estimand is sensible, then the regression specification is assured to estimate a
reasonable quantity under some reasonable model of treatment assignment—thereby
dispelling concerns of “negative weights” altogether. When that happens, the regression
can even be called doubly robust: It retains a causal interpretation either when the
treatment is assigned under the implicit design or when the regression correctly specifies
the conditional means of the potential outcomes.

If we fail to find a plausible implicit design, researchers should then reexamine the
interpretation of the regression specification. If the researcher remains confident of a
outcome-model-free, design-based identification strategy, then they should perhaps be more
explicit about the design—including using methods that are more natively design-based.
In that case, implicit designs may be used to guide the model of treatment assignment.
On the other hand, if the researcher insists on the regression
specification, then a purely design-based interpretation is insufficient and the researcher should
justify the corresponding functional form assumptions.[For instance, in a panel
context, one might justify that treatment is not necessarily randomly assigned but mean
potential outcomes satisfy parallel trends.] Moreover, the implicit design also opens the
door to design-based inference  in observational settings where the true design is unknown
<cit.>; design-based inference is useful to quantify uncertainty
especially when a sampling model is difficult to conceptualize or justify.

When applied to some common regression specifications, our framework uncovers new
theoretical insights. In the literature, many specifications have been shown to have
design-based interpretation under certain restrictions on the treatment assignment
process. For many of these specifications, our framework can recover these results and
supply a converse of sorts: We show that the only designs under which these
specifications estimate causal contrasts are exactly those analyzed in
the literature. Concretely, we supply these converses for <cit.>,
<cit.>, and <cit.>. We emphasize that we
obtain all of these results—themselves about rather distinct
settings[<cit.> studies regressions that adjust for covariates in a
binary-treatment cross-sectional setting. <cit.> study similar
specifications with multiple treatments. <cit.> study two-way fixed
effects specifications in panel settings under a design-based framework.]—by simply
mechanically computing the potential weights, implicit design, and implicit estimands,
with little tailored analysis.

Our framework also uncovers novel results for specifications that are less
well-understood. First, it is well-known that <cit.>'s
specification—where the outcome is regressed on an interaction of a binary treatment and
demeaned covariates—recovers the average treatment effect if the covariates are
saturated. We recover a version of this result, but additionally show that the saturation
is essentially necessary in observational settings. Without saturating, even when
the propensity score is linear in the covariates, this regression generally fails to
retain a design-based interpretation—in contrast to the specification without
interaction <cit.>. Second, we show that, in a panel setting with staggered
adoption, two-way fixed effect (TWFE) regressions with time-varying covariates also
generally do not have design-based interpretations, in the sense that the set of implicit
designs is empty. Lastly, with staggered adoption and an unbalanced panel, we show
that TWFE regressions also fail to have design-based interpretation unless the
distribution of treatment timing is invariant across units with different missingness
patterns.

This paper is particularly related to <cit.>, who characterize
and interpret the weights on observed outcomes for several leading linear regression
specifications. Our perspective departs from theirs by treating these weights as the
observed counterpart to some potential weights, much like observed outcomes are potential
outcomes evaluated at the observed treatment.

This paper proceeds as follows. <Ref>
introduces our framework in population, without estimation error. <Ref>
discusses estimation of potential weights and implicit design and establishes their
consistency. <Ref> then applies our framework to a litany of
regression specifications, recovering known results and providing new  ones.
<Ref> concludes with further discussion and practical recommendations.








§ POTENTIAL WEIGHTS AND IMPLICIT DESIGNS



Consider a finite population of units i ∈ [n] ≡1,…, n. Each individual receives one of
J+1 treatments ∈, where || = J+1  < ∞. We denote by _i the
realized treatment of unit i. Each unit is associated with a vector of covariates
_i and J+1 vector-valued potential outcomes of length T, _i () ∈^T : ∈.[For expositional clarity, we assume that the dimension of
the outcome vector is the same across individuals (i.e., balanced panels). Our results in
<ref> do discuss imbalanced panels.] Upon treatment realization, we observe a
realized outcome _i = _i(_i).

We consider a design-based framework in which the potential outcomes and covariates are
treated as fixed,[Here, “design-based” is used in contrast to
“sampling-based.” In the sampling-based framework, uncertainty is assumed to have come
from sampling from an infinite superpopulation, rather than solely from the random
assignment of the treatment to a finite population
<cit.>. This is a distinct meaning
from “design-based” justifications for an estimand; there, “design-based” is in
contrast to “model-based”, referring instead to the fact that causal identification
rests on assumptions about treatment assignment rather than assumptions about functional
forms of potential outcomes.] and the only randomness comes from the random assignment of
_i. To that end, let Π^* denote the joint distribution of (_1,…, _n)
and let ^* denote the associated marginal treatment assignment
probabilities:[We do not a priori assume the analogue of unconfoundedness in this
finite population context (i.e., units with the same covariates have the same π_i^*).
The probabilities π^*_i simply define treatment assignment probabilities conditional
on potential outcomes and covariates, and as a result can accommodate rich forms of
selection. However, as we shall see, our results often imply that many regression
specifications are only justified under designs that satisfy unconfoundedness.] 
    ^*
    = (π^*_1, …, π^*_n)  where π_i^*() = _Π^*(_i = ).
 We
refer to such a vector  of marginal treatment probabilities as a
design,[Following broadly adopted terminology, we also often refer to
π_i^* as a propensity score.] In
particular, to ^* as the true design, and to a joint distribution
Π on _1,…, _n as a joint design.

This setup is sufficiently general to encompass both cross-sectional (T=1) and panel
settings (T > 1) as well as both binary treatment (J+1=2) and multivalued treatment
(J+1 > 2) settings. When we are in a cross-sectional setting (T=1), we shall identify
= 0,…, J and, to simplify notation, we shall not bold the symbols Y_i,
y_i, w, W_i, x_i. If the treatment is binary, then we additionally use π_i to denote
π_i(1), where π_i (0) = 1-π_i. On the other hand, in a panel setting, it is
often natural to identify ⊂0,1^T as the set of treatment paths
<cit.>. Throughout this section, we illustrate with the standard cross-sectional, binary
treatment setting as a running example.

[Cross-sectional setting, binary treatment]
    
     Here, units are associated with potential
    outcomes y_i(1) and y_i(0). We observe treatment assignments W_i, covariates
    x_i, and observed
    outcomes Y_i = y_i(W_i). The covariates and potential outcomes are assumed to be
    fixed. The design ^* = (π_1^*,…, π_n^*) ∈ [0,1]^n describes the
    probabilities that W_i = 1 for every unit i ∈ [n], i.e., π_i^* = (W_i = 1).


Suppose we are presented with a regression specification by an empirical researcher. The
researcher hopes that this specification estimates a causal effect under the true design
^*.  In observational settings, the true design ^* is typically unknown, but
the researcher usually supplies some economic argument that their regression specification
estimates a causal parameter under the unknown design. This paper argues that merely
presuming the regression specification estimates a causal effect is often quite informative of ^* and of the estimand of the regression under ^*.
In particular, it is sufficient to answer questions <ref>–<ref>
posed in the introduction.

As an overview, our approach proceeds in two steps. First, we characterize the
counterfactual behavior of the regression specification under the design ^*. That
is, we examine how the regression makes decisions with respect to each unit's
potential outcomes. We define a notion of potential weights that encode
these decisions, where the potential weight for unit i and treatment  is the weight
that the regression would assign to the potential outcome y_i(), if the
realized treatment _i were equal to .

Second, we consider restrictions on the potential weights that make the regression
estimand admit a causal interpretation. We observe that when we require the regression
estimand to be insensitive to equal level shifts in each unit's potential outcomes, the
expected potential weight under ^* must be zero for every unit. This leads to linear
restrictions on ^* that often allow us to exactly pinpoint ^*.



 §.§ Population regression specification and potential weights

To proceed, we first describe what we mean by a regression specification from a population
perspective. Consider a regression of _it on a covariate transform z_t (_i,
_i) ∈^K.[In a panel setting, this setup accommodates unit-level fixed
effects by defining z_t(_i, _i) such that ∑_t=1^T z_t (_i, _i) = 0.
In particular, for any given covariate transform z̃_t that excludes a unit fixed
effect, we can incorporate the unit fixed effect by defining z_t = z̃_t -
1/T∑_t=1^T z̃_t as the within-transformed version of z̃_t.
Importantly, our subsequent arguments do not accommodate unit fixed effects parametrized
as a unit-level dummy variable in _i. See <ref> for an
explanation.]
Let β denote the K-dimensional coefficient vector associated with the covariate
vector z_t—whose precise population definition is specified next. The empirical researcher hopes that a subvector of dimension k, =
Λβ, has a causal interpretation, for a known matrix Λ∈^k ×
K.

[<ref> continued]
    
    As an example, consider the regression Y_i = α + τ W_i + ϵ, where the
    coefficient of interest is τ. Then the covariate transform is z(x_i, w) = [1,
    w]' and the coefficient contrast corresponds to Λ = [0,1]. For a more general
    regression specification with covariate transform z(x_i, w) with corresponding
    coefficients β, if we are interested in a scalar contrast τ = λ'
    β, then Λ = λ'.


Such a regression has the following population (i.e., expectation over Π^*)
first-order condition 
    1/n∑_i=1^n ∑_t=1^T __i
    ∼π_i^*[z_t (_i, _i)z_t (_i, _i)']_G_n(^*)β =
    1/n∑_i=1^n ∑_t=1^T _ _i ∼π_i^*[z_t(_i,
    _i) y_it(_i)]_H_n(^*),

where we define G_n(^*) as the population Gram matrix
of the regression, and
H_n(^*) as the population covariance between the outcome and the covariate
transform.[Throughout, we assume that the
population Gram matrix G_n(^*) is invertible.] The regression estimand β is then
defined to be β≡ G_n(^*)^-1 H_n(^*), and the coefficients of
interest are in turn defined to be ≡Λ  G_n(^*)^-1 H_n(^*).

Formally, we make the following definition of a population regression specification. We
take the perspective that when we are presented with a regression specification, we are:

    
  * shown how to transform the covariates and treatment values to put on the right-hand side of
    the regression (i.e., z_t(_i, ·)),[It is important that
    counterfactual values of z_t(_i, ) are known, which rules out specifications that
    include, say, lagged outcomes on the right-hand side. See <ref> for a
    discussion.]
    
  * told which
    coefficient contrasts to interpret causally (i.e., Λ), and
    
  * shown the
    population Gram matrix G_n = G_n(^*) of the regression.


The following definition formalizes this, which we illustrate with the running example.

    
    Let G_n = G_n(^*), assumed to be invertible. Let (_i, ·) be a T
    × K matrix whose rows collect z_t(_i, ·)'. A population
    regression specification collects the objects 
    (Λ, G_n, (_1, ·),
        …, (_n, ·)).





[<ref> continued]
    
    Consider regressing Y_i on z(x_i, W_i), where β is the coefficient vector
    corresponding to z(·). We are interested in the target parameter τ =
    λ'β. This describes the following population regression specification: 
    Λ   = λ' 
    
             G_n    = 1/n∑_i=1^n [z(x_i, W_i)z(x_i, W_i)'] 
       = 1/n∑_i=1^n π_i^* z(x_i, 1)z(x_i, 1)' + (1-π_i^*) z(x_i, 0)z(x_i, 0)' 
    (x_i, w)    = z(x_i, w)', w ∈0,1.





It is worth remarking here that our perspective effectively treats the population Gram
matrix G_n as known. This is a modeling device that resembles identification analysis
<cit.>, in which we often assume the population distribution of
the observed data is known and abstract away from estimation.
Despite G_n being a function of ^*, this perspective is reasonable, as G_n, along with H_n ≡
H_n(^*), admits a natural estimator

    Ĝ_n ≡1/n∑_i=1^n ∑_t=1^T z_t(_i, _i) z_t(_i, _i)'
        Ĥ_n ≡1/n∑_i=1^n ∑_t=1^T z_t(_i, _i) _i,

 which we use to replace the unknown G_n in finite samples. It is quite reasonable to
 hope that Ĝ_n ≈ G_n. In fact, we ought to believe Ĝ_n, Ĥ_n are
 close to G_n, H_n if we think the ordinary least-squares (OLS) estimator is a good
 estimator.  Indeed, Ĝ_n, Ĥ_n are directly used to substitute the unknown
 G_n and H_n when we estimate the regerssion via OLS.  An empirical researcher who
 estimates the regression in question via OLS exactly justifies the estimator by hoping
 that Ĝ_n ≈ G_n and Ĥ_n ≈ H_n. More formally, these hopes are
 usually validated, as under mild conditions, we can usually embed the finite population
 in a sequence of populations with growing sizes n, such that Ĝ_n - G_n  0
 and Ĥ_n - H_n  0. Some examples of these conditions are presented in
 <ref> and <ref>.

There is also sometimes reason to think that G_n is directly observable and equal to
Ĝ_n; we discuss this in the following remark.

[Fixed Gram designs]

    Note that for some regression specifications,
there may be joint designs Π^* with
the property that Ĝ_n = G_n almost surely under Π^*. We call such joint designs
fixed Gram designs.
If the treatments are assigned under a fixed Gram design, then the
observed sample Gram matrix is equal to the population Gram matrix, making the latter
known in a more literal sense. OLS estimators in design-based causal inference are
frequently analyzed under fixed Gram designs, as they are unbiased for the estimand when
the Gram matrix is fixed <cit.>.

Fixed Gram designs can arise naturally. For instance, suppose we regress Y_i on a
constant and the binary treatment indicator W_i ∈0,1 as in
<ref>. Then completely randomized experiments are
fixed Gram designs.[Under a completely randomized experiment with N_1 treated units all
nN_1 subsets of size N_1 are equally likely to be the treated units. When
this is the case, the Gram matrix for this regression is equal to 
    Ĝ_n = [       1   N_1/n;   N_1/n N_1 / n ] = G_n.

]
In general, such designs often correspond to
randomly choosing certain permutations of the treatment assignments, and are thus
naturally related to randomized experiments and permutation inference.[More
precisely speaking, given a realized vector of treatments (_1,…, _n), a fixed
Gram design exists if there are a set of permutations σ∈𝒮_n such that (_1,…, _n)
≠ (_σ(1), …, _σ(n)) and that  
    Ĝ_n = Ĝ_n(σ) ≡1/n∑_i=1^n ∑_t=1^T z_t(_i,
    _σ (i))z_t(_i, _σ (i))'.

for any (_1,…, _n).
Any distribution over elements of 𝒮_n then induces a fixed Gram design by
sampling the corresponding permutation of the treatment assignments.]





Given a population regression specification as defined by <ref>, we can now unpack exactly how 
compares different potential outcomes:

    ≡Λ G_n^-1 H_n    = 1/n∑_i=1^n ∑_∈π_i^*() ∑_t=1^T Λ G_n^-1z_t(_i, ) _it()
       = 1/n∑_i=1^n ∑_∈π_i^*() ·Λ G_n(^*)^-1(_i, )'
        __i() ∈^k × T_i() 
       = 1/n∑_i=1^n _∼π_i^* [ _i()_i()].

<Ref> reveals that the vector of coefficients  is a comparison of
potential outcomes _i() under ^* in the sense  that each potential
outcome vector is weighted by the matrix _i(). These weighted potential outcomes are
then aggregated under the design ^* to produce . Motivated by this
observation, we refer to _i() as the potential weights associated with
the regression specification.

    Given a population regression specification, the potential weights for
    individual i are matrices defined by
    _i() ≡Λ G_n^-1(_i, )'.

Since the potential weights are solely functions of the population regression
specification, they are known under our perspective.


Before moving on to define causal restrictions on the potential weights, we summarize and illustrate our
description of the regression specification and the potential weights in the running
example <ref>.
[<ref> continued]
    
    Each unit has two scalar potential weights ρ_i(1) and ρ_i(0), where 
    ρ_i(w) = _i(w) = λ' G_n^-1 z(x_i, w) ∈ ,    w ∈0,1.

    The population regression estimand is decomposed according to (<ref>) as 
    τ = 1/n∑_i=1^n π_i^* ρ_i(1) y_i(1) + (1-π_i^*) ρ_i(0) y_i(0).






 §.§ Causal restrictions, implicit designs, and implicit estimands

Not all regression estimands admit a causal interpretation.
In other words, imposing a causal
interpretation requires that the potential weights satisfy certain properties.
We limit
ourselves to considering causal contrasts, where treatment arms
are compared against each other. A minimal requirement for a causal contrast is
level irrelevance.[We follow the terminology in <cit.>.] That is, if we
added c_it to _it() for all , then we should expect the causal contrast
to be unchanged.



We say that  is a vector of causal contrasts if it satisfies level irrelevance:
That is,  is unchanged if we replaced all potential outcomes _it() with
_it () + c_ it, for any i∈[n], t∈ [T], and c_it∈.


On the one hand, <ref> is a weak requirement for  in the
 sense that it does not rule out certain problematic behaviors. For instance, in a
 cross-sectional, binary-treatment setting, a weighted average treatment effect with
 negative weights[That is, τ = 1/n∑_ i=1^n  ω_i (y_i(1) -
 y_i (0)), where there exists (i,j) such that ω_i < 0 < ω_j] is a causal
 contrast by <ref>. Whether stronger properties hold can be
 assessed under our framework as well, once we use level irrelevance to pinpoint ^*.
 On the other hand, <ref> is a strong requirement because it allows  any perturbation c_it to the potential outcomes. Thus, estimands
 whose validity hinges on a model for the potential outcomes usually fail to satisfy
 <ref>. In this sense, <ref> encodes a
 requirement that causal identification purely comes from design-based assumptions.


This paper finds that assuming  are causal contrasts under  in the sense of
<ref> is often sufficient to characterize ^*. The reason is
that _i() specifies exactly how the regression estimands weight different
potential outcomes
_i() under randomized treatment assignment. For  to satisfy
<ref>, these weights should be on average zero for every
unit—otherwise, shifting _it() by 1 would increase  by [_it(_i)].  Indeed, if  is vector of causal contrasts in the sense of
<ref>,
then by examining (<ref>), we immediately have that for all i, the average
potential weight under ^* is zero: 
    ∑_∈π_i^*() _i() = 0 ,  ∑_∈π_i^*()  = 1.

Viewed as a set of restrictions on ^* with known potential weights, (<ref>) defines a set of kT + 1 linear equations in J+1
variables for π_i^*. We refer to the solutions of these restrictions as implicit designs.



Given the potential weights, we call a vector = (π_1,…, π_n) that
satisfies
(<ref>) for every i an (improper) implicit
design. If each π_i() ≥ 0, then we call  a proper implicit
design. If  is proper and it generates the same Gram matrix G_n, 
    G_n = G_n(^*) = G_n() ≡1/n∑_i=1^n
    ∑_t=1^T __i ∼π_iz_t(_i, _i) z_t(_i, _i)',
 then we say
that  is Gram-consistent.


Implicit designs are solutions to (<ref>), viewed as
linear equations in ^*. They admit an interpretation as a plausible marginal
distribution of treatment if they are proper and Gram-consistent—meaning that they are
probability measures on  and are consistent with the known population Gram matrix
G_n.

We can often expect that implicit designs are unique, if they exist—at least modulo
minor exceptions. This is because the number of equations is at least the number of
unknowns (kT ≥ J) in many common situations. For instance, in a binary treatment,
cross-sectional setting, we have that k = T = J = 1. In a multivalued-treatment setting,
we often have k=J contrasts. In a panel setting with staggered adoption, we usually have
fewer unique treatment times (J+1) than time horizon T.
<Ref> in the appendix gives conditions for
the uniqueness of implicit design in cross-sectional settings (T=1).

To be concrete, we explicitly compute the restriction in the binary treatment,
cross-sectional setting.

[<ref>, continued]

     Specialized to the setting of <ref>, the
restrictions on the potential weights—which we compute in <ref>—reduce to the single equation 
    π_i ρ_i(1) + (1-π_i) ρ_i(0) = 0.

This equation admits the sole solution π_i = -ρ_i(0)/ρ_i(1) - ρ_i(0)
if ρ_i(1) ≠ρ_i(0). Thus, the set of improper implicit designs is empty if any
unit has ρ_i(1) = ρ_i(0) ≠ 0; otherwise, the set of improper implicit
designs are real vectors  such that π_i = -ρ_i(0)/ρ_i(1) - ρ_i(0) for all units with
nonzero potential weights. Among these, a proper implicit design exists if ρ_i(1) ρ_i(0) ≤ 0 for all i.


Relative to a proper and Gram-consistent implicit design , the regression
specification estimates the -weighted average of the potential outcomes, which we
term the corresponding implicit estimand, indexed by potential outcome weights
_i(,) = π_i ρ_i().

    
    Let  denote a proper and Gram-consistent implicit design of a given regression specification. The
    corresponding implicit estimand is the -weighted average of the
    potential weights 
    () = 1/n∑_i=1^n ∑_∈_i(, ) _i(),

    where _i(, ) = π_i _i(). Without introducing ambiguity, we
    also use “implicit estimand” to refer to the weights _i(, ): i∈ [n], ∈.





 §.§ Implicit designs as diagnostics


The following theorem summarizes  our discussion of causal interpretations of the
population regression specification.

theoremthmmain
    
     is a vector of causal contrasts in the sense of <ref>
    if and only if it is equal to some implicit design.
    When this happens, if, furthermore, there is a unique implicit design , then
  is proper, Gram-consistent, and equal to ^*.


Having computed the potential weights, implicit designs, and implicit estimands,
<ref> allows us to answer the three questions <ref>–<ref> for
the regression our colleague presents us with. If  is a vector of
causal contrasts under ^*, then the true design ^* satisfies
(<ref>). Thus, inspecting the solutions of
(<ref>)—or lack thereof—gives us information about
^*.

To answer  <ref>, we can reject any design-based interpretation of the regression
if no proper and Gram-consistent implicit weights exist. When that happens, there is no
treatment assignment process—consistent with the population Gram matrix—that
rationalizes the regression estimand as a causal contrast. As a result, the regression
estimand cannot be interpreted causally, at least when we are unwilling to restrict the
behavior of potential outcomes. Researchers committing to such a regression specification
thus must entertain and justify models of potential outcomes.

On the other hand, if a proper and Gram-consistent implicit design does exist and is
unique, then we know that ^* must be equal to it—if we presume innocence for the
regression specification. This answers  <ref>. A candidate implicit design also
leads to further diagnostics and insights. We can inspect, for instance, whether the
implicit design conforms to our substantive knowledge of the treatment assignment process.
Moreover, when the implicit design is unique, units with same covariates _i must
necessarily have the same implicit design π_i. This means that the regression
specification must be justified by a finite population analogue of selection on
observables.[See <ref> for a detailed discussion, and see
<ref> and
<ref> for some primitive conditions under
which the implicit design is unique.]

Lastly, we can compute the implicit estimand under ^* by computing the weights
(^*, ), thereby answering <ref>. Having computed the estimand, we
can then inspect whether it is a reasonable quantity—we thus far only know that it
satisfies level irrelevance. For instance, we might inspect whether it is a positively
weighted average of individual treatment contrasts and whether it aligns with our
intuition of the regression estimand. <Ref> contains
examples where either requirement may fail even when an implicit design exists.

If these checks all pass—we find a proper and Gram-consistent implicit design under
which the regression estimates a reasonable causal contrast—then we can rest assured
that, under this treatment assignment process, our regression aggregates a sensible set of
comparisons. This immediately dispels worries about “negative weights” when we are
willing to defend the implicit design as reflecting reality. This also automatically
imbues the regression specification with a double robustness property: The regression
specification estimates causal quantities either if the treatment is assigned according to
the implicit design, or if the regression correctly specifies the conditional means of the
potential outcomes given treatment and covariates.[The meaning of these causal
quantities may be different under design-based and model-based justifications, however.]

In sample, however, these takeaways are complicated by the need to estimate G_n with
Ĝ_n, which we consider in detail in <ref>. That said, if the design
Π^* is a fixed Gram design (<ref>), then the potential weights are
directly observable, which yields direct answers to <ref>–<ref> when
combined with <ref> without needing to discuss estimation. In particular, for
<ref>, if we fail to find a proper and Gram-consistent implicit design, then the
regression specification cannot be interpreted causally under any fixed Gram
design.[Even if we find such implicit designs , but if we cannot find a
fixed Gram Π corresponding to , then we similarly cannot interpret the
regression causally under any fixed Gram design.] For <ref> and <ref>,
if we do find such designs —and if we further find a joint design Π
whose marginals are —then the regression has a design-based interpretation under
Π with implicit estimands given by <ref>.

When applied to the leading setting of binary treatments in a cross section
(<ref>), <ref> results in a sharp
characterization of the design-based interpretations of regression estimands, as the implicit design is unique up to units with zero potential weights.

[Binary treatment, cross-sectional setting]corcorbinary
    
    In the cross-sectional, binary treatment setting of
    <ref>, τ is a  causal contrast under
    the true design ^* if and only if ρ_i(1) ρ_i(0) ≤ 0 for all i and 
    π_i^* = -ρ_i(0)/ρ_i(1) - ρ_i(0)

    for all i with one of ρ_i(1) and ρ_i(0) nonzero.

    When this happens, the
    implicit estimand is defined by ω_i(^*, 1) = π_i^* ρ_i(1) and ω_i(^*, 0)
    = -ω_i(_i^*, 1), and thus 
    τ = 1/n∑_i=1^n ω_i^* (y_i(1) - y_i(0))  for ω_i^* ≡ω_i(^*, 1).

    The unit-level weights ω_i^* are negative if and only if ρ_i(1) < 0 < ρ_i(0).


Namely, the
regression estimand τ is a causal contrast if and only if the true design is equal to
the implicit design, which is uniquely defined up to units with equal zero potential weights.
Moreover, the implicit estimand is a weighted average treatment effect. All unit-level
weights ω_i in the implicit estimand are nonnegative if no unit has the “wrongly ordered”
potential weights ρ_i(1) < 0 < ρ_i(0).[If unit i has wrongly ordered potential
weights, then the regression compares unit
i's untreated
potential outcomes against its treated potential outcome, rather than the other way
around.]


Before discussing estimation of G_n, potential weights, and implicit designs in <ref>, we conclude this section with a few remarks on invariances of potential weights
to equivalent specifications, on restrictions placed on the
covariate transform, and on interpretation of .


[Invariances of potential weights]
    
    There are two ways in which the potential weights are invariant. First, the potential
    weights are invariant under reparametrizations of the covariate transform. That is, if
    we replace z_t(_i, _i) with z̃_t(_i, _i) ≡ M z_t(_i,
    _i) for an invertible M ∈^K × K and correspondingly replace
    Λ with Λ̃= Λ M'—so that Λβ and Λ̃β̃ are the same contrast—then the corresponding potential weights
    remain unchanged.

    Second, the potential weights satisfy a
    Frisch–Waugh–Lovell property. That is, suppose we partition z_t(, ) into
    z_t1(, ) and z_t2(,) with dimensions K_1, K_2 respectively.
    Assume that Λ loads solely on coefficients of z_t1(, ). Then the
    potential weights _i() are the same as the potential weights from the
    regression of _it on z̃_t(_i, _i) ≡ z_t1(_i, _i) - B_1→ 2 z_t2(_i, _i).
    Here, B_1→ 2∈^K_1 × K_2 is the matrix of population regression
    coefficients of z_t1 on z_t2: 
    B_1→ 2 = 1/n∑_i,t
        [z_t1(_i, _i) z_t2(_i, _i)']1/n∑_i,t
        [z_t2(_i, _i) z_t2(_i, _i)']^-1,

    where both expectations are taken under _i ∼π_i^*. Both claims are stated and proven in <ref>.



[Unit fixed effects]
    
    Including unit fixed effects in a regression specification can be parametrized by
    ensuring that the covariate transform z_t(_i, ) sum to zero: ∑_t=1^T
    z_t(_i, ) = 0 for all i and ∈. This holds by construction if we
    preprocess the covariates via the within transformation.

    It may be tempting to parametrize unit fixed effects as unit-level dummy
    variable in the covariate transform, so that the dimension of z_t grows with
    the size of the population. We can define potential weights relative to this
    parametrization, but these potential weights are going to be different in general to
    the potential weights defined under a within-transformation.
    Moreover, the Gram matrix G_n associated with this parametrization is in general no longer
    consistently estimable under reasonable conditions.[For unit i, let z̃_t(_i, _i) denote the covariate transforms that exclude the unit dummy. Then
    ∑_t=1^T [z̃_t(_i, _i)] is in the Gram matrix (it is the
    interaction between z̃_t and the unit-i dummy variable). However, this
    quantity is not consistently estimable as unit i is only observed once.] This is a
    manifestation of the incidental parameter problem. As a result, we maintain that
    unit-level fixed effects are accommodated by applying the within-transformation to the
    covariates z_t.


[Completion of (_, )]
    
    To compute the potential weights, it is crucial that (_i, ) is known for
    all counterfactual values of the treatment assignment. This rules out specifications
    in which post-treatment variables are included as covariates (e.g., lagged outcomes in
    panel settings). Similarly, this rules out specifications that include endogenous
    treatment variables on the right-hand side in an instrumental variables (IV) or
    mediation setting.[For IV estimators that are ratios of two linear treatment
    effect estimators, we can analyze the numerator (i.e., reduced form, or the treatment
    effect of the instrument on the outcome) and the denominator (i.e., first stage, or
    the treatment effect of the instrument on the endogenous treatment) separately under
    our framework. This accommodates, for instance, two-stage least squares.] In such
    cases, the counterfactual weight put on counterfactual outcomes depend on other
    counterfactual outcomes, and thus we cannot “complete” the regression representation
    in (<ref>).



[Unconfoundedness]
    
    When the implicit design is unique and  is a causal contrast under ^*,
    <ref> shows that the true design ^* is equal to the implicit design.
    The implicit design, however, is a function solely of the potential weights
    ρ_i(), which only depends on the covariates _i in the sense that two
    individuals with the same covariate values _i = _j must also then have the
    same potential weights ρ_i() = ρ_j() for all . If the implicit
    design is unique, then these two individuals must also have π_i = π_i^* = π_j^*
    = π_j. This property of ^* is a finite-population version of the
    unconfoundedness assumption, where the treatment is randomly assigned “conditional on
    _i.”

    Therefore, if a design-based interpretation is possible for parameters  under a
    _i-on-(_i, _i) regression, then this interpretation must be
    under a design that satisfies this unconfoundedness property—where selection
    propensities into
    treatment only depends on the covariates.




§ ESTIMATING POTENTIAL WEIGHTS



This section studies the properties of the estimated potential weights 
    _i() = ΛĜ_n^-1(_i, )'
     where Ĝ_n = 1/n∑_i=1^n ∑_t=1^T z_t(_i, _i) z_t(_i, _i)'.
 By considering finite populations of increasing size, it is often possible to show that
Ĝ_n - G_n  0—which is often used to show consistency for the OLS estimator.
This, coupled with an assumption that G_n is suitably invertible, allows us to conclude
that the estimated potential weights are consistent for their population counterparts.
Consistency of the inverse Gram matrix also implies that the OLS estimator is close to an
oracle Horvitz–Thompson estimator, at an order of O(Ĝ_n - G_n). In many
cases, the consistency of potential weights  implies consistency of estimated implied
designs and estimands, though the precise statement depends on further regularity
conditions. Quantitative bounds on the deviation Ĝ_n - G_n also allows us to
derive tests for the existence of proper implicit designs. Practically, these results
assure us that the estimated potential weights and implicit designs—by simply plugging
in the sample Gram matrix—are close to their
population counterparts in large samples, and further provides formal inference
procedures.

To discuss asymptotics, we embed our finite population in a sequence of populations.
Formally, let 
    _1(·),
    _1, _2(·), _2, …
 denote a sequence of potential outcomes and
covariates. Let π_1,n^*,…, π_n,n^* denote a triangular array of marginal treatment
assignment probabilities, and let Π_n^* denote the
joint distribution of the treatment assignments _1, …, _n under the nþ
population with marginals equal to π_1,n^*,…, π_n,n^*.




 §.§ Consistency of potential weights
 We begin by establishing the consistency of
_i for _i.  The results in this subsection are based on the following
assumption—that the sample Gram matrix is close to the population Gram matrix—regardless of whether the
population regression specification has a causal interpretation.


    
    Ĝ_n - G_n  0.


<Ref> holds whenever the underlying populations are such that a law of
large numbers hold. If the treatments are independently assigned, for instance, then
<ref> is true under standard laws of large number (e.g., Theorem 2.2.6 in
<cit.>), which we state in the following lemma.
lemmalemmalln
    
    Suppose that in each Π_n^*, treatments are independently assigned according to
    π_i^* = π_i,n^*. Let G_i = ∑_t=1^T z_t(_i, _i) z_t(_i, _i)'. Assume that
    for all 1 ≤ j ≤ K, 1≤ℓ≤ K, the average second moment
    of G_i, jℓ grows slower than n:  as n→∞, 1/n1/n∑_i=1^n _π_i,n^*[G_i, jℓ^2]→ 0.
    Then <ref> holds.


Ĝ_n is consistent for G_n under more complex designs as well. <Ref> verifies <ref> for
designs that involve sampling without replacement <cit.>.

Next, we impose the following assumption on the population Gram matrices to ensure that
Ĝ_n^-1 - G_n^-1 0 as well.[Under <ref> and
<ref>, one can also show that Ĝ_n is invertible with probability
tending to one, and thus we may write Ĝ_n^-1 without essential loss of generality.]


    
    The sequence of population Gram matrices G_n is such that their minimum eigenvalues are
    bounded below: For some ϵ > 0, for all n, λ_min(G_n) ≥ϵ > 0.


<Ref> makes sure that the population regression specification is strongly
identified, in the sense that the Gram matrix is bounded away from singularity. Under
<ref> and <ref>, we can show that the estimated
potential weights (<ref>) are consistent for their population counterparts.

proppropweightconsistent
    Under <ref>, the estimated potential weights are
    consistent: _i() - _i()  0. If z_t(_i, )_∞ is bounded uniformly in i∈[n] and ∈, then
    the consistency is also uniform: 
    max_i ∈ [n], ∈ |_i() - _i()|_∞ 0.

    where |·|_∞ takes the entrywise maximum absolute value.




 §.§ OLS as approximately Horvitz–Thompson

Under <ref>, we can also make a connection between the OLS
estimator and an oracle Horvitz–Thompson estimator. Note that the OLS estimator for  is an
average of the observed outcome _i weighted by the observed estimated potential weights
(_i): 
    = 1/n∑_i=1^n _i(_i) _i = 1/n∑_i=1^n ∑_∈(_i = ) _i() _i().

The corresponding population estimand is = 1/n∑_i=1^n ∑_∈π^*_i() _i(_i) _i irrespective of whether
it is a causal contrast. With knowledge of _i(_i) and π^*_i(), a natural
Horvitz–Thompson estimator targeting  is 
    _  = 1/n∑_i=1^n (_i = )/π^*_i(_i)π^*_i()
    _i(_i) _i = 1/n∑_i=1^n ∑_∈(_i = ) _i() _i().

Thus, the OLS estimator (<ref>) can be viewed as the analogue of the
Horvitz–Thompson estimator (<ref>) with the estimated potential weights
replacing their population counterparts. The following proposition makes that precise
under mild boundedness conditions on the sequence of populations.
proppropht
    Assume that the sequence of populations is such that the average covariance between
    the covariates and potential outcomes is bounded 
    1/n∑_i=1^n max_∈(_i, )'
        _i()_∞  = O(1).

    Under <ref>, let constants b_n>0 be such that
    |Ĝ_n - G_n|_∞ = O_P(b_n). Then
    - __∞ = O_P(b_n).

Thus, in this sense, the OLS estimator is close to a natural design-based estimator for
the estimand .[While  and _ are close, since Ĝ_n - G_n_∞ is usually
on the order of 1/√(n), the two estimators are not generally
first-order asymptotically equivalent.] When  has a causal interpretation in the
sense of <ref>, the OLS estimator can be interpreted as an
estimator of a causal quantity, asymptotically justified under design-based assumptions.

With a proper estimated implicit design , we can also write the OLS
estimator in a form suggestive of inverse propensity weighting: 
    = 1/n∑_i=1^n ∑_∈(_i =
    )/π̂_i()π̂_i() _i() _i().
 When  is consistent for ^*, this representation is directly informative
of the design-based interpretation of . In that case, the OLS estimator can be
interpreted as a Horvitz–Thompson estimator for the
estimand 1/n∑_i∑_π̂_i() _i() y_i(), with
propensity score π̂_i(·).
Crucially, however, we emphasize that the OLS estimator is close to the Horvitz–Thompson
estimator regardless of whether  is close to ^*, as long as the estimated
potential weights are consistent for their population counterparts.




 §.§ Consistency of implicit designs



Naturally, we define the set of estimated (improper) implicit designs as the set of solutions to the
sample counterpart of (<ref>):  is an
estimated implicit design if for every i, 
    ∑_∈π̂_i() _i() = 0,   ∑_∈π̂_i() = 1.

The next proposition, <ref>, verifies that the estimated implicit designs are consistent in the
sense that the population implied weights have zero expectation in the limit under members
of . In that sense, the estimated implicit designs are limiting solutions to the
population level irrelevance restriction (<ref>).

proppropobjective
    
    Suppose that, for some 0 < C < ∞, with probability tending to one, an estimated implicit
    design  exists and is bounded in the sense that:
    max_i max_∈|π̂_i()| < C.
 Let  be a bounded
    estimated implicit design if it exists, and let it be an arbitrary probability vector
    otherwise. If |_i() - _i()|_∞ 0, then ∑_∈ρ_i() π̂_i()  0.
    If further (<ref>) holds, then the convergence is uniform: 
    max_i ∈ [n], ∈[]∑_∈_i() π̂_i()_∞ 0.



Coupled with an assumption about the uniqueness of population implicit design (i.e.,
uniqueness of solutions to (<ref>)), the estimated
implicit designs are then consistent for ^* in the usual sense. We illustrate with
the following result in the cross-sectional case (T=1) where the system of equations
(<ref>) is exactly determined (k=J). To state this
result, let  R_i = [_i(0),…, _i(J)] ∈^k × (J+1) be the matrix
whose columns are potential weights at each treatment level. Let R_i = ∑_r=1^Jσ_ir u_ir v_ir' be its singular value decomposition, for σ_1i≥…≥σ_iJ≥ 0, u_i1,…, u_iJ orthonormal vectors in ^J and
v_i1,…, v_iJ orthonormal vectors in ^J+1.

proppropimplicit
    
    Assume that k= J. Suppose that:
    
        
  * The estimand  is a vector of causal contrasts in the sense of
        <ref> under ^*.

        
  *   Given ρ_i(·), the smallest singular
        value of R_i is uniformly bounded away from zero: For some ϵ > 0,
        lim inf_n→∞min_i ∈ [n]σ_iJ > ϵ > 0.

        
  * The estimated potential weights are consistent in the sense of
        (<ref>).
    
    Then, for some C>0, a unique estimated implicit design  that is bounded by
   C exists with probability tending to one.
    Let  be an estimated implicit design if it exists, and otherwise let
    be an arbitrary probability vector. Then  is consistent for
   ^* in the sense that 
    max_i ∈ [n], ∈ |π̂_i() - π_i^*()|  0.



The key condition in <ref> is
<ref>, which ensures that the solution to the population level
irrelevance restriction (<ref>) is unique and robust to
small perturbations of the potential weights. In the binary treatment, cross-sectional
setting, <ref> is equivalent to that the potential weights are bounded
away from zero: ρ_i(1)^2 + ρ_i(0)^2 > ϵ^2. <Ref>
further provides sufficient conditions for <ref> to hold in the
cross-sectional case.

Unfortunately, in general, the estimated implicit designs are not always well-behaved.
This is because they are the solutions to a system of overdetermined linear equations with
noisy coefficients, and small perturbations to the coefficients can result in large
changes in the solutions—or even to their existence in the first place. For instance, it
is possible for noise in the coefficients of a underdetermined system to turn the system
into an overdetermined one. Thus, it is possible that a population implicit design
exists—meaning that (<ref>) is under or exactly
determined—but no estimated implicit designs do. For instance, the theoretical
applications in <ref>
(<ref>) may exhibit this phenomenon. Nevertheless, the
estimated implicit designs—or their existence thereof—are directly informative about
any interpretations under fixed-Gram designs.

Finally, it is in principle possible to perform statistical inference for implicit
designs. For a confidence set of implicit designs, one simple method is to project a
confidence set for G_n, perhaps itself derived from large-deviation inequalities for the
sample Gram matrix. For a given (1-α)-confidence set 𝒢̂_n for
G_n,[For instance, such a confidence set can be obtained from the simultaneous
confidence set for all entries of G_n—which are themselves population means of sample
averages and obey central limit theorems under typically mild conditions.] we can project
it to form a confidence set of proper and Gram-consistent implicit designs: 
    Π̂= { :     G ≡1/n∑_i=1^n ∑_∈π_i() (_i, ) (_i, )' ∈𝒢̂_n, Gram-consistent
    and in 𝒢̂_n
    
        for all i, π_i(·) ≥ 0,  ∑_∈π_i() = 1, Proper
       ∑_∈π_i() Λ G^-1(_i, )' = 0 }. Level irrelevance


Unfortunately, computing this set is a nonconvex, nonlinear problem,[To be
precise, checking whether this set is nonempty is a nonconvex and nonlinear feasibility
problem. ] which we conjecture to be computationally challenging. Nonetheless, checking
whether a candidate  belongs to this confidence set is straightforward. Thus, if the
researcher has a particular design in mind—perhaps in response to the red flags raised by
estimated implicit designs—they can check whether such a design is a plausible
justification for the specification in question.[In particular, given a
proper implicit design, we can compute the corresponding Gram matrix under that design and
check whether it belongs to 𝒢̂_n.]



§ THEORETICAL APPLICATIONS AND EXAMPLES



Having established the machinery of potential weights and implicit designs, we now reap
its fruits.
Our results are useful both for theoretical investigation and for practical diagnostics
with a given dataset. To emphasize,
for both purposes, our results essentially reduce the problem to computing the
(estimated) potential weights and the (estimated) set of implicit designs.

This section computes the potential weights and implicit designs for a number of
population regression specifications. Some of these specifications are well-studied in the
literature and have known causal interpretations under certain design assumptions <cit.>.
We show that the corresponding implicit
designs essentially rediscover these properties and—moreover—supply a converse of sorts.
Specifically, the implicit design is often unique given the population regression
specification, meaning that there is a unique design that is (a) consistent with a given
Gram matrix G_n and (b) grants the regression a causal interpretation in the sense of
<ref>. In this sense, then, the design assumptions justifying
these specifications in the literature are the only such assumptions.

When applied to a number of examples that are less known, our results uncovers some new
insights. We show that the following regression specifications generally do not have
causal interpretations, in the sense that their sets of implicit designs are either empty or do
not include ^*, except under certain knife-edge cases:

    
  * In a cross-section, binary-treatment setting, consider τ in the
    specification Y_i = α + τ W_i + γ'(x_i - x̅) + δ'W_i (x_i -
    x̅) + ϵ_i <cit.>  where x̅ =
    1/n∑_i=1^n x_i. The covariates x_i do not represent saturated
    categorical covariates.

    
  * In a balanced panel setting with staggered adoption, consider τ in the TWFE specification Y_it = α_i +
    β_t + τ W_it + γ'x_it + ϵ_it, where inclusion of the
    covariates changes the coefficient on W_it.

    
  * In an unbalanced panel setting with staggered adoption, consider τ in the TWFE specification
    Y_it = α_i +β_t + τ W_it + ϵ_it, where the distributions of treatment times
    are different across units with different observable time periods.


While we present a litany of results here in starkly distinct settings, we emphasize
that in each example we uncover these results by mechanically computing the potential
weights, implicit designs, and estimands, with little tailored analysis. Thus our results
provide a unified understanding of design-based interpretations of regression
specifications. For ease of exposition, we divide these results into cross-sectional
settings and panel settings. Throughout this section, we assume that the population Gram
matrix is invertible, and our results about estimated potential weights and estimated
implicit designs are limited to the event that the sample Gram matrix is invertible.



 §.§ Cross-sectional applications





  §.§.§ Binary treatments, <cit.>
 We begin with the leading case of cross-sections with binary treatments. We first study
the regression specification in <cit.>, where one “controls for” the
covariates x_i  in a outcome-on-treatment regression: 
    Y_i = τ W_i + γ' x_i + ϵ_i.

Loosely speaking, <cit.> finds that (<ref>) estimates the
π^*_i (1-π^*_i)-weighted average treatment effect when the propensity score π_i^*
is linear in x_i. The following proposition shows a converse: Given any true design
^*, the set of implicit designs contains only ^*'s linear projection onto the
covariates. Thus, for ^* to justify a causal interpretation of τ, it must then
be linear in the covariates to begin with.

proppropangrist
    
    Let the population regression specification be described by (<ref>) under
    the design ^* = (π^*_1, …, π^*_n), where the coefficient of interest is
    τ. Let π_i = x_i'β_w → x be the linear projection of ^* onto
    x_i: 
    β_w→ x = 1/n∑_i=1^n x_i x_i'^-11/n∑_i=1^n x_i π^*_i.

    Then the improper implicit design is unique and equal to . If τ is a causal
    contrast under ^*, then (i) ^* =, (ii) ^* is linear in x_i, and
    (iii)  is proper. When that happens, the
    implicit estimand is the π_i(1-π_i)-weighted average treatment effect 
    τ = 1/n∑_i=1^n π_i(1-π_i)/1/n∑_j=1^n π_j (1-π_j)_ω_i (y_i(1) - y_i(0)),

    and ω_i is nonnegative if and only if π_i ∈ [0,1].


<Ref> presents the sample analogue of <ref>. We show that the
estimated implicit design is uniquely π̂_i = x_i'β̂_w→ x, where
β̂_w→ x is the regression coefficient of the observed treatment indicators
W_1,…, W_n on the covariates. The estimated estimand is then the π̂_i(1-π̂_i)-weighted average treatment effect.

<Ref> also immediately implies a few corollaries. First, if the covariates
only include a constant—in which case τ is a simple mean comparison between treated
and non-treated units—then the implicit design is necessarily constant (π_i = π).
We thus know
that for τ to have a design-based causal interpretation, the true design must be a
randomized experiment. Second, if the covariates represent saturated versions of an
underlying categorical variable c_i,[That is, x_i = e_m if c_i = m for c
taking values in 1,…, M and e_m the mþ standard basis vector.] then the
implicit design is simply a stratified randomized experiment on levels of c_i, which is
then required if τ were to be causal under ^*. Moreover, when the design is a
stratified randomized experiment, a fixed Gram design exists by running a completely
randomized experiment within each stratum—when that is true, the estimated implicit
design (described by the within-stratum realized proportions of treatment) is exactly the
true design. Finally, if the linear probability model is obviously misspecified in the
sense that π_i ∉[0,1] for some i, then we must reject the causal
interpretation of τ under ^*.



  §.§.§ Binary treatment, interacted specification
 For a researcher interested in the average treatment effect, the regression specification
(<ref>) can be unsatisfying, as it upweights units with propensity scores
closer to 1/2.[The interpretation of the estimand thus depends on the
experimental design, which may be unrelated to policy objectives. On the other hand,
<cit.> do give this estimand an interpretation as the
easiest-to-estimate
weighted average treatment effect—the treatment effect with the most favorable
semiparametric efficiency bound.] It is well-known that when the covariates are
saturated versions of some
categorical variable, the following regression—which interacts the treatment indicator
with the covariates—recovers the average treatment effect with τ under a stratified
randomized experiment <cit.>: 
    Y_i = α + τ W_i + γ'(x_i - x̅) + δ' W_i (x_i - x̅) + ϵ_i
      x̅≡1/n∑_i=1^n x_i
 Analogous to <ref>, <ref> likewise shows that, when x_i
represents saturated categorical covariates, stratified randomized experiments are the
only designs justifying a causal interpretation of τ; when this happens, the
corresponding implicit estimand is the average treatment effect.

Given <ref>, one might conjecture that the same is true when x_i is not
saturated—for instance when x_i is continuous—as long as the true propensity scores
π_i^* are linear in the covariates. Surprisingly, we show that (<ref>) does not
in general have a causal interpretation justified by design,
even when π_i^* is linear in the covariates.

To describe this result, we define a few population projection coefficients. Let the population regression specification be described by (<ref>) under
    π_i^*.
    Let π̃_i be the linear
    projection of π_i^* onto a constant and x_i. Moreover, let Γ_0, Γ_1
    be the population projection coefficient of π_i^*(x_i-x̅) on a constant and
    x_i-x̅, respectively. Let Γ_2 be the population projection coefficient of
    W_i - π̃_i on the residual W_i (x_i - x̅) - Γ_0 - Γ_1(x_i - x̅).[That is, 
    Γ_2    = 1/n∑_i=1^n _W_i
        W_i (x_i - x̅)W_i (x_i - x̅) -
        Γ_0 - Γ_1(x_i - x̅)'
        ^-1·
       1/n∑_i=1^n _W_i
            (W_i - π̃_i) ·W_i (x_i - x̅) - Γ_0 - Γ_1(x_i - x̅).
] In other words, the population linear projection of W_i on 1, x_i-x̅, W_i
    (x_i - x̅) can be written as 
    W_i    = π̃_i + Γ_2'
                W_i ( x_i - x̅) - Γ_0 - Γ_1 (x-x̅)
             + u_i
            
       = - Γ_2'Γ_0 + Γ_2'W_i ( x_i - x̅) + (π̃_i -
        Γ_2'Γ_1(x-x̅)) + u_i, 
         [u_i 1, (x_i-x̅)', W_i
        (x_i - x̅)''] = 0.

    Finally, we say that vectors v_1,…, v_n are generically non-affine in x_1,…, x_n
    if the affine projection v_i = A_0 + A_1 x_i + r_i has residuals that are full rank:
    1/n∑_i=1^n r_i r_i' ≻ 0.



propproplinneg
    
    Under the setting described above for the specification (<ref>), the implicit
    design is equal to 
    π_i = π̃_i -Γ_2' (Γ_0 + Γ_1(x_i - x̅))
        / 1- Γ_2'(x_i-x̅).

    for units with Γ_2'(x_i - x̅) ≠ 1.
Moreover,

    
  *  Suppose π_i^*(x_i - x̅) is affine in x_i. Then
    τ has a causal interpretation under π_i^* if and only if π_i^* is affine
    in x_i. When this happens, π_i^* = π̃_i for all i, π_i = π̃_i = π_i^* for all i with Γ_2'(x_i - x̅) ≠ 1,  and the estimand
    is a weighted average treatment effect with weights proportional to ω_i ∝π_i(1-π_i) (1-Γ_2'(x_i - x̅ )).
    
  *  Suppose π_i^* is affine in x_i but π_i^*(x_i - x̅) is generically
    non-affine in x_i. Then τ has a causal interpretation under π_i^* if and
    only if 1/n∑_i=1^n π_i^*(1-π_i^*)(x_i - x̅) = 0. When this happens,
    the estimand is a weighted average treatment effect with weights proportional to π_i^*(1-π_i^*).



<Ref> gives two scenarios where the regression specification (<ref>)
has causal interpretation under affine propensity scores. The upshot is that neither
scenario is plausible, unless either π_i^* is constant or x_i represents saturated
discrete covariates. This thus suggests that a design-based interpretation for
τ in (<ref>) requires a randomized experiment or a saturated
discrete covariate. In general, therefore, the “saturate” in interact-and-saturate can
not be omitted; the landscape of causal interpretation in binary-treatment regressions is
summarized in <ref>.



The first
scenario <ref> requires not only that the propensity score is affine in x_i, but also that
π_i^*(x_i - x̅) is affine in x_i. This scenario is unlikely since π_i^*(x_i -
x̅) is a quadratic expression in x_i if π_i^* is affine in x_i. Nevertheless, π_i^*(x_i -
x̅) can be affine when
x_i takes at most (x_i) + 1 distinct values, which effectively means that x_i
represent saturated discrete covariates. It also holds when π_i^* is constant in
x_i—i.e., the true design is a randomized experiment
<cit.>. On the other hand, if π_i^*(x_i - x̅) is not affine in
x_i, then <ref> states that, under linear propensity scores, τ has a causal interpretation only when
the mean covariate value x̅ also happens to be the weighted mean under
π_i^*(1-π_i^*). This is a knife-edge scenario[This scenario is not
impossible: It holds, say, when x_i has
support on [0,1] that is symmetric about 1/2, and π_i^*(x) = x, for instance.] and
is generically violated.

The reason for this difficulty is that the implicit design
(<ref>) is defined by a strange expression. This
expression is a rearrangement of the condition that the potential weights have mean zero
under the true design:

    π_i^* - π̃_i_Linear approximation error of π_i^* =
    Γ_2'(π_i^* (x_i - x̅) - Γ_0 - Γ_1(x_i - x̅))_Linear approximation error of π_i^*(x_i - x̅).

<Ref> states that the linear approximation error in the true
propensity score just so happens to be linearly related to the linear approximation error
in π_i^*(x_i - x̅), where the coefficient in that relationship is exactly
Γ_2. If the true design is linear, then π_i^* - π̃_i = 0, and the two
scenarios <ref> come from either π_i^* (x_i - x̅) = Γ_0 + Γ_1(x_i - x̅) or Γ_2 = 0. <Ref> also therefore suggests that
causal interpretation under nonlinear propensity scores is difficult, since it relies on a
coincidental relationship between two approximation errors.



  §.§.§ Multiple treatments, <cit.>



Our results also recover insights from the recent work by
<cit.>. They consider the
following regression specification for a multivalued treatment W_i ∈0,…, J,
analogous to (<ref>):

    Y_i = α_0 + ∑_j=1^J τ_j (W_i = j) + γ'x_i + ϵ_i.

In this specification, we might be tempted to interpret τ_j as a treatment effect
between the jþ treatment value and the baseline treatment value W_i = 0.
<cit.> show that, under propensity scores linear in x_i,
τ_j does not solely measure the contrast between treatment j and control (W=0). In
contrast, τ_j is contaminated by treatment effects between other treatments (ℓ∉0, j) and the control condition.

It is simple to analyze this regression specification fully numerically, at least given a
concrete data-generating process. To do so, we follow the numerical example in Section 2.2 of
<cit.>.
Consider the specification (<ref>) with binary
x_i and J=2. Suppose exactly half the units have x_i = 1. For the
units with x_i = 0, π^*_i(0) = 0.5, π^*_i(1) = 0.05, and π^*_i(2) = 0.45. For the
units with x_i = 1, π_i^*(0) = 0.1, π_i^*(1) = 0.45, and π_i^*(2) = 0.45.
The population regression specification is described by (<ref>) under
these assignment probabilities, where the coefficients of interest are = [τ_1,
τ_2]'.

Numerically, we can verify that the assignment
probabilities ^* are the only solution to
(<ref>), and thus ^* is the only implicit design
for  this regression specification. The corresponding
implicit estimand for τ_j is
then 
    τ_j = 1/n∑_i=1^n ∑_j=0^J π_i(j)ρ_i(j)_ω_i(, j) y_i(j),

which is the  sum of potential outcomes weighted by ω_i(, j).

We can calculate the implicit estimand as well, and inspecting the implicit estimand
allows us to recover the contamination bias in <cit.>. For each x_i value and each estimand, we
tabulate ω_i(·) for individuals with those x_i: 
    3cτ_1   3cτ_2
       ω_i(0)   ω_i(1)   ω_i(2)   ω_i(0)   ω_i(1)   ω_i(2)
    x_i = 0   -140/106   41/106   99/106   -160/106   9/106   151/106
    x_i = 1   -72/106   171/106   -99/106   -52/106   -9/106   61/106
 We find that the implicit estimand for τ_j does not solely involve y_i(0) and
y_i (j), contrary to intuition; moreover, τ_j does not measure the same estimand
for individuals with x_i=0 and x_i=1. This echoes the result by
<cit.>.[In fact, under ^*, the estimand
τ_1 is 
    τ_1 = 1/241/106τ̅_1 | x=0 +
    99/106τ̅_ 2 | x=0 + 1/2171/106τ̅_1 |
    x=1 - 99/106τ̅_ 2 | x=1
 where τ̅_k | x=j is the
mean of y_i(k) - y_i(0) among those with x_i = j. This decomposition exactly matches
the decomposition (7) in <cit.>.] Our framework automatically
finds qualitatively and quantitatively the same contamination issue—we show that, for
this population regression specification, if both τ_1 and τ_2 satisfy level
irrelevance, then the design must be the stratified randomized experiment under ^*,
under which τ_1 and τ_2 are contaminated estimands even from a design-based
perspective.

Our framework also allows researchers to verify that the interacted version of
(<ref>) does retain a design-based interpretation in this case with binary x_i. If we compute the
same potential weights for = [τ_1,τ_2]' in the regression with interaction

    Y_i = α_0 + ∑_j=1^J τ_j (W_i = j) + γ x_i +∑_
    j=1^J  δ_j W_j (x_i - x̅) + ϵ_i,

we find that (a) the only implicit design is ^* and (b) under this design, the
implicit estimand is the ATE.[That is, _i(^*, 1) = 1,
_i(^*,0) = -1, _i(^*,2) = 0 for τ_1 and _i(^*,1) =
0, _i(^*,0) = -1, _i(^*,2) = 1 for τ_2.] However, consistent
with a natural extension of <ref>, the same is not true if the covariates
are not saturated—we generally find a unique implicit design that does not equal
^*.



 §.§ Panel settings



We now turn to panel settings (T > 1). We identify the set of treatments as
treatment paths ⊂0,1^T for a binary treatment. Since we write potential
outcomes as _i (), defined with respect to the treatment path, we do not rule out
dynamic or anticipated treatment effects.  Let us first consider a balanced panel setting
for notational clarity; the last result in this subsection concerns unbalanced panels. We
define a leading treatment assignment pattern in panel settings—staggered adoption.












We say ⊂0,1^T satisfies staggered adoption if _t ≤_s
for all t ≤ s and for all ∈.









  §.§.§ Two-way fixed effects

A popular specification is the two-way fixed effect (TWFE)
regression,[<Ref> states that one-way fixed effect, in contrast,
usually does not have causal interpretation under staggered adoption and compares with
recent work by <cit.>.] where one regresses
the outcome on some transformations of the treatment path f_t() as well as
individual and time treatment effects. This encompasses both the simple regression where
f_t() = _t is the contemporaneous treatment status and more complex event-study
specifications, where f_t() is the vector of lags relative to treatment time. We
find that the implicit designs of such specifications contain—sometimes uniquely—the
design where units are randomly assigned treatment paths.

propproptwfe

Consider the regression Y_it = α_i + γ_t + β' f_t(_i) with target
estimand = Λβ under some design ^*. We have:

	
  * Whether or not  satisfies level
irrelevance under the true design ^*, one proper and Gram-consistent implicit design
is 
    π_i () ≡π() ≡1/n∑_
    j=1^n π_j^* ().

Correspondingly, one estimated implicit design sets π̂_i() to be the empirical
frequency of treatment path  for all i.


  * When f_t(_i) = W_it, so that the regression is Y_it = α_i +
γ_t + τ W_it + ϵ_it, if  satisfies staggered
adoption and  excludes an always-treated unit, then the set of implicit designs is
a singleton equal to (<ref>).
Correspondingly, the only solution to
(<ref>) sets π̂_i()
as the empirical frequency of treatment path  for all i.



The first claim, <Ref>(i), simply states that, algebraically, the design where
_i is randomly assigned according to their marginal probability solves the level
irrelevance condition (<ref>) and is Gram-consistent.
Thus, if ^* does correspond to random assignment, then  is a vector of causal
contrasts: That is, coefficients in any two-way fixed effect specification
correspond to causal contrast under random assignment of treatment paths. An empirical
consequence is that using (<ref>) in practice to
find an implicit design will always at least return the design where units are randomly
assigned  according to their empirical frequency.  The second claim,
<ref>(ii), shows that, at least in the staggered adoption case,  random
assignment of treatment time is the only design that generates parameters
satisfying level irrelevance, given a population regression specification.[There
is nothing particularly special about staggered adoption—the same argument for
uniqueness extends to any case where the nonzero paths in  are linearly independent
and do not span 1_T.]


<Ref> echoes and complements the analysis of <cit.>, who
analyze the specification Y_it = α_i + γ_t + τ W_it + ϵ_it
under staggered adoption and complete random assignment of treatment timing. This design is a fixed Gram
design, and thus the potential weights are directly observable without estimation error. In particular, following (<ref>), the TWFE
estimator can be written as 
    τ̂=
    ∑_t=1^T ∑_∈_t(^*, )y̅_t,  _t(^*, ) ≡π^*() _t()

where y̅_t, is the sample mean of time-t outcomes of individuals receiving
treatment path .[We drop the i subscript since all π_i and _i
are the same.] This decomposition is identical to Lemma 5 in <cit.>. The
expectation of τ̂, under completely random treatment timing, is the following
weighted average of mean potential outcomes

    ∑_t=1^T ∑_ ∈_
    t (^*, ) 1/n∑_ i=1^n y_it(),
 which echoes Theorem 1 in
<cit.>. While
<cit.> show that the two-way fixed effect estimand has this
interpretation under random treatment timing, <ref>(3) shows a converse: τ fails to satisfy level irrelevance under any other design with the same
population Gram matrix.

Furthermore, we can often inspect the weights _t(^*, ). One
requirement and verify if the weights satisfy some additional requirements. For instance,
one might wish to impose that the post-treatment weights are nonnegative (i.e.,
_t(^*, ) ≥ 0 if _t = 1). Failure of this condition implies that
post-treatment units are severely used as comparisons for newly treated units, echoing the
“forbidden comparison” issue in the recent difference-in-differences literature
<cit.>. <Ref>
shows that when  has only
two elements and includes a never treated unit (i.e., classical difference-in-difference),
all weights post treatment are non-negative, but such forbidden comparisons are possible
in other cases.



  §.§.§ TWFE with time varying covariates
 It is common empirical practice to
include time-varying covariates in a two-way fixed effect regression. Surprisingly, by our
criteria, such a specification usually does not have a design-based
interpretation—unless the time-varying covariates do not covary with treatment paths.
This is in the sense that the set of implicit designs can be and often is empty. In
practice, this also means that the equation (<ref>)
would fail to admit any (proper and Gram-consistent) solutions .

propproptimevarying

Let the population regression specification be described by the regression Y_it =
α_i + γ_t + τ W_it + δ'x_it under ^* where τ is the
coefficient of interest. Let
β_w→ x be the population projection coefficient of W_it on x_it,
including individual and time fixed effects. For the set of implicit designs to be
non-empty, a necessary condition is

    _i - 1/n∑_j=1^n _jβ_w→ x∈(∪1_T)  for all i∈ [n].



<Ref> states that a necessary condition for the set of implicit
designs under ^* to be nonempty is that a particular linear combination of the demeaned
covariates sits in the linear span of  and 1_T for every unit. Under staggered adoption, when
there are relatively few adoption dates, ∪1_T is a small linear
subspace of ^T. If β_w→ x is nonzero under , it is thus highly
knife-edge that _i - 1/n∑_ j=1^n _jβ_w→ x happens to
be located in that subspace, barring special structure on the covariates _i. On the
other hand, if β_w → x is zero under ^*, then by the Frisch–Waugh–Lovell
theorem, including the covariates makes no difference to the coefficient on W_it in
population. In any case, it is highly unlikely that the sample analogue of β_w →
x is zero, and so we would usually fail to find any solution to
(<ref>).




These caveats aside, we interpret <ref> to mean that two-way fixed
effects regression with time-varying covariates does not in general have a design-based
interpretation, in the sense that the set of implicit designs is empty. Researchers using such a specification either believe that the covariates do not
affect treatment assignment and are thus immaterial to identification from a design-based
perspective, or they are implicitly embedding certain outcome modeling assumptions.



  §.§.§ TWFE with unbalanced panels

Lastly, we consider unbalanced panels. Suppose units
are observed in some subperiod _i ⊂1,…, T. We consider the TWFE
specification 
    Y_it = α_i + γ_t + τ W_it + ϵ_it

over i =1,…, n and t ∈_i. The objects _1,…, _n are considered
fixed and observed. Much like the case with time-varying covariates, we find that TWFE
with unbalanced panels also does not have a causal interpretation under staggered
adoption—unless treatment time is uniformly randomly assigned and uncorrelated with
_i.

To introduce the result, let _ be a multiset collecting the non-zero treatment paths
restricted to some subperiod ⊂ [T]. That is, _ collects the vectors
(_t: t ∈) for all ∈∖0_T, possibly with repetition. We say that  has rich variation on
 if the vectors in _ are linearly independent and their linear span excludes the vector
of all ones. As an example, note that if  satisfies staggered adoption and excludes an always
treated unit, then the period that spans all adoption dates, = t_min - 1,…, t_max—where t_min is the
first adoption date and t_max is the last adoption date, is a subperiod on which
 has rich variation.

proppropunbalanced
    
    Let the population regression specification be characterized by
    (<ref>) under ^*, where τ is the coefficient of
    interest. Suppose  satisfies staggered adoption and excludes always-treated
    units. Assume further that there is a commonly observed period ⊂⋂_i=1^n _i
    such that the treatment paths have rich variation in .

    Let π̅() ≡1/n∑_i=1^n π_i^*(). Let Q_i() =
    ∑_t∈_i_t/|_i| be the proportion of treated periods within
    unit i's observed period for treatment path .
    Then an implicit design exists if and only if ^* is uncorrelated with the
    missingness pattern in the sense that for all t ∈ [T]: 
    ∑_∈π̅() ∑_i: t ∈_i (_t -  Q_i()) = ∑_∈∑_i: t ∈_iπ_i^*() (_t - Q_i()).

    When this happens, the implicit design is unique and satisfies π_i() = π̅() for all i.



<Ref> gives a necessary and sufficient condition for an implicit design
to exist, and characterizes the implicit design when it does. In short, an implicit design
exists if and only if the missingness patterns are uncorrelated with the treatment timing
in a particular sense. When it exists, the implicit design is equal to a design where the
treatment timing is randomly assigned. Therefore, if ^* justifies τ as a
design-based estimand in the sense of <ref>, then ^* must
equal to the implicit design and thus must describe random assignment.

<Ref> generalizes <ref>. When applied to a balanced panel, the
condition (<ref>) is always satisfied, and we recover the
characterization of the implicit design in <ref>. With unbalanced panels, the condition
(<ref>) is in general difficult to satisfy unless ^*
describes random assignment, and when it fails the set of implicit designs is empty. In this sense, (<ref>) usually does not
have a causal interpretation unless the treatment timing is uniformly randomly assigned.

<Ref> assumes that there is a commonly observed period where the
treatment paths have rich variation. This is not a strong condition. If every unit's
observed period includes the period spanning all adoption dates, then this condition is
automatically satisfied. We also emphasize that this condition is only a sufficient
condition for a clean theoretical characterization, and we expect the set of implicit
designs to be generally empty when ^* is not a random assignment of treatment
timing, outside of certain knife-edge configurations.

The sample analogue of (<ref>) is that for all t ∈ [T], 
    ∑_∈π̂() ∑_i: t ∈_i (_t -
    Q_i())_Expected sum (among units observing t) of demeaned
    treatment, if treatment
    is randomly assigned according
    to π̂(·) =
    ∑_i : t ∈_i(W_it - Q_i(_i))_Sum of realized demeaned
    
    treatment among units observing t,

where π̂() is the empirical frequency of _i =. This is a difficult
condition to satisfy unless the empirical frequency of each path is identical across units
with different observation patterns _i. Thus, in practice, the set of estimated implicit
designs is usually empty, unless the treatment timing is completely randomized within each
observation pattern. Thus, even if the true design ^* is random assignment, the
estimated implicit designs usually do not exist, illustrating the difficulty of estimating
implicit designs that we allude to in <ref>. Nevertheless, when the set of
estimated implicit designs is empty, we can still reject the existence of a fixed-Gram
design that rationalizes τ as a causal contrast—thus ruling out a result like
<cit.> for the unbalanced setting.



§ DISCUSSION AND CONCLUSION



This paper examines the design-based interpretation of linear regression estimators.
Rather than asking whether a specification estimates a causal parameter under a given
design, we fix the linear specification and ask what design exists to rationalize the
linear regression as a causal estimator. We define potential weights and find that they
encode the causal comparisons the linear estimator takes, and seek a design that
rationalizes these comparisons. We call these designs implicit designs. The potential
weights themselves are also readily estimable in practice. We show that these potential
weights provide a simple diagnostic for whether a linear estimator has a design-based
interpretation.

When applied to a number of specifications encompassing binary treatment, multivalued
treatment, and panel settings, the implicit designs and estimands conform either to
intuition or to existing theoretical results. For instance, the implicit design of
a outcome-on-treatment-and-covariates regression is a stratified randomized experiment,
whose treatment assignment probability is linear in the covariates <cit.>.
The implicit estimand of this regression, with multivalued treatments, turns out to suffer
the same contamination bias as shown in <cit.>. In panel
settings with staggered adoption, the two-way fixed effect regression has implicit design
equal to randomization of treatment timing <cit.>.

These results supply converses to these existing intuition and results. Namely, not only
do these regressions have design-based interpretations under these treatment assignment
processes, but these processes are the only such processes consistent with a causal
interpretation and the population Gram matrix of the regression estimator. Moreover, our
framework uncovers new results as well. We find that a number of common regression
specifications do not have design-based interpretations, unless certain knife-edge
conditions hold.

An advantage of our approach is that it is extremely widely applicable. The potential
weights of any regression can be estimated by simple formulae—and some potential weights
are already calculated when one calculates the OLS estimator. We hope these potential
weights and the corresponding implicit design and implicit estimand are useful to
practitioners, and we conclude the paper by discussing a few ways these objects can be
used.



 §.§ A diagnostic for whether a design-based interpretation is possible
 First,
whether a plausible implicit design exists in the first place provides a binary diagnostic
for the design-based interpretation of a regression specification, especially for those
not covered by existing theoretical results. Finding a plausible implicit design that
generates a sensible implicit estimand shows that there is some treatment
assignment mechanism under which the regression specification is sensible, regardless of
potential outcomes.[Even if a proper implicit design exists, it might not be
plausible. Perhaps it is poorly calibrated—meaning that those with treatment assignment
probability approximately p do not appear to receive treatment with proportion p. Or
perhaps the implicit design relates to the covariates x_i in a way that is not
economically plausible.] If the researcher believes that the implicit design accurately
describes treatment assignment probabilities, then they can rest assured that the
regression estimator would not suffer a “negative weights” problem.

On the other hand, if one cannot find an implicit design that generates a sensible
implicit estimand, then this calls for either defending outcome-modeling assumptions or
for more explicitly modeling the treatment assignment process instead. Regression
specifications are naturally interpreted as outcome models (e.g., in a cross-sectional
context, we may model [Y(w) | X=x] = z(x, w)). Though these models often build in
implicit assumptions about treatment effect homogeneity and other functional form
assumptions, they nevertheless can be useful and plausible. One way forward is thus
explicitly stating the regression specification as structural assumptions on the joint
distribution of potential outcomes <cit.>.

If the researcher is nevertheless confident in a design-based interpretation free of
outcome modeling, then one way forward is to abandon the regression altogether and use
methods that put design first. These methods include weighting, matching, balancing, and
doubly robust approaches that have been recently influential in the causal inference
literature. Of course, these methods in turn require an explicit model of the design and
often require explicit estimation of the relevant propensity scores.

Here, the implicit design might guide and anchor our modeling choices. Perhaps the
researcher, by choosing the regression specification considered, is encoding some
substantive knowledge of the design. In that case, we may have reason to suspect that the
implicit design of the regression is “close” to the true design, despite the implicit
design being implausible (e.g. containing probabilities that lie outside of [0,1]). In
that case, the implicit design might serve as a reasonable starting point for modeling the
treatment assignment process. Perhaps a simple way forward is to simply re-calibrate the
implicit design. Practically, for a binary treatment, we can discretize the values of
π̂_i into bins, and for an observation i in a given bin, recompute π̃_i
∈ [0,1] as the observed frequency of treatment within the bin.[π̃_i
can be viewed as calibrated versions of π̂_i via histogram binning. We can likewise
consider other methods from the machine learning literature on calibrating binary
classifiers <cit.>. This procedure can also be viewed as blocking on an
estimated propensity score <cit.> where the estimated
propensity score is the implicit design of a regression specification.] One can then
estimate causal effects by, say, inverse propensity weighting with π̃_i. Such a
procedure would admit a design-based interpretation, and it is in a sense close to the
original regression specification.



 §.§ Assessing whether the implicit design is reasonable


Suppose an implicit design—as well as the associated estimand—does exist and the
regression passes the basic diagnostic. Having the implicit design means researchers can
inspect it and check whether they conform to economic intuition. One way is to check
whether the relationship between π̂_i and covariates x_i—preferably those
excluded from the regression specification—conform to economic intuition. Another is to
check whether the π̂_i are well calibrated, by comparing observed values W_i
against π̂_i, expecting that [W_i |π̂_i] ≈π̂_i. If
π̂_i are poorly calibrated, then the regression specification is correspondingly
suspect.



 §.§ Design-based inference


Finally, if we are willing to take the regression specification—and therefore the
implicit design—seriously, then we might consider conducting design-based inference,
particularly in settings (e.g., settings where units are geographies) where the
superpopulation sampling thought experiment is difficult to rationalize
<cit.>. Design-based inference allows one to
quantify uncertainty without appealing to sampling, and sometimes without asymptotic
arguments.

Design-based inference usually assumes a known assignment process, which is often difficult
to articulate  and specify in observational studies. For instance,
<cit.> specify counterfactual assignment of Chinese high-speed rail
lines by permuting the ordering of line completion in time. <cit.>
proposes to find counterfactual locations for a spatial treatment using convolutional
neural networks. Our approach provides an additional option in observational settings to
elicit the assignment process by tethering it to a regression specification that one
contemplates using. Since the implicit design is often unique, if one uses the regression
specification and is internally consistent, then one is implicitly specifying the design.

In practice, we could consider design-based inference taking the estimated implicit design
π̂_i as the true design, ignoring the fact that π̂_i is estimated from data.
We anticipate  doing so is justifiable when the ^* is a fixed Gram design, as is true in
many examples. When it is not, we nevertheless suspect doing so to be justifiable in
large samples when  is consistent, but we leave formal justification to future
work.










ecca










§ PROOFS


*

For the first statement and the  direction, if  is a vector of causal
contrasts for the regression specification, then π_i^*(·) solves
(<ref>), and thus ^* is an implicit design. For the
⟸ direction, if ^* is an implicit design, then ^* solves
(<ref>), which means  satisfies level
irrelevance.

When ^* is an implicit design, if there is only one implicit design, then ^*
must be that design. Moreover, ^* is proper and Gram-consistent by definition.


*

For the  direction, if τ is a causal contrast, then ^* is
an implicit design. Hence 
    π_i^* ρ_i(1) + (1-π_i^*)ρ_i(0) = 0.

Since π_i^* ∈ [0,1], this implies that ρ_i(1), ρ_i(0) must weakly be on opposite
sides of zero. If one of them is nonzero, then π_i^* = -ρ_i(0)/ρ_i(1) -
ρ_i(0). On the other hand, for the ⟸ direction, suppose
ρ_i(1)ρ_i(0) ≤ 0 and π_i^* = -ρ_i(0)/ρ_i(1) -
ρ_i(0)∈ [0,1] for all i where one of ρ_i(1), ρ_i(0) is nonzero. Then 
    π_i^* ρ_i(1) + (1-π_i^*)ρ_i(0) = 0

for all i where one of ρ_i(1), ρ_i(0) is nonzero by construction. For other
units, the
condition holds trivially since both potential weights are zero. Thus τ satisfies level irrelevance under π^*.

Note that, when this happens, ω_i(^*, 0) = (1-π_i^*) ρ_i(0) = -π_i^*
ρ_i(1). This proves the representation of τ. Moreover, 
    π_i^*
        ρ_i(1) < 0 ρ_i(1) < 0  and ρ_i(0) ≠ 0 ρ_i(1) < 0 < ρ_i(0
        )

since ρ_i(1)ρ_i(0) ≤ 0.


*


    Fix a coordinate j,k.
Let S_n = ∑_i=1^n G_i,jk and note that Ĝ_n,jk = 1/n S_n. Define μ_n =
[S_n] and σ_n^2 = (S_n). Theorem 2.2.6 in <cit.>
states that if σ_n^2 / n^2 → 0 then 
    S_n - μ_n/n = Ĝ_n,jk - G_n,jk 0.

Note that 
    σ_n^2 ≤∑_i=1^n [G_i,jk^2].

Thus the condition that 1/n^2∑_i [G_i,jk^2] → 0 is sufficient for
σ_n^2 / n^2 → 0. Thus Theorem 2.2.6 applies and Ĝ_n,jk - G_n,jk 0
for every entry. Since there are finitely many entries, Ĝ_n - G_n 0.


*

    Take η > 0. For the result with fixed i, it suffices to show that [_i() -
    _i()_F > η] → 0.
Note that 
    _i() - _i()_F > η   ≤[λ_min(Ĝ_n) ≤ϵ/2] 
       + Λ (Ĝ_n^-1 - G_n^-1) (_i,
        )'_F > η,
    λ_min(Ĝ_n) > ϵ/2.

The first term converges to zero by <ref> and the
Hoffman–Wielandt inequality. Note that, by the submultiplicativity of the Frobenius norm,
when λ_min(Ĝ_n) > ϵ/2, 
    Ĝ_n^-1 - G_n^-1_F = G_n^-1_F G_n - Ĝ_n_F Ĝ_n^-1_F ≤2/ϵ^2G_n - Ĝ_n_F  0

Thus, for some C dependent on ϵ and the bound on z_t(_i, )_∞, 
    Λ (Ĝ_n^-1 - G_n^-1) (_i, )'_F > η,
    λ_min(Ĝ_n) > ϵ/2≤C Ĝ_n^-1 - G_n^-1_F >
    η→ 0.

Therefore _i() - _i() → 0 for every fixed i.

For the uniformity in i, note that, if λ_min(Ĝ_n) > ϵ/2, up to constants,
    max_i |_i() - _i()|_∞   ≲max_i _i() -
        _i()_F 
       ≲max_i (_i, )_F  ·Ĝ_n^-1 -
        G_n^-1_F 
       ≲Ĝ_n^-1 -
        G_n^-1_F ·max_i max_t z_t(_i, )_∞
       ≲Ĝ_n^-1 - G_n^-1_F.

Therefore,
    
    max_i |_i() - _i()|_∞ > η   ≤[λ_min(Ĝ_n) ≤ϵ/2] + Ĝ_n^-1 - G_n^-1_F ≳η→ 0.



*


Fix M > 0 and sequence a_n > 0. We  compute 
    a_n - __∞ > M
        ≤[λ_min(Ĝ_n) ≤ϵ/2] + 
        a_n - __∞ > M, λ_min(Ĝ_n) >
        ϵ/2.


Note that if λ_min(Ĝ_n) > ϵ/2, then, since the Frobenius norm is
submultiplicative,

    - __2    ≤Λ_F Ĝ_n^-1 - G_n^-1_F ·[]1/n∑_i=1^n
        (_i, _i)'_i_F_O(1)  by assumption
       ≤2/ϵ^2 O(Ĝ_n - G_n_F) 
       = O|Ĝ_n-  G_n|_∞.

Thus, for some C, 
    a_n - __∞ > M
        ≤[λ_min(Ĝ_n) ≤ϵ/2] +  C a_n |Ĝ_n - G_n|_∞ >
        M.

    Take a_n = b_n^-1, we have that for every η > 0, there exists M such that
    the right-hand side is bounded by η for all sufficiently large n. This is
    equivalent to
     
    - __∞ = O_P(b_n).



*

    Fix η > 0.
Note that 
    []∑_∈_i() π̂_i()
            _∞ > η   ≤[ a bounded implicit design does not exist ] 
        + []∑_∈_i() π̂_i()
            _∞ > η, a bounded implicit design exists

Note that if π̂_i is a bounded implicit design, then 
    []∑_∈_i() π̂_i()
        _∞ = []∑_∈ (_i() - _i()) π̂_i()
        _∞≤ C (J+1) max_∈ |_i() - _i()|_∞.

Thus, 
    []∑_∈_i() π̂_i()
            _∞ > η   ≤[ a bounded implicit design does not exist ] 
       + 
            C (J+1) max_∈ |_i() - _i()|_∞ > η→ 0.

by assumption. The claim of uniform-in-i convergence follows by essentially the same
argument, where we take maximum over i.


*

Let R̂_i = [_i(0),…, _i(J)] be the sample analogue of R_i. Both
R̂_i and R_i may depend on n. Assume n is sufficiently large such
that min_i ∈ [n]σ_iJ > ϵ/2 > 0. Thus, each R_i is a J ×
(J+1) matrix with rank J such that R_i π_i^* = 0. Let u_i =
π_i^*/π_i^* be the unit vector in the (one-dimensional) null space of R_i.
Note that 1/(J+1) ≤π_i^*_∞≤π_i^*≤π_i^*_1 = 1,
and thus 1'u_i = 1/π_i^*∈ [1, J+1] is bounded above and below.

Fix some C to be chosen. Let E be complement to the event that there exists a unique bounded estimated implicit
design. Note that E = ⋃_i ∈ [n] E_i, where each E_i is the complement to the
event that R̂_i π̂_i = 0 has a unique solution π̂_i with 1'π̂_i=1 where π̂_i_∞≤ C. Let σ̂_i1≥⋯≥σ̂_iJ≥ 0 be the singular values of R̂_i.
Suppose σ̂_iJ > 0, then R̂_i has full rank and the equation R̂_i
π̂_i = 0 has a unique one-dimensional space of solutions. If that space exists, let û_i be a unit vector in that space, unique up to sign. Note that if |û_i'1| >
1/C, then π̂_i = û_i / û_i'1 is a bounded estimated implicit design with
π̂_i_∞≤ C.

Note that 
    E_i ⊂σ̂_iJ≤ϵ/4∪σ̂_iJ > ϵ/4,
    |û_i'1| < 1/C.


For the event σ̂_iJ≤ϵ/4, the Hoffman–Wielandt inequality implies that 
    |σ̂_iJ - σ_iJ|^2 ≤∑_k=1^J |σ̂_ik - σ_ik|^2 ≤R_i - R̂_i_F^2,

and thus 
    σ̂_iJ≥ϵ/2 - R_i - R̂_i_F σ̂_iJ <
    ϵ/4⊂R_i - R̂_i_F > ϵ /4.


Let û_i = ĉ_i u_i + √(1-ĉ_i^2)û_⊥,i where ĉ_i, û_⊥,i are uniquely
chosen so that û_⊥,i is a unit vector orthogonal to u_i. Then 
    |û_i'1| ≥ |ĉ_i 1'u_i| - √(1-ĉ_i^2) | 1'û_⊥,i| ≥ |ĉ_i| - √(1-ĉ_i^2)√(J+1).

Now, note that 
    R_i û_i = (R_i - R̂_i) û_i≤R_i - R̂_i_F.

On the other hand, 
    R_i û_i = √(1-ĉ_i^2)R_i û_⊥,i≥√(1-ĉ_i^2)σ_iJ≥√(1-ĉ_i^2)ϵ/2,

and thus 
    1-ĉ_i^2 ≤4/ϵ^2R̂_i - R_i_F^2  |ĉ_i| ≥√(1 - 4/ϵ^2R̂_i - R_i_F^2).

Hence, 
    |û_i'1| ≥√(1 - 4/ϵ^2R̂_i - R_i_F^2) -
    2√( J+1 )/ϵR̂_i - R_i_F.

Suppose R_i - R̂_i_F < ϵ/(4√(J+1)), then 
    |û_i'1| ≥√(1 - 1/J+1)-1/2≥√(1/2) - 1/2 ≥ 0.207.

Thus, if we pick C so that 1/C < 0.207, then

    E_i ⊂σ̂_iJ < ϵ/4∪σ̂_iJ > ϵ/4,
    |û_i'1| < 1/C⊂R_i - R̂_i_F > ϵ/4 √(J+1).


As a result, 
    ⋃_i E_i≤max_i ∈ [n]R_i - R̂_i_F > ϵ/4 √(J+1)→ 0

by assumption. This proves the first part regarding the existence of a unique bounded
implicit design.

For the second part, note that when σ̂_iJ > 0, 
    π̂_i - π_i = û_i/1'û_i - u_i/1'u_i = 1'/1'û_i (u_i - û_i) û_i/1'u_i + 1/1'u_i (û_i - u_i).

Thus 
    π̂_i - π_i_2 ≤1/0.207√(J+1)u_i - û_i_2 +
    û_i - u_i_2

if R_i - R̂_i_F ≤ϵ/(4√(J+1)).

Now, 
    û_i - u_i_2    ≤ (1-ĉ_i) + √(1-ĉ_i^2)≤√(1-ĉ_i^2)√(1-ĉ_i) + √(1+ĉ_i)/√(1+ĉ_i)≤√(1-ĉ_i^2)(1+√(2)) 
       ≤2(1+√(2))/ϵR_i - R̂_i_F

Therefore, 
    π̂_i - π_i_2 ≤1/0.207 (√(J+1) + 1)
    2(1+√(2))/ϵ_M/ϵR_i - R̂_i_F.


Hence, for η > 0, 
    max_i ∈ [n]π̂_i - π_i_2 > η⊂max_i ∈ [n]R_i - R̂_i_F > maxϵ/4√(J+1), ηϵ/M

The probability on the right-hand side converges to zero by assumption. This proves the
convergence of π̂_i to π_i uniformly in i in ·_2. Since
·_∞≤·_2 in ^J+1, this concludes the proof.  















*

By <ref>, the potential weights for this regression specification are the
same as the regression specification with the covariate W_i - x_i'β_w→ x, where
β_w→ x is the population projection of W_i (and hence π_i^*) on x_i.
Thus the potential weights are 
    ρ_i(w) = V^-1w - x_i'β_w→ x.

where V > 0 by <ref>. Note that since V > 0, ρ_i(1) ≠ρ_i(0) for all i, and
thus the implicit design is unique and equal to π_i = x_i'β_w→ x. The corresponding implicit weight ω_i is then V^-1π_i(1-π_i).

Lastly, we verify that V = 1/n∑_i=1^n π_i(1-π_i): 
    0 < V = 1/n∑_i π_i (1-π_i)^2 + (1-π_i)π_i^2 = 1/n∑_i
        π_i(1-π_i).

Note that ω_i = π_i(1-π_i)/V is negative if and only if π_i ∉[0,1].


*

    By <ref>, the potential weights for this regression specification are
    equivalent to the potential weights in the specification 
    Y_i = τ (W_i - π̃_i - Γ_2' (W_i (x_i - x̅) - Γ_0 -
        Γ_1(x_i - x̅))) + ϵ_i.

    We can then readily compute that 
    ρ_i(0) = V^-1 (-π̃- Γ_2'(-Γ_0 - Γ_1(x_i-x̅)))   ρ_i(1) = V^-1 (1-π̃- Γ_2'(x_i-x̅ - Γ_0 - Γ_1(x_i-x̅))).

    where V > 0 by <ref>.

    Thus, all implicit designs must satisfy 
    π_i = -ρ_i(0)/ρ_i(1) - ρ_i(0) = π̃_i -
        Γ_2'(Γ_0 + Γ_1(x_i - x̅))/1-Γ_2'(x_i - x̅)

    for those with 1-Γ_2'(x_i - x̅) ≠ 0.

    For <ref>, τ is a causal contrast under ^* if and only if
    ^* belongs to the set of implicit designs. This is further equivalent to
    (<ref>). When π_i^* (x_i - x̅) is affine, then the
    right-hand side of (<ref>) is zero, and thus
    (<ref>) is equivalent to ^* =, meaning that
    ^* is affine in x_i. When this happens, 
    π_i = π_i^* - π_i^*
        Γ_2'(x_i - x̅) /1- Γ_2'(x_i-x̅) = π_i^*
 when 1-Γ_2'(x_i
    - x̅) ≠ 0. Computing the implicit weight in this case yields 
    ω_i    = π_i^* ρ_i(1)  
       = V^-1π_i^*(1-π_i^*) -
            Γ_2'(π_i^* (x_i - x̅) - π_i^*Γ_0 - π_i^*Γ_1(x_i - x̅)
        )
       = V^-1π_i^*(1-π_i^*) - Γ_2'(1-π_i^*)(Γ_0 + Γ_1(x_i - x̅))
        π_i^* (x_i - x̅) = Γ_0 + Γ_1(x_i - x̅)
       = V^-1π_i^*(1-π_i^*) (1-Γ_2'(x_i - x̅)) π_i^* (x_i - x̅) = Γ_0 + Γ_1(x_i - x̅).

    Hence, ω_i is proportional to π_i^*(1-π_i^*) (1-Γ_2'(x_i - x̅)).

    For <ref>, τ is a causal contrast if and only if
    (<ref>) holds. Since π_i^* is assumed to be affine, the
    left-hand side of (<ref>) is zero. Since
    π_i^*(x_i-x̅) is generically non-affine, the right-hand side is zero for every
    i if and
    only if Γ_2=0. By the definition of Γ_2 as a projection coefficient, it
    is zero if and only if 
    1/n∑_i=1^n _W_i
                (W_i - π̃_i) ·W_i (x_i - x̅) - Γ_0 - Γ_1(x_i - x̅) = 0.

    This condition is further equivalent to 
    1/n∑_i=1^n π_i^*(1-π_i^*)(x_i - x̅) = 0.

When this happens,
    since Γ_2 = 0, the implicit weight is proportional to π_i^*(1-π_i^*).




*



	
  * By <ref>, it suffices to compute potential weights for the two-way
residualized specification. That is, let 
    _i() = 3f_1()'⋮f_T()'≡()

be the covariate transform. Note that the population residual of projecting _i(_i)
on unit and time fixed effect is 
    ()    = () - 1_T 1_T'/T() - 1/n∑_i=1^n
    ∑_' ∈π_i^*(') (') + 1_T 1_T'/T1/n∑_i=1^n
    ∑_' ∈π_i^*(') (') 
       = () - 1_T 1_T'/T()  -
    ∑_' ∈π()(') + 1_T 1_T'/T∑_' ∈π(') (')

Thus the potential weights are 
    _i() =  Λ1/n∑_i=1^n
    [()'()]^-1()'

Note that 
    ∑_∈π() _i()  = Λ1/n∑_i=1^n
    [()'()]^-1∑_∈π()
    ()'_ = 0
    = 0.

Therefore  defined by π_i() = π() is a valid implicit design. It is proper by
definition. Now, let () be the within-transformed covariate transform for this regression (which includes the time
fixed effects), which does not
depend on i. Note that the Gram matrix is 
    G_n (^*) = 1/n∑_i=1^n [ ()' () ] = ∑_∈π()  ()' () = G_n().

Thus  is Gram-consistent.

The argument for the estimated implicit design is analogous. The sample residuals of
(_i) regressing on unit and time fixed effects is 
    (_i) - 1_T
    1_T'/T(_i) - ∑_' ∈π̂(') (') +   ∑_' ∈π̂(')1_T 1_T'/T(')
 where π̂() = 1/n∑_i=1^n (_i = ). The estimated potential weights are proportional to these
residuals evaluated at _i =. Thus  defined by π̂_i(· ) =
π̂(·) solves (<ref>).





  * In this case, the population residuals are

    ()
       =  - 1_T 1_T'/T  -
    ∑__1 ∈π(_1) _1 + 1_T 1_T'/T∑__1 ∈π(_1) _1.

Any implicit design π̃_i satisfies 
    ∑_∈π̃_i()  - 1_T ∑_∈π̃_i()
        1_T'/T - ∑_∈π()  + 1_T
        ∑_∈π() 1_T'/T = 0

This implies that the vector 
    ∑_∈π̃_i() - π()∈(1_T)

    However, the only vector in (1_T) and () is the zero vector by
    assumption. Thus ∑_∈π̃_i() - π() = ∑_∈, ≠ 0π̃_i() - π() = 0. Since ∖0 is a linearly independent collection of vectors, we conclude that
    π̃_i() = π() for all ≠ 0. Since both probability vectors
    sum to one, we conclude that π̃_i() = π(). Therefore, π() in
    (<ref>) is the unique implicit design. The proof for the
    uniqueness of the estimated implicit design is analogous. 






*


By <ref>, we can focus on the population regression of _i on _i -
_iβ_w → x. The potential weight vector is then 
    _i() =  -
    _iβ_w → x/1/n∑_i=1^n __i ∼π_i^*(·)(_i -
    _iβ_w → x)'(_i -
    _iβ_w → x).

where the denominator is strictly positive by <ref>.
Here, the notation _i for T× m matrices _1(), …, _n() is defined as the population residual
against unit and time fixed effects: 
    _i = _i - 1_T 1_T'_i/T - 1/n∑_j=1^n ∑_
    _j () π_j^* ()
    + 1_T 1_T'/T1/n∑_j=1^n ∑__j() π_j^*().



The condition (<ref>) is then 
    0    = ∑_∈π_i() ρ_i() 
    _i β_w→ x   =
    ∑_
    ∈π_i ()

The right-hand side is a linear combination of the columns of  and 1_T. Thus, a
necessary condition for the existence of implicit designs is that _i
β_w→ x∈(∪1_T). The left-hand side is further more 
    _i - 1/n∑_j=1^n _jβ_w→ x + c 1_T

for some scalar c. Thus, a necessary condition is furthermore 
    _i - 1/n∑_j=1^n _jβ_w→ x∈(∪1_T).




*

    For a given unit i and t ∈_i, let L_i = |_i| and let R_i() =
    ∑_s ∈_i_s. The covariate transform—where we specify the unit fixed
    effect via within transformation—is 
    z_it()' =
        [_it1,…,_itT, _t]

    where 
    _its = 
            0     s ∉_i 
    
            1-1/L_i    s = t 
    
            -1/L_i    s ∈_i ∖t

    is the demeaned time dummy and 
    _t = _t - Q_i().


    By <ref>, it suffices to analyze the potential weights of regressing
    Y_it on Ẅ_it, where 
    Ẅ_it = Ẇ_̇i̇ṫ - ∑_s = 1^T δ_s _its = Ẇ_̇i̇ṫ
        - ∑_s ∈_iδ_s _its.

    and δ_s are the population projection coefficients of Ẇ_it on
    _its. Let V ≡1/n∑_i=1^n ∑_t ∈_i[Ẅ_it^2], which is strictly positive by <ref>. The potential weights are then 
    _it() = V^-1_t - ∑_s ∈_iδ_s
        _its = V^-1_t - δ_t + 1/L_i∑_s∈_iδ_s.


    An implicit design  exists if and only if there is some π_i(·) and
    ∑_π_i() =1 such that 
    ∑_∈π_i() _it() = 0  for all i and t∈_i.

    This is further equivalent to that for all i, t∈_i, 
    ∑_∈π_i() _t - R_i()/L_i = δ_t - 1/L_i∑_s∈_iδ_s.


Suppose first that such a  exists. Fix t ∈ and consider two units i,j,
where _i ∩_j ⊃ by assumption.
Then, by (<ref>), we have that

    δ_t = ∑_∈π_i() _t - ∑_∈π_i() R_i/L_i + 1/L_i∑_s∈_iδ_s = ∑_∈π_j() _t - ∑_∈π_j() R_j/L_j + 1/L_j∑_s∈_jδ_s.

Thus, rearranging, we have for all t ∈,   
    ∑_∈, ≠ 0 (π_i() - π_j()) _t = M(i,j)

where M(i,j) does not depend on t. Viewed as linear combinations for vectors in
_, the left-hand side collecting over all t ∈ lies in the span of
_. The right-hand side lies in the span of the constant vector on ||
dimensions. By assumption, these two spans contain only the zero vector in common. Thus,
both sides are equal to zero: 
    ∑_∈, ≠ 0 (π_i() - π_j()) _t = 0  for all t ∈.

By linear independence of vectors in _, we conclude that π_i() =
π_j(). Therefore, the implicit design must be constant across units: 
    π_i() = π()

for some π(·).

Next, we show that if  exists, then it must equal to π̅(·). The fact that
δ_t are projection coefficients is equivalent to the following orthogonality
conditions holding for all t ∈ [T]: For all s ∈1,…, T, 
    0    = 1/n∑_i=1^n ∑_t ∈_i∑_∈π_i^*()
        _it_its
       = 1/n∑_i: s ∈_i∑_∈π_i^*() ∑_t ∈_i_t - R_i()/L_i - ∑_ℓ∈_iδ_ℓ_itℓ_its
       = 1/n∑_i: s ∈_i∑_∈π_i^*() ∑_t ∈_i_t - R_i()/L_i_its - 1/n∑_i: s ∈_i∑_t ∈_i∑_ℓ∈_iδ_ℓ_itℓ_its
       = 1/n∑_i: s ∈_i∑_∈π_i^*() ∑_t ∈_i_t - R_i()/L_i(t=s) - 1/n∑_i: s ∈_i∑_t ∈_i∑_ℓ∈_iδ_ℓ(t=ℓ) _its and
        _i sum
        to zero over t∈_i
       = 1/n∑_i: s ∈_i∑_∈π_i^*() _s -
        R_i()/L_i
        - 1/n∑_i: s ∈_i∑_ℓ∈_iδ_ℓ_iℓ s
       = 1/n∑_i: s ∈_i∑_∈π_i^*() _s -
        R_i()/L_i
        - 1/n∑_i: s ∈_i∑_ℓ∈_iδ_ℓ ((ℓ = s) - 1/L_i)

    Rearranging, we have that the orthogonality condition is equivalent to 
    1/n∑_i: s ∈_i∑_∈π_i^*() _s -
            Q_i() = 1/n∑_i: s∈_iδ_s - 1/L_i∑_ℓ∈_iδ_ℓ

    for all s ∈ [T].

    Pick t ∈ and consider the corresponding (<ref>).
    Note that t ∈_i for all i, and hence  
    δ_t = 1/n∑_i=1^n 1/L_i∑_ℓ∈_iδ_ℓ +
        ∑_∈1/n∑_i=1^n π_i^*()_π̅()_t - 1/n∑_i=1^n ∑_∈π_i^*() Q_i().

    Therefore, again, given (<ref>) and π_i() = π(), 
    ∑_∈, ≠ 0 (π() - π̅()) _t = M'(i)

    for some M'(i) that does not depend t ∈. By the same argument as above, we
    have that 
    π() = π̅()  for all ≠ 0π() = π̅()
         for all ∈.


    Now, we plug π_i() = π̅() into
    (<ref>), and then plug the expression into
    (<ref>). We obtain that for all t ∈ [T],
    ∑_i : t ∈_i∑_∈π_i^*() (_t - Q_i())
        = ∑_i: t ∈_i∑_∈π̅() (_t - Q_i()).

    This is (<ref>). Thus, we proved that if an implicit design exists,
    then the above condition holds and it must be uniquely equal to π̅(·).

    For the reverse direction, let us assume that (<ref>) holds.
    For a given t, set 
    δ_t = ∑_∈π̅() _t,

    and thus 
    δ_t - 1/L_i∑_ℓ∈_iδ_ℓ = ∑_∈π̅() _t - Q_i().

    This means that π_i() = π̅() satisfies
    (<ref>). Thus π̅(·) is an implicit design
    if (<ref>) defines the projection coefficients.

    Summing over i where t ∈_i:
    
    1/n∑_i: s∈_iδ_s - 1/L_i∑_ℓ∈_iδ_ℓ = 1/n∑_i: s∈_i∑_∈π̅() _t - Q_i() = 1/n∑_i: s∈_i∑_∈π_i^*() (_t - Q_i()),

where the last equality follows from (<ref>). This verifies the
orthogonality condition (<ref>) for t. Therefore, our choice
of (<ref>) does indeed equal the projection coefficients. This concludes the
proof. 

















§ ADDITIONAL RESULTS




 §.§ Invariances




    
Consider a population regression specification in the sense of
<ref>.
Partition z_t(_i, ) into z_t1(·) and z_t2(·). Suppose Λ =
[Λ_1, 0] loads solely in entries in z_t1: Λ z_t
= Λ_1 z_t1.
Let 
    Γ
     = 1/n∑_i,t∑_∈π_i^*() z_t1(_i, )
     z_t2(_i, )'1/n∑_i,t∑_∈π_i^*()
     z_t2(_i, ) z_t2(_i, )'^-1

 be the population projection matrix of z_t1 onto z_t2. Define z̃_t
 (_i, ) = z_t1(_i, ) - Γ z_t2(_i, ). Consider the regression
 specification defined by Λ_1, z̃_t(·), and 
    G̃_n = 1/n∑_i,t∑_∈π_i^*() z̃_t(_i,
     ) z̃_t(_i, )'.

 Then:
 
    
  * Γ, G̃_n are functions of the original Gram matrix G_n. Since
    G_n is assumed to be invertible, Γ is well-defined and G̃_n is
    positive definite.
    
  * The potential weights associated with the two population regression specifications
    are the same.
 



Note that we can partition 
    G_n = 1/n∑_i,t∑_∈π_i^*() z_t(_i, ) z_t(_i, )' = [ G_11 G_12; G_21 G_22 ]

conformably where G_11 is the Gram matrix associated with z_t1 and G_22 is the Gram matrix
associated with z_t2. Observe that Γ = G_12 G_22^-1 and that G̃_n
= G_11 - Γ G_21. This proves (i).

Since G_n is symmetric and positive definite by assumption,
G_11 and G_22 are invertible. By the block matrix inversion formula, 
    G_n^-1 = [                                         (G_11 - G_12 G_22^-1 G_21)^-1                           -(G_11 - G_12 G_22^-1 G_21)^-1 G_12
    G_22^-1;                           -G_22^-1 G_21 (G_11 - G_12 G_22^-1 G_21)^-1 G_22^-1 +
        G_22^-1 G_21 (G_11 - G_12 G_22^-1 G_21)^-1 G_12 G_22^-1 ].

Hence 
    Λ G_n^-1 = Λ_1(G_11 - Γ G_21)^-1   -Λ_1 (G_11
        - Γ G_21)^-1Γ

Now, the potential weights for the original regression specification are 
    _it()    = Λ_1(G_11 - Γ G_21)^-1 z_t1(_i, ) -Λ_1
        (G_11 - Γ G_21)^-1Γ z_t2(_i, ) 
       = Λ_1(G_11 - Γ
        G_21)^-1z̃_t(_i, ) 
       = Λ_1 G̃_n^-1z̃_t(_i, ).

This proves (ii).



    
    Consider a population regression specification in the sense of
<ref>.
Partition z_t(_i, ) into z_t1(·) and z_t2(·). Suppose Λ =
[Λ_1, 0] loads solely in entries in z_t1: Λ z_t
= Λ_1 z_t1. Suppose the sample Gram matrix is invertible.
Let 
    Γ̂= 1/n∑_i,t z_t1(_i, _i)
     z_t2(_i, _i)'1/n∑_i,t
     z_t2(_i, _i) z_t2(_i, _i)'^-1

 be the sample projection matrix of z_t1 onto z_t2. Define z̃_t
 (_i, ) = z_t1(_i, ) - Γ̂z_t2(_i, ). Consider the regression
 specification defined by Λ_1, z̃_t(·).
 Then the estimated potential weights associated with the two regression specifications
    are the same.



    The proof entirely follows from the proof of <ref> by setting π_i^*() =
    ( = _i).




    
    Consider a population regression specification in the sense of
    <ref>. Consider an invertible matrix M and
    another population specification defined by z̃_t (_i, ) = M z_t (_i,
    ) and Λ̃= Λ M', so that the two vectors of coefficients
    represent the same underlying contrasts. Then the potential weights associated with
    the two regression specifications are the same. The estimated potential weights are
    also the same.




    We can easily see that the second regression specification has Gram matrix G̃_n = M G_n M' (and G̃_n = M Ĝ_n M'). Thus the potential weights for the second specification are 
    _it() = Λ̃(M')^-1 G_n^-1 M^-1 M z_t(_i, ) =
        _it().

    The corresponding equalities similarly hold for the estimated potential weights.





 §.§ Consistency of Ĝ_n to G_n under rejective sampling



This subsection considers a law of large numbers under a version of sampling with
replacement <cit.>, and largely
follows <cit.>. In particular, we assume that the treatment is
binary, and  there is a sequence of unconditional probabilities
p_1, p_2,… as well as a sequence of sample sizes N_n such that Π^*_n
describes the joint distribution of W_1,…, W_n conditional on the event
∑_i=1^n W_i = N_n, where W_i ∼(p_i) independently unconditionally.
Correspondingly, let π_i,n be the probability that W_i = 1 under
Π^*_n.


    
    Write π_i(w) = π_i,n(w) and omit the n subscript.
    Consider y̅_w = 1/n∑_i (W_i = w) y_i where [y̅_w] =
    1/n∑_i π_i(w) y_i(w).
    Suppose C_n ≡∑_i=1^n π_i(w)(1-π_i(w)) →∞ as n →∞.
    Then, as n →∞ 
    (y̅_w) = (1+o(1))/nC_n/n∑_i=1^n π_i(w) (1-π_i(w))/C_n
            y_i(w) - ∑_j=1^n π_j(w) (1-π_j(w))y_j(w)/C_n^2



    This is a restatement of Theorem 6.1 in <cit.>. The notation
    y_i in the theorem corresponds to π_i(w) y_i(w) in our notation.


A sufficient condition for the variance to tend to zero is the following:

    
    For all entries k,
z_k(x_i, w) is uniformly bounded by 0 < M < ∞ and C_n(w) ≡∑_i=1^n
π_i(w)(1-π_i(w)) →∞ as n →∞.

The boundedness condition for z_k is stronger than needed. In particular, what is
needed is that the π_i(w)(1-π_i(w))-weighted variance of y_i(w) is O(1).


    
    Under rejective sampling, suppose <ref> holds, then Ĝ_n  G_n in probability.



    We prove this claim by showing all entries converge in probability, since there are
    finitely many entries. Fix some j,k. Note that 
    Ĝ_n,jk = ∑_w ∈0,11/n∑_i=1^n (W_i = w)
        z_j(x_i, w) z_k(x_i, w)_z̅_n(w).

    By <ref> and <ref> applied to z̅_n where
    y_i(w) is taken to be z_j(x_i, w)z_k(x_i, w), we have that (z̅_n(w)) → 0 as n
    →∞, since C_n(w)/n ≤ 1. Since [z̅_n(w)] = 1/n∑_i
    π_i(w) z_j(x_i, w) z_k(x_i, w), we have that 
    Ĝ_n,jk - G_n,jk = ∑_w ∈0,1z̅_n (w) - 1/n∑_i
        π_i(w) z_j(x_i, w) z_k(x_i, w) 0

    by Chebyshev's inequality.




 §.§ Uniqueness of implicit designs in cross-sections


Consider a cross-sectional setup with J+1 treatments where the potential weights are 
    ρ_i(w) = Λ̃G̃_n^-1z̃(x_i, w)

and Λ̃ is a J × K matrix with rank J. It is possible to
reparametrize the regressors (i.e. choose an invertible M such that z(·) = Mz̃(·))
such that 
    ρ_i (w) = Λ G_n^-1 z(x_i, w)

and Λ = [I_J, 0]. Without essential loss of generality, let us assume the sequence
of reparametrized specifications satisfy <ref>.

Partition z into z_1 and z_2 where z_1 is J-dimensional. By <ref>, the potential weights are further
equivalent to 
    ρ_i(w) = G_n, 1|2^-1 z_1|2(x_i, w)

where z_1|2(x_i, w) = z_1(x_i, w) - Γ'z_2(x_i, w) for population projection
coefficients Γ and 
    G_n, 1|2 = 1/n∑_i,wπ_i^*(w) z_1|2(x_i, w) z_1|2(x_i, w)'.



    
    The residualized covariate transform is non-singular in the sense that the J × (J+1) matrix 
    Z_i ≡ [z_1|2(x_i, 0), …, z_1|2(x_i, J)]

    whose columns are covariate transforms evaluated at a treatment level has minimum
    singular value greater than some η > 0, uniformly for all i.



    
    The maximum eigenvalue of G_n is bounded above by M < ∞ for all n.



    
    Under <ref>, condition
    <ref> in <ref> is satisfied for
some lower bound on the minimum (i.e. Jþ) singular value.


Note that since the potential weight matrix G_n,1|2^-1 Z_i is J × J+1 and has
J positive singular values bounded below, if there exists an implicit design, then it must be
unique.


    We first show that <ref> is sufficient to show that the minimum
    eigenvalue of G_n,1|2 is bounded away from zero and maximum bounded by M. Note that G_n,1|2 is a Schur complement of a submatrix of G_n. Since G_n,
    1|2^-1 is a principal submatrix of G_n^-1, the eigenvalues of G_n, 1|2^-1
    interlace the eigenvalues of G_n^-1 by Cauchy's interlace theorem. In particular, the spectrum of G_n, 1|2^-1
    is included in the range of the spectrum of G_n^-1. Hence λ_min(G_n,1|2) >
    ϵ by <ref> and λ_max(G_n,1|2) < M by <ref>.

    Next, it suffices to show that G_n,1|2^-1 Z_i has minimum singular value bounded
    below. Write 
    Z_i = UΣ V'
 for diagonal J× J matrix Σ with
    UU'=U'U=V'V = I_J. Similarly, write 
    G_n, 1|2^-1 = QDQ'
 for an orthogonal
    matrix Q and diagonal D. Then 
    G_n,1|2^-1 Z_i Z_i' G_n,1|2^-1 = QDQ' U
        Σ^2 U' Q D Q'.
 is a real symmetric matrix. The spectrum of this matrix is the
    same as the spectrum of 
    DQ' U Σ^2 U' Q_positive definite matrix with
        spectrum Σ^2  D.
 Since λ_min(AB) ≥λ_min(A)
    λ_min(B) for two positive definite matrices A,B,[To see this,
    note that the operator norm (largest eigenvalue) is submultiplicative AB_op≤A_opB_op. Apply this inequality to A^-1 B^-1.] we have
    that the minimum eigenvalue of G_n,1|2^-1 Z_i Z_i' G_n,1|2^-1 is bounded
    below by η^2/λ_max(G_n,1|2)^2. Thus, the minimum singular
    value of G_n,1|2^-1 Z_i is bounded below by η/M.







 §.§ Additional results for cross-sectional specifications



    
    Consider the cross-sectional specification Y_i = τ W_i + γ'x_i +
    ϵ_i and a sample (Y_i, W_i, x_i)_i=1^n. Suppose x_i includes a constant. Suppose Ĝ_n is invertible
    on this realization. Let γ̂_w→ x be the OLS coefficient of W_i on
    x_i, and let π̂_i be the OLS fitted value π̂_i = γ̂_w→
    x'x_i. Then:
    
        
  * The potential weights are 
    ρ̂_i(w) = W_i - π̂_i/1/n∑_i=1^n π̂_i(1-π̂_i)

    and 1/n∑_i=1^n π̂_i(1-π̂_i) > 0.
    
  * The estimated implicit design exists and is uniquely equal to π̂_i.
    


    (i) follows from <ref> immediately, upon verifying that 
    0 < 1/n∑_i=1^n (W_i - π̂_i)^2 = 1/n∑_i=1^n π̂_i(1-π̂_i).

    The equality follows because 
    1/n∑_i=1^n (W_i - π̂_i)^2    = 1/n∑_i=1^n W_i - 2 W_i
        π̂_i + π̂_i^2 
       = 1/n∑_i=1^n π̂_i - 2 (W_i - π̂_i)
        π̂_i - 2π̂_i^2 + π̂_i^2 1/n∑_i π̂_i =
        1/n∑_i W_i  since x_i includes a constant
       = 1/n∑_i=1^n π̂_i - π̂_i^2 1/n∑_i=1^n (W_i
        - π̂_i) π̂_i = 0 by orthogonality of residuals
       = 1/n∑_i=1^n π̂_i(1-π̂_i).


    For (ii), note that the implicit design is equal to 
    -ρ_i(0) / (ρ_i(1) - ρ_i(0)) = π̂_i

    uniquely by <ref>.



    
    Consider the specification (<ref>) where x_i is a saturated vector of
    underlying categorical covariates. That is, suppose c_i ∈0,…,L and
    x_i is a vector such that x_iℓ = (c_i = ℓ) for ℓ = 1,…, L.
    Then the only implicit design sets π_i, for c_i = ℓ, to be the mean
    π_j^* among those j with c_j = ℓ. The corresponding implicit estimand is
    the average treatment effect.


    The specification (<ref>) is equivalent to the following specification: 
    Y_i = ∑_ℓ=0^L α_ℓ + τ_ℓ W_i

    where τ = 1-∑_ℓ=1^L x̅_ℓτ_0 + ∑_ℓ=1^Lx̅_ℓτ_ℓ≡∑_ℓ = 0^Lx̅_ℓτ_ℓ.

    For this specification, let us order the covariates to be 
    [(c_i = 0), (c_i=0) W_i, (c_i=1), (c_i=1)W_i, …, (c_i=L), (c_i=L)W_i].

    Note that the population Gram matrix is of the form 
    G_n = [ G_0;     G_1;       ⋱;     G_L ]

    where 
    G_ℓ = [     x̅_ℓ x̅_ℓπ(ℓ); x̅_ℓπ(ℓ) x̅_ℓπ(ℓ) ]

    where π(ℓ) is the mean of π_i^* among those with c_i = ℓ. Since G_n
    is block-diagonal, its inverse is similarly block-diagonal with 
    G_ℓ^-1 =
        1/x̅_ℓπ(ℓ)(1-π(ℓ)) [  π(ℓ) -π(ℓ); -π(ℓ)     1 ]
 on the diagonal.
    Note that λ = [0, x̅_0, 0, x̅_1, …, 0, x̅_L]'. Thus 
    λ'G_n^-1 = -π(0)/π(0) (1-π(0)), 1/π(0)(1-π(0)),
        …, -π(L)/π(L)(1-π(L)), 1/π(L)(1-π(L)).

    For someone with c_i = ℓ, the covariate transform is of the form 
    z(x_i, w) = [0,…, 0, 1, w, 0,…,0]'

    where it is nonzero at the ℓþ pair. Thus 
    ρ_i(w) = λ'G_n^-1 z(x_i,w) = w - π(ℓ)/π(ℓ)(1-π(ℓ)).


    By <ref>, the only implicit design sets π_i = π(ℓ) for c_i
    = ℓ. Note that the implicit design is described by ω_i = π(ℓ) ρ_i(1)
    = 1. Therefore the implicit estimand is the average treatment effect.




 §.§ Additional results for panel specifications



    
    Consider the regression Y_it = α_i + τ W_it + ϵ_it. If 
    satisfies staggered adoption without an always treated unit, then the set of implicit
    designs is empty.



    Let =  - 1_T 1_T'/T.  Then the potential weights are equal
    to 
    ρ_i(w) = /1/n∑_i=1^n ∑_∈π_i^*() ' .

    Thus, if π_i corresponds to some implicit design , 
    ∑_∈π_i()  - 1_T 1_T'/T = 0.

    In particular, this implies that 1_T is in the linear span of . By assumption,
    this is not the case. Therefore the set of implicit designs must be empty.


Intuitively, this is because one-way fixed effects fail to account for the case where the
treatment path correlates with underlying potential outcomes. If there are no treatment
effects whatsoever, but the baseline potential outcome correlates with the treatment path
(e.g., later potential outcomes tend to be larger in staggered adoption), then one-way
fixed effects would in general estimate a nonzero coefficient, violating level
irrelevance.

These results contrast with the results in
<cit.>, where the one-way fixed effect regression admits a
design-based interpretation under within-unit random assignment of W_it. To reconcile,
<cit.> consider a sampling-based setup and the unconfoundedness
restriction 
    W_it (Y_it(0), Y_it(1)) |1/T∑_s=1^T W_is,

which is their (2.2) in our notation. In staggered adoption, however, the associated
propensity score W_it=1 |1/T∑_s=1^T W_is is always degenerate, since 1/T∑_s=1^T W_is perfectly distinguishes
which treatment path unit i is assigned. Conversely, if W_it is randomly assigned
within a unit and that permuting the time index results in valid counterfactual
assignments, then  is large enough to contain 1_T.




    
    Consider the TWFE specification Y_it = α_i + γ_t + τ W_it under
    staggered adoption. Suppose the treatment time is randomly assigned so that
    π_i^*() = π^*().
    If there are two treatment paths with one being never-treated, = 0,,
    then _it() ≥ 0 for all post treatment periods _t = 1 and i ∈
    [n]. Otherwise, in all other configurations where || ≥ 2, there exists a
    choice of π^*() such that _it() < 0 for some treatment path  with
    positive assignment probability (π^*( ) > 0) and some post-treatment period
    t (_t = 1).



    The potential weight can be computed in closed form. For some V > 0, 
    _it() = V^-1_t - 1'/T - ∑_∈π^*() _t + ∑_∈π^*() 1'/T.

    When = 0,, then for a post-treatment t, 
    _it() = V^-1 (1- π^*())(1-1'/T) ≥ 0.


    Otherwise, let  be the treatment path with the earliest adoption date, and
    consider t= T. By assumption,  contains a path that adopts later than . Then 
    _it() = V^-1
            (1- π^*()) (1-1'/T) - ∑_≠π^*() 1-1'/T

    Pick π^*() = 1/2 = π^*() where  adopts later than
    . Then 
    _it() = V^-1
               1/2 (1-1'/T) - 1/2 1-1'/T = 1/2 V^-1 1'(- )/T < 0

    since  adopts later than .




















 Paper Title: Operator on Operator Regression in Quantum Probability.


eqnarray
t1Theorem[section]
d1Definition[section]
c1Corollary[section]
l1Lemma[section] r1Remark[section]
e1Example[section]
p1Proposition[section]
a1Result[section]







Operator on Operator Regression in Quantum Probability
    Suprio Bhar 

	IIT Kanpur

	Department 
 of Mathematics and Statistics 

	Kanpur 208016, India

	Email: suprio@iitk.ac.in 

	Subhra Sankar Dhar 

	IIT Kanpur

	Department of Mathematics and Statistics 

	Kanpur 208106, India

	Email: subhra@iitk.ac.in

        Soumalya Joardar

        IISER Kolkata

        Department of Mathematics and Statistics

        Mohanpur 741246, India

        Email : Soumalya@iiserkol.ac.in

    
============================================================================================================================================================================================================================================================================================================================================================================================================================================



    Abstarct 

 

This article introduces operator on operator regression in quantum probability. Here in the regression model, the response and the independent variables are certain operator valued observables, and they are linearly associated with unknown scalar coefficient (denoted by β), and the error is a random operator. In the course of this study, we propose a quantum version of a  class of estimators (denoted by M estimator) of β, and the large sample behaviour of those quantum version of the estimators are derived, given the fact that the true model is also linear and the samples are observed eigenvalue pairs of the operator valued observables. 



§ INTRODUCTION

Regression analysis is one of the most well-known toolkit in statistical modelling. More specifically, regression analysis studies the “relationship" between the dependent/response variable and one or more independent/explanatory variables. This “relationship" can be linear, non-linear or even non-parametric as well (see, e.g., <cit.> and a few relevant references therein), and using various statistical techniques, one may estimate the unknown parameters involved in the “relationship". Recently, the concepts of regression have been extended to functional valued random elements also; for instance, the interested reader may look at <cit.> and a few relevant references therein regarding function on function regression. However, to the best of our knowledge, regression analysis is an area unexplored in the setting of quantum probability. This work aims to investigate various statistical issues involved in operator on operator regression in quantum probability. We first provide a literature review and then brief the research problem without much technical details in the following. 

In the context of other statistical concepts studied in the language of quantum probability theory, recently, <cit.> extended the concepts of sufficient statistic and Rao-Blackwellization in the framework of quantum probability, and a few years before, <cit.> overviewed and reviewed the problem of statistical sufficiency in the setting of quantum statistics. They provided sufficient and necessary conditions for sufficiency and established many results from traditional statistical theory in the language of noncommutative geometry, and at the same time, <cit.> derived the noncommutative version of well-known factorization theorem in traditional Statistics (see, e.g, <cit.>). 

Let us now consider a pair of operator valued observables (X, Y) (see, e.g., <cit.> for details on observables) with observed n pairs of eigen values (λ_1, μ_1), …, (λ_n, μ_n). Observe that here in the setting of quantum probability, we are observing the eigen values of (X, Y), unlike the observations on the random element (X, Y) as in traditional Statistics/classical Probability. Suppose that unobserved operators Y and X are linearly associated in true sense with scalar valued coefficient (see (<ref>)), and in the sense of regression modelling, the operator valued observables are assumed to be linear with scalar valued coefficient having some operator valued error (see (<ref>)). In this work, based on (λ_1, μ_1), …, (λ_n, μ_n), we estimate (denoted by β̂_n as an estimator) the unknown parameter β involved in the model described in (<ref>). In the course of this study, we establish the connection between the concepts in quantum probability and classical probability (see Section <ref>), and in view of this relation, we investigate the large sample properties β̂_n (see Section <ref>).  

The rest of the article is structured as follows. Section <ref> describes the preliminaries of the model in terms of quantum probability, and for this problem, a key connection between the quantum probability and classical probability is investigated in Section <ref>, and the problem is reformulated in terms of classical probability in Section <ref>. The large sample properties of the proposed estimator is studied in Section <ref>, and Section <ref> contains a few relevant concluding remarks. Finally, necessary technical details are provided in the Appendix in Section <ref>.








§ PRELIMINARIES : MODEL


Let  be a real separable Hilbert space with inner-product ·· and norm ·_. If T and S are two bounded linear operators on  related by T = β S for some non-zero real scalar β, then given any eigen-value λ of S, we have an eigen-value μ = βλ of T with the same eigen vector of unit norm. Again, given any eigen-value μ of T, we have an eigen-value λ = μ/β of S with the same eigen vector of unit norm. In light of this fact, one may expect that the linear regression model associating two operator valued observables can be represented by their corresponding eigen values. The research problem is foramlly described in the following.  

Let X and Y be two compact self-adjoint operator (on ) valued observables related by 

    Y = β_0 X,
 where β_0∈ℝ is unknown. We now consider the operator on operator regression model

    Y = β X + ϵ,
 where ϵ is a random operator valued error, and β∈ℝ is an unknown constant. In this work, our objective is to propose an estimator (denoted by β̂_n) of the unknown β based on the observed eigenvalues of (X, Y). 
The technical assumptions are stated in Section <ref>, where the large sample properties of β̂_n are studied.   


Suppose that (λ_1, μ_1), …, (λ_n, μ_n) are n pairs of observed eigen-values of (X, Y). Motivated by our earlier observation on deterministic operators T and S, we assume that each pair (λ_j, μ_j) appear with a common eigen-vector v_j of unit norm. Here, as mentioned earlier, using the observed (λ_j, μ_j)_j = 1^n, we would like to estimate the unknown parameter β. The estimation procedure of β is described in the following. 

In order to carry our the estimation of β, we first reformulate the problem, from originally stated in terms of X and Y in the quantum probability setup to their eigen-value pairs (λ, μ) in the classical probability setup. Next, using the classical probability model thus obtained the problem is reduced to a standard linear regression problem and we can then use classical statistical tools in order to estimate β. Section <ref> studies the eigen decomposition of operator valued observables X and Y, which is the key concept in reformulating the problem in the setup of classical probability. 



 §.§ Distribution of eigen decomposition of X and Y : From quantum to classical probability


Assume that X and Y are compact self-adjoint operators on  with discrete spectrum given by σ(X) and σ(Y), respectively. As per our discussion above, the eigenspaces corresponding to each feasible eigen-value pairs are the same. Consequently, we have the following spectral decompositions (see <cit.>):

    X = ∑_α∈σ(X)α P_α,    Y = ∑_α∈σ(Y)α P_α,

where the sum is countable, and P_α denotes the projection onto the eigen-space of any eigen-value α. Here, the eigenvalues are all real and the eigen-spaces corresponding to 0 ≠α∈σ(X) are finite-dimensional. Now, given any state ϕ (unit trace), we have the expectations in the setting of quantum probability as



    E X = tr(ϕ X) = ∑_α∈σ(X)α  tr(ϕ P_α),    E Y = tr(ϕ Y) = ∑_α∈σ(Y)α  tr (ϕ P_α).


Note that tr(ϕ P_α) ≥ 0, ∀α∈σ(X). Moreover, ∑_α∈σ(X) tr(ϕ P_α) = tr(ρ) = 1. A similar statement is true when X is replaced by Y. 


Observe that one can think of λ_j's drawn independently from a discrete probability distribution supported on σ(X) and with the probability mass function 
    f_λ(α) = tr(ϕ P_α) 1_(α∈σ(X)).
 Similarly, μ_j's are drawn independently from a discrete probability distribution supported on σ(Y) and with the probability mass function 
    f_μ(α) = tr(ϕ P_α)1_(α∈σ(Y)).
 This fact enables us to reformulate the problem in the set up of traditional statistics.




 §.§ A reformulation : estimation of β


Given n pairs (λ_1, μ_1), …, (λ_n, μ_n) of observed eigen-values of (X, Y), assume that v_1, v_2, ⋯, v_n be the corresponding eigen-vectors of unit norm. Then,

    X v_j = λ_j v_j ,
 where for j = 1, …, n,  ||v_j||_ = 1, and similarly, we have 

    Y v_j = μ_j v_j .

Next, it follows from (<ref>), (<ref>) and (<ref>) that for all j = 1, …, n, Y v_j = β X v_j + ϵ v_j, which yields

    μ_jv_j = βλ_jv_j + ϵ v_j,








and consequently,

    μ_j = v_jμ_jv_j = v_jβλ_jv_j + ϵ v_j = βλ_j + v_jϵ v_j, ∀ j = 1, …, n.


We now estimate β, involved in (<ref>), based on (λ_j, μ_j)_j = 1^n in various ways, by assuming conditions on the error terms v_jϵ v_j,  j = 1, …, n. Now, recall Remark <ref> that λ_j's are i.i.d. random variables following a discrete probability distribution supported on σ(X) and with the probability mass function 
    f_λ(α) = tr(ϕ P_α) 1_(α∈σ(X)),
 and μ_j's are i.i.d. random variables following a discrete probability distribution supported on σ(Y) and with the probability mass function 
    f_μ(α) = tr(ϕ P_α)1_(α∈σ(Y)).


 For the data (λ_j, μ_j) (j = 1, …, n), the M-estimator of β (see (<ref>)), which is denoted by β̂_n, is defined as 

    β̂_n   =    argmin_β∑_j = 1^nρ(v_jϵ v_j)
        =     argmin_β∑_j = 1^nρ(μ_j - βλ_j) ,
 where ρ(.) is a certain convex function, and few more assumptions will be stated in the subsequent section. Observe that for ρ(x) = x^2, β̂_n coincides with the well-known least squares estimator (see, e.g., <cit.>, pp. 42–43) based on the data (λ_j, μ_j)_j = 1^n, and when ρ (x) = |x|, β̂_n coincides with the well-known least absolute deviation estimator (see, e.g., <cit.>, pp. 42–43) based on the data (λ_j, μ_j)_j = 1^n. Besides, it also includes Huber's estimator with ρ(x) = x^21_(|x|≤ c)/2 + (c|x| - c^2/2)1_(|x| > c), where c> 0 (see, e.g., <cit.>, p. 43), L^q regression estimate with ρ(x) = |x|^q, where q∈ [1, 2] (see, e.g., <cit.>) and well-known α-th regression quantiles with ρ(x) = |x| + (2α - 1) x, where α∈ (0, 1) (see, e.g., <cit.>, p. 43, equation (5.5)).  This is one of the reasons to consider the technique of M-estimation, which includes many well-known estimators of the unknown regression parameter. 



§ LARGE SAMPLE PROPERTIES OF Β̂_N


In order to study the large sample properties of β̂_n defined in (<ref>),  one needs to assume the following technical conditions. 

(A1) ρ(.) is a strictly convex function. 

(A2) The error random variables v_jϵ v_j (j = 1, …, n) are i.i.d., and the common distribution of v_jϵ v_j, which is denoted by F (depends on joint distribution of (λ, μ)), is such that F (D) = 0. Here D is the set of discontinuity points of ρ'(.), where ρ'(.) denotes the derivative of ρ(.).

(A3) E[ρ' (v_1ϵ v_1 + c)] = ac + o(|c|) as |c|→ 0. Here a and c are two some constants. 

(A4) g(c) : = E[ρ' (v_1ϵ v_1 + c) - ρ' (v_1ϵ v_1)] exists for some c∈ (-δ, δ), where δ > 0 is a some constant. Moreover, g(.) is a continuous function at 0.

(A5) E[{ρ' (v_1ϵ v_1)}^2] : = D > 0.

(A6) S_n : = ∑_j = 1^nλ_j^2 > 0 for some n≥ n_0 and 

    d_n^2 : = max_1≤ j ≤ nλ_j^2/S_n→ 0 as  n→∞ .



Condition (A1) is required to have unique minima of the equation described in (<ref>), and it is a common assumption across the literature of M-estimation (see, e.g., <cit.>) in traditional Statistics. Condition (A2) indicates certain smoothness of the function ρ along with the fact that the error terms (in the forms of inner product) are i.i.d. random variables. Conditions (A3) and (A4) imply the existence of the derivative of the criterion function of M estimation in some neighbourhood of 0, which is necessary to solve minimization problem involved in M estimation. Next, Condition (A5) imply that the M-estimator β̂ (see (<ref>)), after appropriate normalization, will weakly converge to a non-degenerate random variable (see the statement of Theorem <ref> below, and finally, Condition (A6) is required to have the asymptotic normality of β̂_n, and from statistical point of view, one can interpret as the variation explained by any eigen value will not be a dominating factor.


Now, we state the consistency and the asymptotic normality results associated with 
β̂_n.


Under (A1)–(A6), β̂_np→β as n→∞. Here 
p→ denotes the convergence in probability, and β̂_n and β are the same as defined in (<ref>) and (<ref>), respectively.  



Under (A1)–(A6), 
    T_n^-1/2K_n (β̂_n - β)w→ Z,
 where Z is a random variable associated with standard normal distribution, and w→ denotes the convergence in distribution. Here T_n = D S_n and K_n = a S_n, where D, a and S_n are the same as defined in (A5), (A3) and (A6), respectively. 



In the set up of quantum probability, the observations (λ_j, μ_j)_j = 1^n are n eigen value pairs of (X, Y) unlike the classical probability, where the observations are (X_j, Y_j)_j = 1^n obtained from the distribution of (X, Y). Despite this issue, Theorems <ref> and <ref> assert that β̂_n is a consistent estimator of β, and after appropriate normalization, (β̂_n - β) converges weakly to a standard normal random variable. 




















 


§ CONCLUDING REMARKS

In this work, observe that the true model (see (<ref>)) is also linear with scalar valued coefficient, i.e., in other words, we work on specified model. However, the study will be entirely different if the model becomes mis-specified, which is of great interest in many statistical problem (see, e.g., <cit.> and a few relevant references therein). Investigating this issue may be of interest to future research. Besides, we have assumed our operators to be compact and self-adjoint. The classical analog would be to assume that the real random variables are discrete. It will be of interest when the involved operators are just self-adjoint. Then the spectrum may be continuous and in this case, our technique would not suffice. Moreover, in this work as the parameter β is scalar, the operators involved here are apriori simultaneously disgonalizable and therefore they can be measured simultaneously.  



In the course of this study, it is assumed that the error random variables v_jϵ v_j (j = 1, …, n) involved in (<ref>) are i.i.d., and the assertions in the main results (i.e., Theorems <ref> and <ref>) rely on this assumption. However, this assumption can be weakened by following the approach considered in <cit.>. In this work, we did not dig in this issue as establishing the large sample properties of β̂_n is not the main theme of the work, and therefore, we kept the assumption simpler so that the readers can appreciate the notion on quantum probability conveniently for operator on operator regression.  

Acknowledgement: Subhra Sankar Dhar gratefully acknowledges his core research grant (CRG/2022/001489), Government of India. Suprio Bhar gratefully acknowledges the Matrics grant MTR/2021/000517 from the Science and Engineering Research Board (Department of Science & Technology, Government of India). Soumalya Joardar acknowledges support from the SERB MATRICS grant (MTR 2022/000515). 



apalike 




§ APPENDIX : TECHNICAL DETAILS


Proof of Theorem <ref>: The arguments in the proof are parallel to the proof of Theorem 2.2 in <cit.>. Without loss of generality, we assume that β = 0. It is now enough to show that 

    P[|β̂_n| ≥ a_n]→ 0
 as n→∞, where {a_n}_n≥ 1 is such that a_n→∞ as n→∞. Observe that it follows from (3.11) in <cit.> that there exists a sequence a_n^' such that a_n^'→∞ as n→∞ and a_n^'≤ a_n for all n∈ℕ. Moreover, in view of it, we also have 

    sup_|β|≤ a_n^'|∑_j = 1^n[ρ(v_jϵ v_j - βλ_j) - ρ(v_jϵ v_j) + βλ_jρ'(v_jϵ v_j)] - β^2K_n/2|p→ 0
 as n→∞, where K_n is the same as defined in the statement of Theorem <ref>. 

Further, when |β| = a_n^' for some n∈ℕ, then observe that 

    β^2K_n/2≥a_n^'2max_1≤ j≤ nλ_j/2,
 which follows from the definition of of K_n defined in the statement of Theorem <ref> and S_n defined in (A6) before Remark <ref>. Moreover, observe that using (A4) on (<ref>), we have 

    ∑_j = 1^nλ_jρ'(v_jϵ v_j) = O_p(1) ,
 and hence, 

    sup_|β|≤ a_n^'|β∑_j = 1^nλ_jρ'(v_jϵ v_j)| = O_p(a_n^').


Therefore, using (<ref>), (<ref>), (<ref>) and (<ref>), we have 

    P(inf_|β| = a_n^'∑_j = 1^nρ(v_jϵ v_j - βλ_j) - ρ(v_jϵ v_j) ≤ 0)→ 0
 as n→∞, and next, using (A1), we have 

    P(inf_|β| ≥ a_n^'∑_j = 1^nρ(v_jϵ v_j - βλ_j) - ρ(v_jϵ v_j) ≤ 0)→ 0
 as n→∞. 

Finally, by the definition of β̂_n in (<ref>), we have 

    P(|β̂_n|≥ a_n^')→ 0
 as n→∞. As a_n^'≤ a_n for all n∈ℕ, the proof follows.  


Proof of Theorem <ref>: Without loss of generality, we here consider β = 0. First of all, using (A6) and in view of Lindeberg Feller CLT (see <cit.>), we have 

    ∑_j = 1^nλ_jρ'(v_jϵ v_j) : = K_nβ_nw→ Z,
 as n→∞, where Z is a random variable associated with standard normal distribution, K_n is the same as defined in the statement of Theorem <ref>, and `w→' denotes weak convergence.
 Hence, in order to prove this theorem, it is enough to show that 

    β̂_n -β_np→ 0
 as n→∞, 
where β_n and β̂_n are the same as defined in (<ref>) and (<ref>), respectively. Observe that using (<ref>) and the definition of β_n in (<ref>), we 
have 

    ∑_j = 1^n[ρ(v_jϵ v_j - β_nλ_j) - ρ(v_jϵ v_j)] + ββ_n K_n/2p→ 0
 as n→∞, and for some sequence {b_n}_n≥ 1 such that b_n→∞ as n→∞ and for some arbitrary constant δ > 0, we have 

    sup_|β|≤ b_n + δ|∑_j = 1^n[ρ(v_jϵ v_j - βλ_j) - ρ(v_jϵ v_j)] + ββ_n K_n - β^2K_n/2|p→ 0
 as n→∞. Now, it follows from (<ref>) and (<ref>), we have 

    sup_|β - β_n| = δ|∑_j = 1^n[ρ(v_jϵ v_j - βλ_j) - ρ(v_jϵ v_j - β_nλ_j)]  - (β - β_n)^2K_n/2|p→ 0
 as n→∞. Moreover, observe that when |β - β_n| = δ, we have 

    (β - β_n)^2K_n≥(min_1≤ j ≤ nλ_j)δ^2.
 Therefore, using (<ref>) and (<ref>), we have 

    P(|β - β_n|> δ)→ 0
 as n→∞, where δ > 0 is an arbitrary constant. It completes the proof. 


