#  TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs

Cheng Wang${}^{\textbf{1}}$, Xinyang Lu${}^{\textbf{1}\textbf{2}}$, See-Kiong
Ng${}^{\textbf{1}\textbf{2}}$, Bryan Kian Hsiang Low${}^{\textbf{1}}$
1Department of Computer Science, National University of Singapore
2Institute of Data Science, National University of Singapore
{wangcheng<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

The rapid evolution of _large language models_ (LLMs) represents a substantial
leap forward in natural language understanding and generation. However,
alongside these advancements come significant challenges related to the
accountability and transparency of LLM responses. Reliable source attribution
is essential to adhering to stringent legal and regulatory standards,
including those set forth by the General Data Protection Regulation. Despite
the well-established methods in source attribution within the computer vision
domain, the application of robust attribution frameworks to natural language
processing remains underexplored. To bridge this gap, we propose a novel and
versatile TRansformer-based Attribution framework using Contrastive Embeddings
called TRACE that, in particular, exploits contrastive learning for source
attribution. We perform an extensive empirical evaluation to demonstrate the
performance and efficiency of TRACE in various settings and show that TRACE
significantly improves the ability to attribute sources accurately, making it
a valuable tool for enhancing the reliability and trustworthiness of LLMs.

TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs

Cheng Wang${}^{\textbf{1}}$, Xinyang Lu${}^{\textbf{1}\textbf{2}}$, See-Kiong
Ng${}^{\textbf{1}\textbf{2}}$, Bryan Kian Hsiang Low${}^{\textbf{1}}$
1Department of Computer Science, National University of Singapore 2Institute
of Data Science, National University of Singapore {wangcheng,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

The recent era has seen a significant rise in the prevalence of _large
language models_ (LLMs) (Ouyang et al., 2022; Touvron et al., 2023) which have
demonstrated an array of remarkable capabilities. However, studies (Huang et
al., 2023; Liu et al., 2024; Wang et al., 2023a) have highlighted a critical
concern on the accountability of LLMs. Considering the widespread usage and
such a concern, it has brought to the forefront a critical need for source
attribution that involves identifying the specific training data that
contributes to generating part or all of an LLM’s response, which is crucial
for legal and regulatory compliance and enhances the reliability of LLMs.
Various regulations mandate transparency and accountability in data usage,
especially regarding intellectual property and privacy. For instance, the
General Data Protection Regulation (GDPR) in the European Union requires that
individuals have the right to be informed when their personal data is used.
Proper source attribution ensures compliance with such legal frameworks,
mitigating the risk of legal disputes and penalties.

A related topic would be that of _membership inference_ (MI) (Mireshghallah et
al., 2022) whose task is to determine whether a given piece of data was used
during the training of a machine learning model. While MI and source
attribution share some similarities, they differ significantly in their
granularity: MI typically only involves a binary classification task and does
not require identifying a specific data provider. In contrast, source
attribution requires to identify one or more data providers.

Though there are some studies on source attribution (Marra et al., 2018; Yu et
al., 2022), a majority of them are situated within the computer vision domain.
Techniques developed for computer vision tasks cannot be directly applied to
LLMs due to the fundamental differences in the data and model architectures.
To the best of our knowledge, effective source attribution for LLMs still
remains an open and underexplored problem.

While numerous properties are important to a source attribution framework, we
identify accuracy, scalability, interpretability, and robustness as the most
crucial components. These four attributes are fundamental to ensuring the
effectiveness and applicability of the framework across various contexts.
Accuracy is essential to guaranteeing that the framework consistently produces
reliable results. Scalability ensures that the framework can handle increasing
volumes of data and complexity without a significant performance degradation,
making it suitable for large-scale applications. Interpretability is equally
critical as it enables stakeholders to understand and trust the attribution
outcomes, hence fostering transparency and facilitating informed decision
making. Robustness is vital to ensure that the framework remains reliable and
effective even in the face of adversarial distortions.

This paper presents a novel TRansformer-based Attribution framework using
Contrastive Embeddings (TRACE) to achieve source attribution while satisfying
the above three important properties. By detailing our methodology and
presenting empirical results, we seek to demonstrate the accuracy,
scalability, interpretability, and robustness of TRACE.

Our contributions can be summarized as follows:

  * •

We propose the novel TRACE framework based on contrastive learning, which is
designed to achieve effective source attribution. TRACE differs from
traditional contrastive learning by using source information as the label.
Fig. 1 illustrates the TRACE framework.

  * •

We have performed an extensive empirical evaluation of TRACE to demonstrate
its accuracy, scalability, interpretability, and robustness.

Figure 1: Illustration of TRACE framework.

##  2 Preliminaries

#### Contrastive Learning and NT-Xent Loss.

Contrastive learning is a conventional technique commonly used in
representation learning (Arora et al., 2019; Hadsell et al., 2006). Its
underlying idea is that similar objects should exhibit a closer distance in
the embedding space while dissimilar objects should repel each other. This
technique has been widely employed in computer vision tasks due to its
convenient implementation to augment image input to form a self-supervised
problem. Models using contrastive learning have achieved state-of-the-art
performances (Cui et al., 2021; Tian et al., 2020). Apart from the attention
it receives in computer vision, new approaches using contrastive learning in
natural language processing (Meng et al., 2021; Wu et al., 2020) have also
started gaining attention and showcasing great capabilities.

Our TRACE framework assigns the same label to all the data from the same
source, hence naturally forming a _supervised contrastive learning_ problem.
In particular, TRACE utilizes NT-Xent Loss (Sohn, 2016) for supervised
contrastive learning:

$\mathcal{L}=\sum_{i\in I}\frac{-1}{|P_{i}|}\sum_{p\in
P_{i}}\log\left(\frac{\exp\left(\mathbf{z}_{i}\cdot\mathbf{z}_{p}/\tau\right)}{\sum_{a\in
A_{i}}\exp\left(\mathbf{z}_{i}\cdot\mathbf{z}_{a}/\tau\right)}\right)$

where the set $I$ ($P_{i}\subset I$) contains indices of the sentences in the
given batch (sharing the same label as sentence $i$, but does not include
$i$), $A_{i}=I\setminus\\{i\\}$, $\mathbf{z}_{i}$ denotes the embedding of
sentence $i$, and $\tau\in\mathbb{R}^{+}$ is a temperature parameter.
Minimizing $\mathcal{L}$ would maximize the similarity between embeddings (of
sentences) from the same source while minimizing the similarity between
embeddings from different sources.

#### Sentence Encoder.

Similar to the concepts of Word2Vec (Mikolov et al., 2013) and GloVe (Brochier
et al., 2019) which produce meaningful vector representations of words, such
techniques can be applied to larger text units such as sentences. A
straightforward way is to take the average of word embeddings within a
sentence, but this often results in embeddings that lack semantic depth.
Several models have been developed to address this issue, including InferSent
(Conneau et al., 2018), Universal Sentence Encoder (Cer et al., 2018), and
Sentence-BERT (SBERT) (Reimers and Gurevych, 2019). Given its superior
performance and efficiency, SBERT is chosen to generate sentence embeddings in
TRACE. SBERT leverages a pre-trained BERT network and utilizes Siamese and
triplet network structures to produce semantically meaningful sentence
embeddings.

##  3 TRACE Framework

###  3.1 Source-Specific Semantic Distillation

Projecting every piece of data from each provider into the embedding space is
desirable but would incur considerable computational costs. Moreover, it is
prudent to recognize that not all information carries equal importance: For
example, sentences that occur less frequently typically tend to be more
representative of the document. So, we propose to extract principal sentences
from each source by leveraging the _Term Frequency-Inverse Document Frequency_
(TF-IDF) which is effective for identifying significant sentences within
documents. It is generally recommended to select $10$-$20\%$ of the sentences,
thereby striking a balance between complexity and performance; these sentences
are subsequently defined as _principal sentences_. The length of these
sentences is specified by a parameter called WINDOW_SIZE. Section 4.8 presents
an ablation study examining the effect of different WINDOW_SIZEs on accuracy.

SBERT (Reimers and Gurevych, 2019) has proven effective in deriving high-
quality sentence representations. However, to enhance its suitability for
TRACE, we propose several modifications inspired by the work of SimCLR (Chen
et al., 2020). A key finding from SimCLR is that adding a non-linear
projection head significantly improves the representation quality. Following
this insight, we incorporate a projection network at the end of the
traditional SBERT architecture. This projection network is trained together
with the base SBERT model, thus encouraging the learned representations to be
more discriminative in the embedding space.

###  3.2 Supervised Contrastive Embedding Training for Source-Coherent
Clustering

Unlike the other contrastive learning frameworks in computer vision whose
tasks are typically defined to be auto-regressive due to the availability of
various data augmentation techniques to generate positive samples, TRACE aims
to achieve source-coherent clustering. In our case, we already possess the
label of each sentence indicating its source. So, we can frame our task as a
_supervised contrastive learning_ problem. The supervision is derived from the
label information which corresponds to the source. Contrastive learning aligns
with our objective to form clusters based on these various sources.

SimCLR has demonstrated that NT-Xent Loss outperforms other contrastive loss
functions such as logistic loss (Mikolov et al., 2013) and margin loss
(Schroff et al., 2015). So, we employ NT-Xent Loss as the loss function for
TRACE.

###  3.3 Proximity-based Inference

Figure 2: Illustration of the attribution step in TRACE framework.

Once the training phase is completed, we transition to the inference stage
where each data source is represented by its own set of contrastive
embeddings. At this stage, when a language model generates a response, we
employ the _$k$ -Nearest Neighbor_ ($k$NN) algorithm to assign the response to
the closest data source in the embedding space, as demonstrated in Fig. 2.
This ensures accurate source attribution by matching the generated response
with its most similar source representation.

However, responses generated by language models may not always be exclusively
influenced by a single data source: there could be instances where information
from multiple sources contributes to the generated text. To consider this
possibility, we introduce the concept of _multi-source attribution_. Multi-
source attribution acknowledges and accounts for the potential influence of
multiple data sources on the generated response.

We have developed three different implementations for single-source
attribution and multi-source attribution, which allow users to select the most
appropriate inference method based on time constraints and the number of
sources. Section 4 provides a comparison of these methods.

#### Hard $k$NN (Single-Source Attribution).

_Hard $k$NN_ follows the traditional $k$NN algorithm closely. Here, the
attribution is determined by considering the $k$ embeddings that are closest
in distance to the query. The source that appears most frequently among these
$k$ neighbors is assigned as the source of the query.

#### Soft $k$NN (Multi-Source Attribution).

To differentiate from traditional $k$NN where each query is assigned to a
single source, we introduce _soft $k$NN_. Here, $k$ represents the number of
data sources rather than the number of closest neighbors. We rank the
distances from the query to all other embeddings and select them in ascending
order of distance until $k$ distinct sources are covered.

#### Nearest Centroid (Single-Source Attribution).

To reduce inference time, we utilize the nearest centroid method. The centroid
of each cluster is determined by aggregating the normalized embeddings within
that cluster (i.e., corresponding to the sentences with the same
label/source), as elaborated in Appendix A.

We then apply $k$NN using these centroids. This method significantly reduces
inference time as it scales with the number of data providers rather than the
volume of data from each source. We will demonstrate in the next section that
this method maintains an impressively high accuracy.

##  4 Experiments

###  4.1 Experimental Setup

#### Data.

We perform an extensive empirical evaluation of TRACE using three datasets:
booksum (Kryściński et al., 2022), dbpedia_14 (Zhang et al., 2015), and
cc_news (Hamborg et al., 2017); a summary of these datasets can be found in
Table 6 in the appendix. In the booksum dataset, we treat different books as
distinct data providers and vary the number of data providers from $10$, $25$,
$50$, to $100$ to demonstrate TRACE’s scalability to a large number of data
providers. Similarly, each class in dbpedia_14 or each domain in cc_news is
considered a separate data provider. In this section, we primarily present the
experimental results on the booksum dataset with $25$ data providers. Section
4.7 provides additional results.

#### Model.

Focusing primarily on the booksum dataset, we evaluate the performance of
TRACE using three different LLMs of varying sizes: t5-small-booksum (Raffel et
al., 2020), GPT-2 (Radford et al., 2019), and Llama-2 (Touvron et al., 2023).
The t5-small-booksum model is readily available on Hugging
Face,111https://huggingface.co/cnicu/t5-small-booksum. while GPT-2 and Llama-2
have been fine-tuned on a subset of the booksum dataset. This setup allows us
to assess the performance of TRACE across LLMs of different scales. In App. B,
we provide more details about our experiments.

###  4.2 Visualization of TRACE’s Embedding Space

After training for $150$ epochs on booksum, a visualization tool such as UMAP
(McInnes et al., 2020) can be used to view the distribution of principal
sentences. Fig. 3 shows that after the contrastive learning step, the desired
outcome has been achieved, i.e., data coming from the same source form clear
and distinct clusters. This validates that our contrastive learning
successfully groups different data providers. Supposing the responses from an
LLM are projected into the embedding space without incorporating the
contrastive learning step, the resulting neighborhood exhibits chaos and it is
challenging to derive robust information. This further demonstrates the
importance of the contrastive learning step.

Figure 3: Visualization (using UMAP) of the embedding space before (left) and
after (right) contrastive learning.

###  4.3 Accuracy

Evaluating the accuracy of source attribution is particularly challenging due
to the inherent difficulty in obtaining ground-truth test datasets. Even with
a dataset, a language model, and specific inputs, pinpointing the exact parts
of the training data that influence a particular response remains complex.
Here are the key reasons:

  1. 1.

Lack of Explicit Traceability. Language models like LLMs generate responses
based on patterns learned from vast amounts of data. However, these models do
not provide explicit traceability back to the specific training data. This
means we cannot directly observe which parts of the training data contribute
to a given response.

  2. 2.

Intermixed Training Data. The training data for LLMs is often a massive,
intertwined collection of texts from various sources. Disentangling these
sources to identify the precise contribution of each segment to the final
response is nearly impossible due to the sheer volume and complexity.

  3. 3.

Influence of Pre-training Data. It is also likely that the model generates
responses based on data encountered during the pre-training stage, which
comprises a vast and diverse corpus. This pre-training data is often not fully
documented or accessible, making it difficult to determine its influence on
specific responses during fine-tuning or evaluation.

Due to these challenges, obtaining ground-truth test datasets that accurately
reflect the contribution of specific training data to the responses of LLMs is
exceedingly difficult. To address this issue, our approach involves using
training data where the source is known. We then use this known source as the
ground-truth label and evaluate whether TRACE can correctly determine the
source. This allows us to approximate the evaluation of source attribution by
leveraging the known origins of the specific training data.

#### Single-Source Attribution Accuracy.

In this case, accuracy is simply defined as the number of correct source
attributions divided by the total number of attributions evaluated, the latter
of which is $250$ in our experimental setup.

#### Multi-Source Attribution Accuracy.

In certain settings, providing multiple sources and allowing the user to
determine the justification of the attribution is acceptable. For a successful
_soft $k$NN_ attribution in such cases, the ground-truth source must appear
among the _top- $k$_ sources returned by TRACE. Using the same setup as that
of single-source attribution, we have evaluated TRACE on $250$ instances.
Table 1 below shows the results:

Model | Soft $k$NN | Hard $k$NN | Nearest Centroid
---|---|---|---
acc. | top-3 acc. | top-5 acc. | $k=10$ | $k=20$
t5 | 84.4% | 95.3% | 97.3% | 84.4% | 84.4%
GPT-2 | 81.3% | 92.3% | 94.0% | 81.3% | 81.3%
Llama-2 | 86.2% | 96.1% | 97.2% | 86.2% | 86.2%

Table 1: Source attribution accuracy for $25$ data providers on booksum
dataset using TRACE.

It can be observed that the accuracy for models of different sizes remains
consistently high and significantly surpasses the random guess’ accuracy of
$4\%$. Another notable observation from the results is that varying the values
of $k$ in the hard $k$NN approach has minimal impact on accuracy and yields
results identical to that of the nearest centroid method, which we attribute
to the highly compact nature of the embeddings learned under the TRACE
framework. When a query is projected into the embedding space, it becomes
closely associated with its nearest neighbors regardless of the specific value
of $k$. This compactness suggests that the centroid of each cluster serves as
an excellent representative of the entire cluster. Consequently, relying
solely on these centroids can significantly reduce inference time. Even with
$100$ data providers as demonstrated in next subsection, the inference process
remains almost instantaneous.

n_books | t5 | GPT2 | Llama-2
---|---|---|---
acc. | top-$3$ acc. | top-$5$ acc. | acc. | top-$3$ acc. | top-$5$ acc. | acc. | top-$3$ acc. | top-$5$ acc.
10 | 87.5% | 98.3% | 99.4% | 85.3% | 96.8% | 98.7% | 88.2% | 99.2% | 99.5%
25 | 84.4% | 95.3% | 97.3% | 81.3% | 92.3% | 94.0% | 86.2% | 96.1% | 97.2%
50 | 73.1% | 82.0% | 84.0% | 72.9% | 82.9% | 84.1% | 70.3% | 79.8% | 82.2%
100 | 45.4% | 74.8% | 78.8% | 49.0% | 73.2% | 77.7% | 46.7% | 76.8% | 80.2%

Table 2: Source attribution accuracy for different no. of data providers on
booksum dataset using TRACE.

Figure 4: Contrastive loss (left) and soft $k$NN accuracy (right) with
different WINDOW_SIZEs. Note that the results for hard $k$NN (regardless of
the value of $k$) are identical to that of soft $k$NN when $k=1$.

###  4.4 Scalability

Contemporary LLMs often necessitate substantial quantities of training data
and the capability to manage a multitude of data providers. Hence, it is
imperative to demonstrate the scalability of the TRACE framework under such
settings. We assess the scalability of TRACE by selecting $10$, $25$, $50$,
and $100$ distinct books from the booksum dataset, while maintaining a
consistent experimental configuration. The results in Table 2 indicate a
diminishing trend in accuracy with an increasing number of data providers,
which is expected as the task complexity grows. However, despite this
challenge, TRACE exhibits a relatively high level of accuracy across all
settings, thus affirming its scalability.

###  4.5 Interpretability

The TRACE framework not only delivers accurate source attribution but also
provides interpretability by offering additional insights into the attribution
process. This interpretability is crucial for understanding the reasoning
behind the model’s decisions and gaining confidence in its responses. We
illustrate the interpretability of TRACE using responses from the t5-small-
booksum model as a demonstration.

Inference method | deletion | synonym substitution | paraphrasing
---|---|---|---
5% | 10% | 15% | 5% | 10% | 15%
top-1 acc. drop | ↓0.9% | ↓1.2% | ↓1.7% | ↓1.5% | ↓2.9% | ↓3.5% | ↓9.5%
top-3 acc. drop | ↓0.7% | ↓1.3% | ↓1.6% | ↓1.7% | ↓2.3% | ↓2.7% | ↓4.5%
top-5 acc. drop | ↓0.3% | ↓0.7% | ↓0.7% | ↓1.1% | ↓2.1% | ↓2.3% | ↓1.8%
Nearest Centroid drop | ↓0.9% | ↓1.2% | ↓1.7% | ↓1.5% | ↓2.9% | ↓3.5% | ↓9.5%

Table 3: Impact of various text perturbation attacks on TRACE attribution
accuracy using Llama-2 on the booksum dataset, which includes 25 distinct
books. The down arrows indicate the percentage drop in accuracy due to the
corresponding attack ratio.

Table 4 shows a summary of correctly attributed single-source responses from
the t5-small-booksum model. Each response is paired with the nearest principal
sentence from the identified source. This pairing allows users to understand
the specific evidence or context from the source text that influences the
model’s attribution decision.

Moreover, TRACE offers interpretability through the inclusion of different
similarity scores. These scores provide insights into the model’s confidence
levels regarding the attribution outcomes. By examining the similarity scores,
users can gauge the strength of the connection between the response and the
identified source.

Overall, TRACE enhances interpretability by not only delivering the final
attribution outcomes but also by providing supporting evidence from the source
text and indicating the model’s confidence levels through similarity scores.
This transparency and insight into the attribution process empower users to
trust and understand the model’s responses, which makes TRACE a valuable tool
for source attribution tasks.

Response |  Nearest Principal Sentence
---|---
Morel is in Sheffield, and he feels guilty towards Dawes, who is suffering and despairing, too. And besides, they had met in Nottingham in a way that is more or less responsible. |  on his knees, feeling so awkward in presence of big trouble. Mrs. Morel did not change much. She stayed in Sheffield
But Emma thought at least it would turn out so. Mrs. Elton was first seen at church: but although devotion might be interrupted, curiosity could not be satisfied by a bride in. Pew, and it must be left for the visits in form which were then paid, to settle whether she was very pretty indeed, or only rather pretty at all. |  or any thing just to keep my boot on.” Mr. Elton looked all happiness at this proposition; and nothing could exceed
to marry Lord Warburton. Isabel enquired. “Your uncle’s not an English nobleman,” said Mrs. Touchett in her smallest, sparest voice. The girl asked if the correspondent of the Interviewer was to take the party to London under Ralph’s escort. It was just the sort of plan, she said, that Miss Stackpole would be sure to suggest, and Isabel said that she did right to refuse him then. |  he told Ralph he’s engaged to be married.” “Ah, to be married!” Isabel mildly exclaimed. “Unless he breaks it off. He seemed

Table 4: Sample responses with correct single-source attribution from
t5-small-booksum model.

###  4.6 Robustness

In the context of adversarial scenarios, it is essential to evaluate the
robustness of TRACE against attacks wherein the attacker has access to the
model’s response. We consider attackers who can apply distortions to the
response to alter the source attribution results but have black-box access to
the model itself. This section focuses on evaluating the robustness of TRACE
under three specific types of distortions: deletion, synonym substitution, and
paraphrasing. The summarized results are presented in Table3. Appendix B
presents the attack details.

The results indicate that while deletion and synonym substitution do have some
impact on attribution accuracy, the extent of this impact is minimal. However,
paraphrasing proves to be a more potent attack. This is because paraphrasing
essentially alters the sentences to a larger extent, which causes more
influence on the source attribution results. This observation highlights the
need for future research to develop effective defense mechanisms for TRACE
against such adversarial techniques.

###  4.7 Additional Experimental Results

We conduct additional experiments to assess the performance of TRACE on
alternative datasets, thereby evaluating its versatility. Table 5 summarizes
the results. For a consistent comparison, we employ the same LLM across these
datasets.

Dataset | Data Providers | Soft $k$NN | Hard $k$NN | Nearest Centroid
---|---|---|---|---
$k=1$ | $k=3$ | $k=5$ | $k=10$ | $k=20$
booksum | 10 | 85.3% | 96.8% | 98.7% | 85.3% | 85.3%
dbpedia_14 | 10 | 88.2% | 94.1% | 97.2% | 88.2% | 88.2%
booksum | 25 | 81.3% | 92.3% | 94.0% | 81.3% | 81.3%
cc_news | 25 | 83.1% | 90.8% | 92.1% | 83.1% | 83.1%

Table 5: Source attribution accuracy on dbpedia_14 and cc_news datasets using
TRACE.

Our additional experiments affirm the adaptability of the TRACE framework
across various datasets, thereby validating its applicability across various
knowledge domains and settings.

###  4.8 Ablation Study

The most important factor in TRACE is the user-defined WINDOW_SIZE. If the
WINDOW_SIZE is too small, the principal sentences cannot capture sufficient
contextual information, hence deteriorating the performance. However, an
exceedingly large WINDOW_SIZE will not only require more computational
resources and time to train but also the meaning will be diluted by other
redundant information. This presents a natural trade-off between source
attribution performance and computational efficiency. Therefore, in this
subsection, we will analyze this trade-off and present the results in Fig. 4.

It can be observed that a larger WINDOW_SIZE facilitates faster model
convergence. However, model loss alone is not a comprehensive indicator of the
clustering quality. So, we evaluate the source attribution accuracy on the
test dataset. When the WINDOW_SIZE is set to $30$, our TRACE framework
achieves its highest accuracy. We hypothesize that this is primarily because
the WINDOW_SIZE of $30$ is sufficient to capture essential contextual
information without excessively diluting it.

##  5 Related Work

#### Source Attribution.

Though source attribution remains relatively underexplored in the domain of
natural language processing, WASA (Wang et al., 2023b) stands out as a notable
framework.222Note that neither the source code nor comprehensive details of
the experimental setup have been provided in (Wang et al., 2023b), making a
fair comparison with WASA infeasible. Operating on the principle of
watermarking, WASA embeds distinct source identifiers within the training data
to ensure that responses convey pertinent data provider information. However,
WASA necessitates extensive manipulation of training data and training the
entire LLM from scratch, which is a time-consuming process given their sizes.
In contrast, TRACE distinguishes itself by being model-agnostic, i.e.,
requiring no knowledge about the model. This characteristic enhances
efficiency and adaptability.

In the context of identifying information sources for quotes, Quobert (Vaucher
et al., 2021) is a minimally supervised framework designed for extracting and
attributing quotations from extensive news corpora. Additionally, Spangher et
al. (2023) have developed robust models for identifying and attributing
information in news articles. However, these approaches are primarily focused
on specific domains such as news. In contrast, TRACE is designed to handle
knowledge across a wide range of domains and hence provides a more generalized
and versatile solution for source attribution tasks.

#### Information Retrieval.

A related topic to our work here is information retrieval. Traditional
retrieval techniques like BM25 (Robertson et al., 1994) hinge heavily on
frequency-based rules which prove to be inadequate when dealing with responses
that share semantic similarities without significant lexical overlap. More
contemporary methods, such as ANCE (Xiong et al., 2020) and Contriever
(Izacard et al., 2022), opt for generating compact, dense representations of
documents rather than long, sparse ones. Thus, they tend to achieve better
results.

While information retrieval and TRACE both use dense representations to
measure sentence similarity, they differ in objectives and applications.
Information retrieval aims to rank relevant documents for a user’s query. In
contrast, TRACE focuses on identifying and attributing the original source of
specific information, hence ensuring accurate credit and authenticity.

#### Membership Inference Attack.

The concept of membership inference attack was first introduced by Shokri et
al. (2017). The primary objective of this attack is to ascertain whether a
specific piece of information was part of the training data for a given
machine learning model. Various assumptions about the available information
lead to different attack models. For instance, some models assume access to
hard labels (Li and Zhang, 2021), the model’s confidence scores (Watson et
al., 2022; Mattern et al., 2023), or the internal parameters of the model
(Leino and Fredrikson, 2020). Wei et al. (2024) have achieved membership
inference by inserting watermarks into data. Despite the variations, these
attacks fundamentally seek to answer a binary question, i.e., whether the
information was included in the training dataset or not.

In contrast, source attribution entails mapping the response to distinct and
specific sources rather than simply determining the presence or absence of the
data in the training set. Additionally, TRACE adheres to a black-box setting:
It does not require access to internal information such as confidence scores
or model parameters. Instead, TRACE only necessitates the response from a LLM.

##  6 Conclusion

This paper describes a novel TRACE framework which effectively achieves source
attribution. By selecting principal sentences and projecting them into the
embedding space via source-coherent contrastive learning, TRACE enhances the
interpretability of responses generated by LLMs. This enhancement also
conforms to regulations that aim to protect the privacy of users. After
evaluating TRACE on various datasets, we have demonstrated the accuracy ,
scalability, and robustness of our framework.

### Limitations

Our experiments are subject to some limitations that can be addressed in the
future work to ensure a comprehensive interpretation of results. Firstly, the
balanced distribution of data across different sources may impact the final
inference of TRACE given its reliance on the $k$NN algorithm. This uniformity
in data volume may not be representative of real-world settings, which
potentially limits the generalizability of our findings. Secondly, information
within each source is quite distinct with no overlapping data. Future works
can verify the setting where data sources contain similar information. These
limitations underscore the importance of future research in addressing such
challenges to enhance the robustness of TRACE across varied data environments.

### Ethical Considerations

Our TRACE framework introduces a method for achieving source attribution.
Utilizing this framework, a malicious actor may potentially identify the
sources of data providers and reveal sensitive information about them.
Therefore, the application of TRACE within this context necessitates
meticulous handling to mitigate privacy concerns.

## References

  * Arora et al. (2019) Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. 2019.  A theoretical analysis of contrastive unsupervised representation learning.  arXiv:1902.09229.
  * Brochier et al. (2019) Robin Brochier, Adrien Guille, and Julien Velcin. 2019.  Global vectors for node representations.  In _Proc. WWW_ , pages 2587–2593.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder.  arXiv:1803.11175.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  arXiv:2002.05709.
  * Conneau et al. (2018) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2018.  Supervised learning of universal sentence representations from natural language inference data.  arXiv:1705.02364.
  * Cui et al. (2021) Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. 2021.  Parametric contrastive learning.  In _Proc. ICCV_ , pages 715–724.
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann Lecun. 2006.  Dimensionality reduction by learning an invariant mapping.  In _Proc. CVPR_ , pages 1735–1742.
  * Hamborg et al. (2017) Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017.  news-please: A generic news crawler and extractor.  In _Proc. ISI_ , pages 218–223.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  arXiv:2311.05232.
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised dense information retrieval with contrastive learning.  arXiv:2112.09118.
  * Kryściński et al. (2022) Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022.  BookSum: A collection of datasets for long-form narrative summarization.  In _Proc. EMNLP Findings_ , pages 6536–6558.
  * Leino and Fredrikson (2020) Klas Leino and Matt Fredrikson. 2020.  Stolen memories: Leveraging model memorization for calibrated white-box membership inference.  In _Proc. SEC_ , pages 1605–1622.
  * Li and Zhang (2021) Zheng Li and Yang Zhang. 2021.  Membership leakage in label-only exposures.  arXiv:2007.15528.
  * Liu et al. (2024) Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2024.  Trustworthy LLMs: A survey and guideline for evaluating large language models’ alignment.  arXiv:2308.05374.
  * Marra et al. (2018) Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. 2018.  Do GANs leave artificial fingerprints?  arXiv:1812.11842.
  * Mattern et al. (2023) Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023.  Membership inference attacks against language models via neighbourhood comparison.  arXiv:2305.18462.
  * McInnes et al. (2020) Leland McInnes, John Healy, and James Melville. 2020.  UMAP: Uniform manifold approximation and projection for dimension reduction.  arXiv:1802.03426.
  * Meng et al. (2021) Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021.  COCO-LM: Correcting and contrasting text sequences for language model pretraining.  arXiv:2102.08473.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  arXiv:1301.3781.
  * Miller (1994) George A. Miller. 1994.  WordNet: A lexical database for English.  In _Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022.  Quantifying privacy risks of masked language models using membership inference attacks.  arXiv:2203.03929.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  arXiv:2203.02155.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  Technical report, OpenAI.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_ , 21(1):5485–5551.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  arXiv:1908.10084.
  * Robertson et al. (1994) Stephen Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994.  Okapi at TREC-3.  In _Proc. TREC_ , pages 109–126.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  FaceNet: A unified embedding for face recognition and clustering.  In _Proc. CVPR_ , pages 815–823.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership inference attacks against machine learning models.  In _Proc. IEEE S &P_, pages 3–18.
  * Sohn (2016) Kihyuk Sohn. 2016.  Improved deep metric learning with multi-class N-pair loss objective.  In _Proc. NIPS_.
  * Spangher et al. (2023) Alexander Spangher, Nanyun Peng, Emilio Ferrara, and Jonathan May. 2023.  Identifying informational sources in news articles.  In _Proc. EMNLP_ , pages 3626–3639.
  * Tian et al. (2020) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020.  What makes for good views for contrastive learning?  arXiv:2005.10243.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and efficient foundation language models.  arXiv:2302.13971.
  * Vaucher et al. (2021) Timoté Vaucher, Andreas Spitz, Michele Catasta, and Robert West. 2021.  Quotebank: A corpus of quotations from a decade of news.  In _Proc. WSDM_ , pages 328––336.
  * Wang et al. (2023a) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023a.  DecodingTrust: A comprehensive assessment of trustworthiness in GPT models.  In _Proc. NeurIPS_.
  * Wang et al. (2023b) Jingtan Wang, Xinyang Lu, Zitong Zhao, Zhongxiang Dai, Chuan-Sheng Foo, See-Kiong Ng, and Bryan Kian Hsiang Low. 2023b.  WASA: Watermark-based source attribution for large language model-generated data.  arXiv:2310.00646.
  * Watson et al. (2022) Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles. 2022.  On the importance of difficulty calibration in membership inference attacks.  arXiv:2111.08440.
  * Wei et al. (2024) Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia. 2024.  Proving membership in LLM pretraining data via data watermarks.  arXiv:2402.10892.
  * Wu et al. (2020) Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020.  CLEAR: Contrastive learning for sentence representation.  arXiv:2012.15466.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  arXiv:2007.00808.
  * Yu et al. (2022) Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. 2022.  Artificial fingerprinting for generative models: Rooting deepfake attribution in training data.  arXiv:2007.08457.
  * Zhang et al. (2019) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Proc. NeurIPS_.

##  Appendix A Proof on the Centroid of Clusters

Given a cluster of embeddings $\bm{z}_{1},\bm{z}_{2},\ldots,\bm{z}_{k}$ with
the same label/source, a good representative of the cluster would be the
centroid $\bar{\bm{z}}$ that maximizes the sum of its cosine similarity with
every normalized embedding $\bm{z}_{i}$ for $i=1,\ldots,k$. Equivalently,
$\bar{\bm{z}}$ minimizes the sum of its standard cosine distance with every
normalized embedding:

$\begin{array}[]{l}\displaystyle\sum_{i=1}^{k}\left(1-\frac{\bm{z}_{i}\cdot\bar{\bm{z}}}{\|\bm{z}_{i}\|\|\bar{\bm{z}}\|}\right)=k-\sum_{i=1}^{k}\frac{\bm{z}_{i}\cdot\bar{\bm{z}}}{\|\bm{z}_{i}\|\|\bar{\bm{z}}\|}\\\
\displaystyle=k-\left(\sum_{i=1}^{k}\frac{\bm{z}_{i}}{\|\bm{z}_{i}\|}\right)\cdot\frac{\bar{\bm{z}}}{\|\bar{\bm{z}}\|}\geq
k-\left|\left(\sum_{i=1}^{k}\frac{\bm{z}_{i}}{\|\bm{z}_{i}\|}\right)\cdot\frac{\bar{\bm{z}}}{\|\bar{\bm{z}}\|}\right|\\\
\displaystyle\geq
k-\left\|\sum_{i=1}^{k}\frac{\bm{z}_{i}}{\|\bm{z}_{i}\|}\right\|\end{array}$

by Cauchy-Schwarz inequality. The equality holds when there exists some
$\lambda\in\mathbb{R}$ such that

$\sum_{i=1}^{k}\frac{\bm{z}_{i}}{\|\bm{z}_{i}\|}=\lambda\frac{\bar{\bm{z}}}{\|\bar{\bm{z}}\|}\
.$

In other words, $\bar{\bm{z}}$ can be obtained by adding all normalized
embeddings and setting $\lambda=1$:

$\bar{\bm{z}}=\sum_{i=1}^{k}\frac{\bm{z}_{i}}{\|\bm{z}_{i}\|}\ .$

##  Appendix B Experimental Setup

#### Data Preparation.

From booksum, we have randomly selected subsets of $10$, $25$, $50$, and $100$
books. From dbpedia_14, we chose $10$ distinct classes. Additionally, we have
extracted text samples from $25$ diverse domains within the cc_news dataset.

Before proceeding with the analysis, we have performed standard preprocessing
steps which include converting all text to lowercase and removing punctuation
to ensure uniformity and cleanliness in the data.

#### Model.

For sentence embedding, we have opted for SBERT (Reimers and Gurevych, 2019).
Leveraging the pre-trained model xlm-r-distilroberta-base-paraphrase-v1 that
is readily accessible on Hugging Face, we have fine-tuned it within our TRACE
framework. Moreover, we have augmented the model with additional feed-forward
layers which serve as the projection network. The dimension for the embeddings
is set as $64$.

#### Training Details.

The hyperparameters utilized in our experimental setup are configured as
follows: the learning rate is $1\times 10^{-5}$, the batch size is $64$, the
number of epochs is $150$, and the temperature in the NT-Xent Loss is $0.1$.
Notably, all training procedures are conducted on a single NVIDIA L$40$ GPU,
obviating model or data parallelism techniques. The results were obtained by
averaging the outcomes of three executions, each with a different random seed.

Statistic | booksum | dbpedia_14 | cc_news
---|---|---|---
Number of Documents | 405 (books) | 560,000 | 149,954,415
Languages Covered | English | English | English
Domains | Books | Encyclopedic | News
Table 6: Statistics of booksum, dbpedia_14, and cc_news datasets.

#### Attack Details.

The primary attack methods utilized in this study are deletion, synonym
substitution, and paraphrasing. For the deletion method, a specified portion
of the response from the LLM is randomly selected and subsequently removed. In
the case of synonym substitution, WordNet (Miller, 1994) serves as the synonym
database. Similar to the deletion method, a random portion of words is
replaced with their synonyms. In deletion and synonym substitution attacks,
the portions of words being modified are 5%, 10% and 15%, as shown in Table 3.
For paraphrasing, we employ the fine-tuned paraphrasing model
pegasus_paraphrase (Zhang et al., 2019), which is available on Hugging
Face.333https://huggingface.co/tuner007/pegasus_paraphrase


$\hat{\mu}_{b}^{*}=\mu_{\mathcal{J}}\hat{W}_{b}^{*}$. Because
$m\rightarrow\infty$ and $m/T_{0}\rightarrow 0$ as $T_{0}\rightarrow\infty$,
$\sqrt{T_{0}}(\hat{\mu}_{0}-\mu_{0})$ converges in distribution by Proposition
3, and $\\{Y_{t}\\}_{t\in\mathcal{T}_{0}}$ is $\alpha$-mixing by Assumption
4.2, it follows from Theorem 4.3.1 of Politis et al. (1999) that
$\sqrt{T_{0}}\mu_{\mathcal{J}}(\hat{W}_{b}^{*}-\hat{W})=\sqrt{T_{0}}(\hat{\mu}_{b}^{*}-\hat{\mu}_{0})\overset{d}{\sim}\sqrt{T_{0}}(\hat{\mu}_{0}-\mu_{0})$.
Therefore,
$\hat{\bar{\alpha}}_{v}-\hat{\alpha}^{*}\overset{d}{\sim}\sqrt{T_{1}}(\hat{\bar{\alpha}}_{v}-\bar{\alpha}_{v})$.

## Appendix B. Additional Formal Results

Examples for Assumption 2: Suppose that $\epsilon_{it}$ are independent across
$i$ and $t$, $E[\epsilon_{it}]=0$, and
$\sup_{i,t}E[|\epsilon_{it}|^{n}]<\infty$ for some even $n\geq 2$. Then
$\text{for all }\eta>0$,

$P(\max_{k<0,i\geq
0}\\{|\frac{1}{T_{0}}\sum_{t\in\mathcal{T}_{0}}\epsilon_{kt}\epsilon_{it}|\\}>\eta)\leq\sum_{0<k\leq
K}\sum_{i=0}^{J}P(|\frac{1}{T_{0}}\sum_{t\in\mathcal{T}_{0}}\epsilon_{kt}\epsilon_{it}|>\eta)$
$\leq\sum_{0<k\leq
K}\sum_{i=0}^{J}\frac{E[(\frac{1}{T_{0}}\sum_{t\in\mathcal{T}_{0}}\epsilon_{kt}\epsilon_{it})^{n}]}{\eta^{n}}\leq
K\frac{J+1}{T_{0}^{n}}(\sum_{h=1}^{n/2}C_{h}T_{0}^{h})$

for some constants $C_{1},..,C_{n/2}$. Therefore, if
$\frac{J}{T_{0}^{n/2}}\rightarrow 0$, then $\max_{k<0,i\geq
0}\\{|\frac{1}{T_{0}}\sum_{t\in\mathcal{T}_{0}}\epsilon_{kt}\epsilon_{it}|\\}\overset{p}{\rightarrow}0$.
By the same reasoning, if $\frac{J}{T_{0}^{n/2}}\rightarrow 0$ then
$\max_{0\leq i\leq
J}\\{|\frac{1}{T_{0}}\sum_{t\in\mathcal{T}_{0}}\epsilon_{it}|\\}\overset{p}{\rightarrow}0$.
Hence, for Assumptions 2.1 and 2.5, by uniformly bounding more of the moments
of the idiosyncratic shocks, we can increase the rate at which $J$ is allowed
to grow relative to $T_{0}$.

Examples for Assumption 3: For Assumption 3.2, consider the case where all
idiosyncratic shocks are independent across unit and time,
$E[\epsilon_{it}]=0$, and $\sup_{i,t}E[\epsilon_{it}^{4}]<\infty$. Then
$\text{for all }\eta>0$,

$P(\max_{0\leq i\leq
J}\\{|\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\epsilon_{it}|\\}>\eta)\leq\sum_{i=0}^{J}P(|\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\epsilon_{it}|>\eta)$
$\leq\sum_{i=0}^{J}\frac{E[(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\epsilon_{it})^{4}]}{\eta^{4}}\leq(J+1)C||v^{T_{1}}||_{2}^{4}$

for some constant $C$. Therefore, if
$J^{\frac{1}{4}}||v^{T_{1}}||_{2}\rightarrow 0$, then $\max_{0\leq i\leq
J}\\{|\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\epsilon_{it}|\\}\overset{p}{\rightarrow}0$
as $T_{1}\rightarrow\infty$.

Examples for Assumption 4: Using Theorem 1.1 of Bradley and Tone (2017), we
can see how Assumption 4 can be satisfied even in the presence of non-
stationarity. For Assumption 4.5, suppose that

$\lim_{T_{1}\rightarrow\infty}E[(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}(\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt}))^{2}]\equiv\Sigma_{v}<\infty,$

that
$\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}E[\alpha_{0t}]-\bar{\alpha}_{v}=o(\frac{1}{\sqrt{T_{1}}})$,
that the sequence
$\\{v_{t}^{T_{1}}(\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt})\\}_{t\in\mathcal{T}_{1}}$
has finite second moments181818Because $||W^{*}||_{1}=1$ and Assumption 2.5,
this will be true if $\\{\alpha_{0t}\\}_{t\in\mathcal{T}_{1}}$ has finite
second moments. and satisfies the Lindeberg condition191919That is, if we let
$\sigma_{T_{1}}^{2}=E[(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}(\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt}))^{2}]$,
then  $\text{for all
}\delta>0,\;\lim_{T_{1}\rightarrow\infty}\frac{1}{\sigma_{T_{1}}^{2}}\sum_{t\in\mathcal{T}_{1}}E[(v_{t}^{T_{1}}(\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt}))^{2}]I(|v_{t}^{T_{1}}(\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt})|>\sigma_{T_{1}}\delta)=0.$
, and

$\sup_{t\in\mathcal{T}_{1},f,g\in\mathcal{L}^{2}}|Corr(f(\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt})),g(\alpha_{0t+1}-\bar{\alpha}_{v}+\epsilon_{0t+1}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt+1})))|<1.$

Lastly, note that by Assumption 4.2, the sequence
$\\{\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt})\\}_{t\in\mathcal{T}_{1}}$
must be $\alpha$-mixing. Then by Theorem 1.1 of Bradley and Tone (2017),

$\sqrt{T_{1}}\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}(\alpha_{0t}-\bar{\alpha}_{v}+\epsilon_{0t}-\sum_{j\in\mathcal{J}}W_{j}^{*}\epsilon_{jt})\overset{d}{\rightarrow}N(0,\Sigma_{v})\text{
as }T_{1}\rightarrow\infty.$

For Assumption 4.3, note that
$(\frac{1}{\sqrt{T_{0}}}\tilde{Y}_{\mathcal{K}}^{pre}(Y_{0}^{pre^{\prime}}-Y_{\mathcal{J}}^{pre^{\prime}}W^{*}))=(\frac{1}{\sqrt{T_{0}}}\tilde{Y}_{\mathcal{K}}^{pre}(\epsilon_{0}^{pre^{\prime}}-\epsilon_{\mathcal{J}}^{pre^{\prime}}W^{*}))$.
The sequence
$\\{\tilde{Y}_{\mathcal{K}t}(\epsilon_{0t}^{\prime}-\epsilon_{\mathcal{J}t}^{\prime}W^{*})\\}_{t\in\mathcal{T}_{0}}$
will also be $\alpha$-mixing by Assumption 4.2 and impose that
$\\{(\lambda_{t},\epsilon_{t}^{\prime})\\}_{t\in\mathcal{T}_{0}}$ have finite
second moments so that
$\\{\tilde{Y}_{\mathcal{K}t}(\epsilon_{0t}^{\prime}-\epsilon_{\mathcal{J}t}^{\prime}W^{*})\\}_{t\in\mathcal{T}_{0}}\\}$
will have finite second moments as well. Therefore, if the sequence
$\\{\tilde{Y}_{\mathcal{K}t}(\epsilon_{0t}^{\prime}-\epsilon_{\mathcal{J}t}^{\prime}W^{*})\\}_{t\in\mathcal{T}_{0}}\\}$
satisfies the Lindeberg condition,

$\lim_{T_{0}\rightarrow\infty}E[\frac{1}{T_{0}}(\epsilon_{0}^{pre^{\prime}}-\epsilon_{\mathcal{J}}^{pre^{\prime}}W^{*})\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}\tilde{Y}_{\mathcal{K}}^{pre}(\epsilon_{0}^{pre^{\prime}}-\epsilon_{\mathcal{J}}^{pre^{\prime}}W^{*})]=\mathcal{V}\text{,
and }$
$\sup_{t\in\mathcal{T}_{0},f,g\in\mathcal{L}^{2}}|Corr(f(\tilde{Y}_{\mathcal{K}t}(\epsilon_{0t}-\epsilon_{\mathcal{J}t}^{\prime}W^{*})),g(\tilde{Y}_{\mathcal{K}t+1}(\epsilon_{0t+1}-\epsilon_{\mathcal{J}t+1}^{\prime}W^{*}))|<1,$

then
$(\frac{1}{\sqrt{T_{0}}}\tilde{Y}_{\mathcal{K}}^{pre}(Y_{0}^{pre^{\prime}}-Y_{\mathcal{J}}^{pre^{\prime}}W^{*}))\overset{d}{\rightarrow}N(0,\mathcal{V})$
as $T_{0}\rightarrow\infty$.

Examples of Consistent Estimators of $\Sigma_{v}$: First note that
$\hat{\bar{\alpha}}_{v}$ can be rewritten as
$\frac{1}{T_{1}}\sum_{t\in\mathcal{T}_{1}}T_{1}v_{t}^{T_{1}}\hat{\alpha}_{0t}$.
Then suppose we let $\hat{\Sigma}$ be equal to lag window type long-run
variance estimator, so

$\hat{\Sigma}_{v}=\frac{1}{T_{1}}\sum_{t\in\mathcal{T}_{1}}\sum_{s\in\mathcal{T}_{1}}\kappa(\frac{t-s}{\tau
T_{1}})(T_{1}v_{t}^{T_{1}}\hat{\alpha}_{0t}-\hat{\bar{\alpha}}_{v})(T_{1}v_{s}^{T_{1}}\hat{\alpha}_{0s}-\hat{\bar{\alpha}}_{v})$

where $\kappa$ is a kernel function and $\tau$ is the ratio of the truncation
lag to $T_{1}$. While the specific choice of truncation and kernel function
may influence the finite sample performance of this inference method,
$\hat{\Sigma}_{v}$ can be a consistent estimator for a variety of choices. To
see this, first note that $\text{for all }t\in\mathcal{T}_{1}$,

$\hat{\alpha}_{0t}=\alpha_{0t}+\lambda_{t}(\mu_{0}-\hat{\mu}_{0})+\epsilon_{0t}-\epsilon_{\mathcal{J}t}(\hat{W}-W^{*})-\epsilon_{\mathcal{J}t}W^{*}=\alpha_{0t}+\epsilon_{0t}-\epsilon_{\mathcal{J}t}W^{*}+O_{p}(T_{0}^{-\frac{1}{2}})$

by Lemma 2 and Lemma 4 where the bound in the last term is uniform in
$t\in\mathcal{T}_{1}$. Similarly, by Assumption 4, Lemma 2, and Lemma 4,

$\hat{\bar{\alpha}}_{v}=(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\alpha_{0t})+(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\lambda_{t})(\mu_{0}-\hat{\mu}_{0})+(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\epsilon_{0t})-(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\epsilon_{\mathcal{J}t})(\hat{W}-W^{*})-(\sum_{t\in\mathcal{T}_{1}}v_{t}^{T_{1}}\epsilon_{\mathcal{J}t})W^{*}$
$=\bar{\alpha}_{v}+O_{p}(T_{0}^{-\frac{1}{2}}+T_{1}^{-\frac{1}{2}}).$

Hence,

$\hat{\Sigma}_{v}=\frac{1}{T_{1}}\sum_{t\in\mathcal{T}_{1}}\sum_{s\in\mathcal{T}_{1}}\kappa(\frac{t-s}{\tau
T_{1}})(T_{1}v_{t}^{T_{1}}\hat{\alpha_{0t}}-\hat{\bar{\alpha}}_{v})(T_{1}v_{s}^{T_{1}}\hat{\alpha_{0s}}-\hat{\bar{\alpha}}_{v})$
$=\frac{1}{T_{1}}\sum_{t\in\mathcal{T}_{1}}\sum_{s\in\mathcal{T}_{1}}\kappa(\frac{t-s}{\tau
T_{1}})(T_{1}v_{t}^{T_{1}}(\alpha_{0t}+\epsilon_{0t}-\epsilon_{\mathcal{J}t}W^{*})-\bar{\alpha}_{v})(T_{1}v_{s}^{T_{1}}(\alpha_{0s}+\epsilon_{0s}-\epsilon_{\mathcal{J}s}W^{*})-\bar{\alpha}_{v})$
$+\frac{2}{T_{1}}\sum_{t\in\mathcal{T}_{1}}\sum_{s\in\mathcal{T}_{1}}\kappa(\frac{t-s}{\tau
T_{1}})(T_{1}v_{t}^{T_{1}}(\alpha_{0t}+\epsilon_{0t}-\epsilon_{\mathcal{J}t}W^{*})-\bar{\alpha}_{v})O_{p}(T_{0}^{-\frac{1}{2}}+T_{1}^{-\frac{1}{2}})$
$+O_{p}(T_{0}^{-\frac{1}{2}}+T_{1}^{-\frac{1}{2}})\frac{1}{T_{1}}\sum_{t\in\mathcal{T}_{1}}\sum_{s\in\mathcal{T}_{1}}\kappa(\frac{t-s}{\tau
T_{1}}).$

The second and third terms converges in probability to zero as long as
$\frac{1}{T_{1}}\sum_{t\in\mathcal{T}_{1}}\sum_{s\in\mathcal{T}_{1}}\kappa(\frac{t-s}{\tau
T_{1}})$ is bounded. Therefore, $\hat{\Sigma}_{v}$ will be consistent as long
as

$\frac{1}{T_{1}}\sum_{t\in\mathcal{T}_{1}}\sum_{s\in\mathcal{T}_{1}}\kappa(\frac{t-s}{\tau
T_{1}})(T_{1}v_{t}^{T_{1}}(\alpha_{0t}+\epsilon_{0t}-\epsilon_{\mathcal{J}t}W^{*})-\bar{\alpha}_{v})(T_{1}v_{s}^{T_{1}}(\alpha_{0s}+\epsilon_{0s}-\epsilon_{\mathcal{J}s}W^{*})-\bar{\alpha}_{v})\overset{p}{\rightarrow}\Sigma_{v}\text{
as }T_{0},T_{1}\rightarrow\infty.$

When $v^{T_{1}}=(\frac{1}{T_{1}},...,\frac{1}{T_{1}})$ and
$\\{\alpha_{0t}-\alpha_{v}+\epsilon_{0t}-\epsilon_{\mathcal{J}t}W^{*}\\}_{t\in\mathcal{T}_{1}}$
is weakly stationary, many choices of kernel would lead to a consistent
estimator such as Bartlett, Parzen, Quadratic Spectral, and Turkey-Hamming.
When there is non-stationarity or unequal weights, there exists kernel-based
estimators that can consistently estimate the long-run variance even in the
presence of heteroskedasticity and more general serial correlation (see for
example Newey and West (1987) and Andrews (1991)). On the other hand, if
instead the treatment effects and idiosyncratic shocks were independent over
time, then $\Sigma_{v}$ could be consistently estimated simply by
$\hat{\Sigma}_{v}=\frac{1}{T_{1}-1}\sum_{t\in\mathcal{T}_{1}}(T_{1}v_{t}^{T_{1}}\hat{\alpha}_{0t}-\hat{\bar{\alpha}}_{v})^{2}$.

Lemma 3 Under the same conditions as in Lemma 2, we have

$\hat{\mu}_{0}=\mu_{\mathcal{J}}\Pi_{\Delta^{J}}(\hat{W}^{GMM})+o_{p}(T_{0}^{-\frac{1}{2}})$

where
$\hat{W}^{GMM}=\operatorname*{arg\,min}_{W\in\mathcal{W}^{GMM}}||W-W^{*}||_{2}$
and $\mathcal{W}^{GMM}$ is the set of optimal solutions for the estimator
defined by equation (2).

Proof: The proof follows Lemma C.1 from Li (2020). Let
$\tilde{W}=\Pi_{\Delta^{J}}(\hat{W}^{GMM})$ and
$\tilde{\mu}_{0}=\mu_{\mathcal{J}}\tilde{W}$. Then let $\epsilon>0$ and
suppose $\sqrt{T_{0}}||\hat{\mu}_{0}-\tilde{\mu}_{0}||_{2}>\epsilon$. Because
$\sqrt{T_{0}}||\mu_{\mathcal{J}}||_{2}||\hat{W}-\tilde{W}||_{2}\geq\sqrt{T_{0}}||\mu_{\mathcal{J}}(\hat{W}-\tilde{W})||_{2}>\epsilon$,
$\sqrt{T_{0}}||\hat{W}-\tilde{W}||_{2}>\epsilon/||\mu_{\mathcal{J}}||_{2}$.
Let
$\pi_{\Delta^{J},T_{0}}(\theta)=\theta^{\prime}(Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0})\theta=\sqrt{T_{0}}\theta^{\prime}(Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2})\sqrt{T_{0}}\theta$.
Then because $\hat{W}\neq\tilde{W}$, $\tilde{W}\in\Delta^{J}$, and
$\hat{W}=\Pi_{\Delta^{J},T_{0}}(\hat{W}^{GMM})$,

$\pi_{\Delta^{J},T_{0}}(\hat{W}^{GMM}-\hat{W})<\pi_{\Delta^{J},T_{0}}(\hat{W}^{GMM}-\tilde{W}).\quad(B.1)$

Then note that because of the quadratic form of
$\pi_{\Delta^{J},T_{0}}(\theta)$,

$\pi_{\Delta^{J},T_{0}}(\hat{W}^{GMM}-\hat{W})=\pi_{\Delta^{J},T_{0}}(\hat{W}^{GMM}-\tilde{W}+\tilde{W}-\hat{W})=\pi_{\Delta^{J},T_{0}}(\hat{W}^{GMM}-\tilde{W})+\pi_{\Delta^{J},T_{0}}(\tilde{W}-\hat{W})+$
$2\sqrt{T_{0}}(\tilde{W}-\hat{W}^{GMM})^{\prime}(Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2})\sqrt{T_{0}}(\hat{W}-\tilde{W}).\quad(B.2)$

Combining B.1 and B.2, the last two terms in B.2 are negative so
$D_{1,T_{0}}+D_{2,T_{0}}<0$ where
$D_{1,T_{0}}\equiv\pi_{\Delta^{J},T_{0}}(\tilde{W}-\hat{W})$ and
$D_{2,T_{0}}\equiv
2\sqrt{T_{0}}(\tilde{W}-\hat{W}^{GMM})^{\prime}(Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2})\sqrt{T_{0}}(\hat{W}-\tilde{W})$.

Let $\mathcal{S}^{J}=\\{a\in\mathbb{R}^{J}:||a||_{2}=1\\}$ denote the unit
sphere in $\mathbb{R}^{J}$. Note that

$D_{1,T_{0}}=\sqrt{T_{0}}(\tilde{W}-\hat{W})^{\prime}(Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2})\sqrt{T_{0}}(\tilde{W}-\hat{W})$
$=\sqrt{T_{0}}(\tilde{W}-\hat{W})^{\prime}(\mu_{\mathcal{J}}^{\prime}\lambda^{pre^{\prime}}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\lambda^{pre}\mu_{\mathcal{J}}/T_{0})\sqrt{T_{0}}(\tilde{W}-\hat{W})+o_{p}(1)$
$=||\sqrt{T_{0}}(\tilde{\mu}_{0}-\hat{\mu}_{0})||_{2}^{2}[\frac{\sqrt{T_{0}}(\tilde{\mu}_{0}-\hat{\mu}_{0})^{\prime}}{||\sqrt{T_{0}}(\tilde{\mu}_{0}-\hat{\mu}_{0})||_{2}}(\lambda^{pre^{\prime}}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\lambda^{pre^{\prime}}/T_{0})\frac{\sqrt{T_{0}}(\tilde{\mu}_{0}-\hat{\mu}_{0})}{||\sqrt{T_{0}}(\tilde{\mu}_{0}-\hat{\mu}_{0})||_{2}}]+o_{p}(1)$
$\geq
T_{0}||\tilde{\mu}_{0}-\hat{\mu}_{0}||_{2}^{2}\inf_{a\in\mathcal{S}^{J}}a^{\prime}(\lambda^{pre^{\prime}}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\lambda^{pre}/T_{0}^{2})a=T_{0}||\tilde{\mu}_{0}-\hat{\mu}_{0}||_{2}^{2}\lambda_{min}(\lambda^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\lambda^{pre}/T_{0}^{2})+o_{p}(1)$
$\geq\epsilon^{2}\lambda_{min}(\lambda^{pre^{\prime}}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\lambda^{pre}/T_{0}^{2})+o_{p}(1)\overset{p}{\rightarrow}\epsilon^{2}\lambda_{min}(\Lambda^{\prime}A\Lambda)>0,$

where the minimum eigenvalue of a square matrix $M$ is denoted by
$\lambda_{min}(M)$. The second equality come from Assumptions 2.1, 2.4, and
2.5. The third equality follows from Lemma C.2 of Li (2020). The second
inequality follows because
$\sqrt{T_{0}}||\tilde{\mu}_{0}-\hat{\mu}_{0}||_{2}\geq\epsilon$ and the final
term is strictly positive because $\Lambda^{\prime}A\Lambda$ is nonsingular.
By rewriting
$Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2}=\mu_{\mathcal{J}}^{\prime}\Lambda^{\prime}A\Lambda\mu_{\mathcal{J}}+Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2}-\mu_{\mathcal{J}}^{\prime}\Lambda^{\prime}A\Lambda\mu_{\mathcal{J}}$,
$D_{2,T_{0}}$ can be rewritten as $D_{2,1,T_{0}}+D_{2,2,T_{0}}$ where

$D_{2,1,T_{0}}\equiv
2\sqrt{T_{0}}(\tilde{W}-\hat{W}^{GMM})^{\prime}\mu_{\mathcal{J}}^{\prime}\Lambda^{\prime}A\Lambda\Omega_{0}\mu_{\mathcal{J}}\sqrt{T_{0}}(\hat{W}-\tilde{W})\text{
and }$ $D_{2,2,T_{0}}\equiv
2\sqrt{T_{0}}(\tilde{W}-\hat{W}^{GMM})^{\prime}(Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2}-\mu_{\mathcal{J}}^{\prime}\Lambda^{\prime}A\Lambda\mu_{\mathcal{J}})\sqrt{T_{0}}(\hat{W}-\tilde{W}).$

By definition of $\tilde{W}$ and Lemma 1.1 in Zarantonello (1971),
$D_{2,1,T_{0}}\geq 0$. By Assumption 2,

$||Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0}^{2}-\mu_{\mathcal{J}}^{\prime}\Lambda^{\prime}A\Lambda\mu_{\mathcal{J}}||_{2}=o_{p}(1).$

Additionally, $\sqrt{T_{0}}(\hat{W}-\tilde{W})=O_{p}(1)$ because

$||\sqrt{T_{0}}(\hat{W}-\tilde{W})||_{2}\leq||\sqrt{T_{0}}(\hat{W}-W^{*})||_{2}+||\sqrt{T_{0}}(\tilde{W}-W^{*})||_{2}$
$=\sqrt{T_{0}}||\hat{W}-W^{*}||_{2}+\sqrt{T_{0}}||\Pi_{\Delta^{J}}(\hat{W}^{GMM})-\Pi_{\Delta^{J}}(W^{*})||_{2}$
$\leq\sqrt{T_{0}}||\hat{W}-W^{*}||_{2}+\sqrt{T_{0}}C||\hat{W}^{GMM}-W^{*}||_{2}=O_{p}(1),$

where C is some constant. The second inequality follows from the Lipshitz
continuity of projection operators shown by Zarantonello (1971) and the final
equality follows from Lemma 4. Therefore $D_{2,2,T_{0}}=o_{p}(1)$. Then since
$D_{2,1,T_{0}}\geq 0$ implies that $D_{2,T_{0}}\geq o_{p}(1)$.

Hence, we have that
$\sqrt{T_{0}}||\hat{\mu}_{0}-\tilde{\mu}_{0}||_{2}>\epsilon$ implies that
$D_{T_{0}}<0$ and that
$D_{T_{0}}=D_{1,T_{0}}+D_{2,T_{0}}\geq\epsilon^{2}\lambda_{min}(\lambda^{pre^{\prime}}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\lambda^{pre}/T_{0}^{2})+o_{p}(1)$.
Using this, we have that

$P(\sqrt{T_{0}}||\hat{\mu}_{0}-\tilde{\mu}_{0}||_{2}>\epsilon)\leq
P(D_{T_{0}}<0)\leq$
$P(o_{p}(1)+\epsilon^{2}\lambda_{min}(Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}/T_{0})<0)\rightarrow
P(\epsilon^{2}\lambda_{min}(\mu_{\mathcal{J}}^{\prime}\Omega_{0}\mu_{\mathcal{K}}A\mu_{\mathcal{K}}^{\prime}\Omega_{0}\mu_{\mathcal{J}})\leq
0)=0$

Therefore $T_{0}||\hat{\mu}_{0}-\tilde{\mu}_{0}||_{2}=o_{p}(1)$, or
equivalently
$\hat{\mu}_{0}=\mu_{\mathcal{J}}\Pi_{\Delta^{J}}(\hat{W}^{GMM})+o_{p}(T_{0}^{-\frac{1}{2}})$.

Lemma 4 Suppose the same conditions as Lemma 2. Let $\mathcal{W}^{GMM}$ denote
the set of optimal solutions for the unconstrained estimator in equation (2)
and let
$\hat{W}^{GMM}=\operatorname*{arg\,min}_{W\in\mathcal{W}^{GMM}}||W-W^{*}||_{2}$.
Then $\sqrt{T_{0}}||\hat{W}^{GMM}-W^{*}||_{2}=O_{p}(1)$ and
$\sqrt{T_{0}}||\hat{W}-W^{*}||_{2}=O_{p}(1)$.

Proof: Let $H_{T_{0}}(W)$ equal the objective function in equation (2) and

$H(W)=(\mu_{0}-\mu_{\mathcal{J}}W)^{\prime}\Lambda^{\prime}A\Lambda(\mu_{0}-\mu_{\mathcal{J}}W).$

As shown in Lemma 1 for the case where $J$ is fixed, $H_{T_{0}}(W)$ converges
uniformly in probability to $H(W)$ on $\Delta^{J}$. Then the fact that $W^{*}$
is the unique solution to $\operatorname*{arg\,min}_{W\in\Delta^{J}}H(W)$ and
Theorem 3.2 of Poskitt (2020) imply that
$\sqrt{T_{0}}||\hat{W}-W^{*}||_{2}=O_{p}(1)$ for each
$\hat{W}\in\operatorname*{arg\,min}_{W\in\Delta^{J}}H_{T_{0}}(W)$. Let
$\delta>0$ and $\mathcal{B}=\\{W\;|\;||W-W^{*}||_{2}\leq 2\delta\\}$. By the
same reasoning as in Lemma 1, because $\mathcal{B}$ is compact, $H_{T_{0}}(W)$
converges uniformly in probability to $H(W)$ on $\mathcal{B}$. Therefore, for
the solution set
$\tilde{W}^{GMM}=\operatorname*{arg\,min}_{W\in\mathcal{B}}H_{T_{0}}(W)$,
Theorem 3.2 of Poskitt (2020) also implies that
$\inf_{W\in\tilde{W}^{GMM}}||W-W^{*}||_{2}=O_{p}(\frac{1}{\sqrt{T_{0}}})$. So
the event that there exists
$\tilde{W}=\operatorname*{arg\,min}_{W\in\tilde{\mathcal{W}}^{GMM}}||W-W^{*}||_{2}$
such that $||\tilde{W}-W^{*}||_{2}<\delta$ occurs with probability approaching
one(wpa1). In this event, for $W\notin\mathcal{B}$, there exists $0<\lambda<1$
such that $\lambda\tilde{W}+(1-\lambda)W\in\mathcal{B}$ so that

$H_{T_{0}}(\tilde{W})\leq H_{T_{0}}(\lambda\tilde{W}+(1-\lambda)W)\leq\lambda
H_{T_{0}}(\tilde{W})+(1-\lambda)H_{T_{0}}W)$

where the second inequality follows from the convexity of $H_{T_{0}}(W)$.
Therefore, $H_{T_{0}}(\tilde{W})\leq H_{T_{0}}(W)$ for all $W$ wpa1 so
$\tilde{W}\in\mathcal{W}^{GMM}$ wpa1. Because
$\hat{W}^{GMM}=\operatorname*{arg\,min}_{W\in\mathcal{W}^{GMM}}||W-W^{*}||_{2}$,
$||\hat{W}^{GMM}-W^{*}||_{2}\leq||\tilde{W}-W^{*}||_{2}$ wpa1. Hence, it also
the case that $\sqrt{T_{0}}||\hat{W}^{GMM}-W^{*}||_{2}=O_{p}(1)$.

Proof that $\hat{W}=\Pi_{\Delta^{J},T_{0}}(\hat{W}^{GMM})$: Let
$\hat{W}^{GMM}$ be defined as in equation (2). In order to see how the
objective function defining $\hat{W}$ in equation (1) can be rewritten in
terms of $\hat{W}^{GMM}$, let
$\hat{u}=Y_{0}^{pre^{\prime}}-Y_{\mathcal{J}}^{pre^{\prime}}\hat{W}^{GMM}$ and
note that

$(Y_{0}^{pre^{\prime}}-Y_{\mathcal{J}}^{pre^{\prime}}W)^{\prime}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}(Y_{0}^{pre^{\prime}}-Y_{\mathcal{J}}^{pre^{\prime}}W)=(\hat{u}+Y_{\mathcal{J}}^{pre^{\prime}}(\hat{W}^{GMM}-W))^{\prime}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}(\hat{u}+Y_{\mathcal{J}}^{pre^{\prime}}(\hat{W}-W))=$
$\hat{u}^{\prime}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\hat{u}+2\hat{u}^{\prime}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}(\hat{W}^{GMM}-W)+(\hat{W}^{GMM}-W)^{\prime}Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}(\hat{W}^{GMM}-W)$
$\hat{u}^{\prime}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\hat{u}+(\hat{W}^{GMM}-W)^{\prime}Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}(\hat{W}^{GMM}-W)$

where the last equality follows from the first order condition for
$\hat{W}^{GMM}$ which is that
$\hat{u}^{\prime}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}=0$.
Since
$\hat{u}^{\prime}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}\hat{u}$
is not dependent on $W$, the set of minimizers of the objective function in
equation (1) is equal to the set of minimizers of
$(\hat{W}^{GMM}-W)^{\prime}Y_{\mathcal{J}}^{pre}\tilde{Y}_{\mathcal{K}}^{pre^{\prime}}A_{T_{0}}\tilde{Y}_{\mathcal{K}}^{pre}Y_{\mathcal{J}}^{pre^{\prime}}(\hat{W}^{GMM}-W)$.

## Appendix C. Additional Simulation Results

Tables A1 and A2 contain the bias and MSE results for the same two scenarios
described in section 4, except now with the probability of treatment being
estimated rather than being uniform. Similarly to Arkhangelsky et al. (2021),
I estimate the probability of each country becoming treated by predicting the
World Bank’s indicators for the country’s income level. That is, I let
assignment indicators for units follow Bernoulli distributions with the
probabilities of being treated being proportional to their estimated
probability using an indicator for the country being high income. This helps
to capture the fact that in SC applications we should expect treatment
assignment to be correlated with unobserved variables related to the outcome
variable. In Table A3, I present bias and MSE results for the scenario with a
single treated unit under the same conditions as in section 4 but with the
untreated unit split into controls and moment generating units uniformly at
random. Table A4 contains the coverage rates for confidence intervals
constructed using the subsampling method described in section 3 under the same
conditions as before, except with $J=50$ instead of $J=10$.

Table A1: Results with Estimated Treatment Probabilities with $K+1$ Treated
Units

| OLS-SCE |  GMM-SCE ($A_{T_{0}}=I_{K+1}$) |  GMM-SCE ($A_{T_{0}}=\hat{\mathcal{V}}^{-1}$) | Uniform SCE
---|---|---|---|---
|  | K=20 | K=5 | K=20 | K=5 |
Bias Magnituide |
$T_{0}=25$,$J=10$ | 0.474 | 0.233 | 0.244 | 0.396 | 0.315 | 1.179
$T_{0}=50$,$J=10$ | 0.425 | 0.156 | 0.173 | 0.296 | 0.233 | 1.179
$T_{0}=100$,$J=10$ | 0.405 | 0.117 | 0.128 | 0.229 | 0.171 | 1.179
$T_{0}=25$,$J=50$ | 0.159 | 0.066 | 0.078 | 0.127 | 0.103 | 1.183
$T_{0}=50$,$J=50$ | 0.124 | 0.027 | 0.030 | 0.074 | 0.038 | 1.183
$T_{0}=100$,$J=50$ | 0.102 | 0.010 | 0.008 | 0.051 | 0.017 | 1.183
$\hat{\alpha}_{0t}$ MSE |
$T_{0}=25$ ,$J=10$ | 6.471 | 6.884 | 7.069 | 6.603 | 6.948 | 7.640
$T_{0}=50$,$J=10$ | 6.203 | 6.751 | 6.925 | 6.485 | 6.803 | 7.640
$T_{0}=100$,$J=10$ | 6.039 | 6.704 | 6.843 | 6.426 | 6.781 | 7.640
$T_{0}=25$,$J=50$ | 2.795 | 2.989 | 3.333 | 2.888 | 3.288 | 7.182
$T_{0}=50$,$J=50$ | 2.657 | 2.960 | 3.198 | 2.855 | 3.159 | 7.182
$T_{0}=100$,$J=50$ | 2.571 | 2.960 | 3.160 | 2.860 | 3.133 | 7.182
$\hat{\bar{\alpha}}$ MSE |
$T_{0}=25$,$J=10$ | 0.531 | 0.407 | 0.431 | 0.496 | 0.449 | 1.698
$T_{0}=50$,$J=10$ | 0.425 | 0.292 | 0.308 | 0.351 | 0.323 | 1.698
$T_{0}=100$,$J=10$ | 0.373 | 0.232 | 0.243 | 0.268 | 0.257 | 1.698
$T_{0}=25$,$J=50$ | 0.181 | 0.175 | 0.210 | 0.185 | 0.204 | 1.539
$T_{0}=50$,$J=50$ | 0.122 | 0.110 | 0.134 | 0.120 | 0.133 | 1.539
$T_{0}=100$,$J=50$ | 0.098 | 0.093 | 0.106 | 0.098 | 0.108 | 1.539

  * •

Notes: All simulations are done with a thousand samples. The percent of
simulations where $\mu_{0}\in\mathcal{M}_{J}$ is 58% when $J=10$ and 100% when
$J=50$.

Table A2: Results with Estimated Treatment Probabilities with One Treated Unit

| OLS-SCE |  GMM-SCE ($A_{T_{0}}=I_{K+1}$) |  GMM-SCE ($A_{T_{0}}=\hat{\mathcal{V}}^{-1}$) | Uniform SCE
---|---|---|---|---
|  | K=3/20 | K=5 | K=3/20 | K=5 |
Bias Magnituide |
$T_{0}=25$,$N_{0}=10$ | 0.490 | 0.266 | 0.300 | 0.355 | 0.375 | 1.355
$T_{0}=50$,$N_{0}=10$ | 0.442 | 0.183 | 0.223 | 0.248 | 0.273 | 1.355
$T_{0}=100$,$N_{0}=10$ | 0.400 | 0.133 | 0.169 | 0.169 | 0.191 | 1.355
$T_{0}=25$,$N_{0}=50$ | 0.159 | 0.088 | 0.079 | 0.163 | 0.135 | 1.183
$T_{0}=50$,$N_{0}=50$ | 0.124 | 0.051 | 0.041 | 0.103 | 0.077 | 1.183
$T_{0}=100$,$N_{0}=50$ | 0.102 | 0.026 | 0.026 | 0.061 | 0.051 | 1.183
$\hat{\alpha}_{0t}$ MSE |
$T_{0}=25$,$N_{0}=10$ | 4.698 | 5.417 | 5.272 | 5.375 | 5.186 | 6.204
$T_{0}=50$,$N_{0}=10$ | 4.465 | 5.327 | 5.194 | 5.243 | 5.088 | 6.204
$T_{0}=100$,$N_{0}=10$ | 4.313 | 5.298 | 5.157 | 5.253 | 5.084 | 6.204
$T_{0}=25$,$N_{0}=50$ | 2.795 | 2.879 | 3.251 | 2.835 | 3.199 | 7.182
$T_{0}=50$,$N_{0}=50$ | 2.657 | 2.769 | 3.138 | 2.770 | 3.094 | 7.182
$T_{0}=100$,$N_{0}=50$ | 2.571 | 2.740 | 3.134 | 2.705 | 3.101 | 7.182
$\hat{\bar{\alpha}}$ MSE |
$T_{0}=25$,$N_{0}=10$ | 0.473 | 0.337 | 0.359 | 0.400 | 0.416 | 2.105
$T_{0}=50$,$N_{0}=10$ | 0.387 | 0.247 | 0.261 | 0.280 | 0.280 | 2.105
$T_{0}=100$,$N_{0}=10$ | 0.326 | 0.194 | 0.209 | 0.212 | 0.212 | 2.105
$T_{0}=25$,$N_{0}=50$ | 0.181 | 0.167 | 0.201 | 0.198 | 0.218 | 1.539
$T_{0}=50$,$N_{0}=50$ | 0.122 | 0.111 | 0.128 | 0.124 | 0.135 | 1.539
$T_{0}=100$,$N_{0}=50$ | 0.098 | 0.086 | 0.098 | 0.091 | 0.100 | 1.539

  * •

Notes: All simulations are done with a thousand samples.

Table A3: Results with One Treated Unit and Randomly Selected Control Units

| OLS-SCE |  GMM-SCE ($A_{T_{0}}=I_{K+1}$) |  GMM-SCE ($A_{T_{0}}=\hat{\mathcal{V}}^{-1}$) | Uniform SCE
---|---|---|---|---
|  | K=3 | K=5 | K=3 | K=5 |
Bias |
$T_{0}=25$,$N_{0}=10$ | 0.488 | 0.417 | 0.331 | 0.507 | 0.387 | 1.342
$T_{0}=50$,$N_{0}=10$ | 0.435 | 0.342 | 0.243 | 0.403 | 0.282 | 1.342
$T_{0}=100$,$N_{0}=10$ | 0.395 | 0.291 | 0.196 | 0.330 | 0.219 | 1.342
$T_{0}=25$,$N_{0}=50$ | 0.120 | 0.065 | 0.090 | 0.092 | 0.100 | 1.351
$T_{0}=50$,$N_{0}=50$ | 0.096 | 0.024 | 0.043 | 0.045 | 0.050 | 1.351
$T_{0}=100$,$N_{0}=50$ | 0.074 | 0.007 | 0.009 | 0.007 | 0.009 | 1.351
$\hat{\alpha}_{0t}$ MSE |
$T_{0}=25$,$N_{0}=10$ | 4.677 | 6.239 | 5.862 | 6.033 | 5.734 | 6.171
$T_{0}=50$,$N_{0}=10$ | 4.432 | 6.246 | 5.809 | 5.989 | 5.701 | 6.171
$T_{0}=100$,$N_{0}=10$ | 4.281 | 6.329 | 5.797 | 6.087 | 5.718 | 6.171
$T_{0}=25$,$N_{0}=50$ | 1.926 | 2.721 | 3.240 | 2.679 | 3.207 | 5.728
$T_{0}=50$,$N_{0}=50$ | 1.837 | 2.676 | 3.154 | 2.627 | 3.122 | 5.728
$T_{0}=100$,$N_{0}=50$ | 1.762 | 2.594 | 3.174 | 2.590 | 3.143 | 5.728
$\hat{\bar{\alpha}}$ MSE |
$T_{0}=25$,$N_{0}=10$ | 0.483 | 0.526 | 0.439 | 0.606 | 0.474 | 2.083
$T_{0}=50$,$N_{0}=10$ | 0.386 | 0.427 | 0.317 | 0.460 | 0.340 | 2.083
$T_{0}=100$,$N_{0}=10$ | 0.326 | 0.376 | 0.268 | 0.387 | 0.281 | 2.083
$T_{0}=25$,$N_{0}=50$ | 0.120 | 0.161 | 0.183 | 0.158 | 0.184 | 1.928
$T_{0}=50$,$N_{0}=50$ | 0.090 | 0.114 | 0.131 | 0.109 | 0.131 | 1.928
$T_{0}=100$,$N_{0}=50$ | 0.064 | 0.086 | 0.103 | 0.082 | 0.103 | 1.928

  * •

Notes: All simulations are done with a thousand samples.

Table A4: Inference Results with$J=50$

| OLS-SCE |  GMM-SCE ($A_{T_{0}}=\hat{\mathcal{V}}^{-1}$, $K=10$)
---|---|---
| 80% CI | 90% CI | 95% CI | 80% CI | 90% CI | 95% CI
$\frac{m}{T_{0}}=.2$ |
$T_{0}=50$,$T_{1}=50$ | 0.706 | 0.816 | 0.880 | 0.548 | 0.680 | 0.767
$T_{0}=100$,$T_{1}=50$ | 0.612 | 0.758 | 0.842 | 0.545 | 0.667 | 0.757
$T_{0}=200$,$T_{1}=50$ | 0.651 | 0.786 | 0.859 | 0.569 | 0.690 | 0.777
$T_{0}=50$, $T_{1}=200$ | 0.912 | 0.960 | 0.973 | 0.725 | 0.840 | 0.905
$T_{0}=100$, $T_{1}=200$ | 0.893 | 0.958 | 0.982 | 0.775 | 0.867 | 0.920
$T_{0}=200$, $T_{1}=200$ | 0.902 | 0.967 | 0.986 | 0.810 | 0.905 | 0.951
$\frac{m}{T_{0}}=.4$ |
$T_{0}=50$,$T_{1}=50$ | 0.568 | 0.687 | 0.786 | 0.526 | 0.659 | 0.741
$T_{0}=100$,$T_{1}=50$ | 0.569 | 0.705 | 0.783 | 0.531 | 0.668 | 0.758
$T_{0}=200$,$T_{1}=50$ | 0.621 | 0.745 | 0.830 | 0.589 | 0.705 | 0.797
$T_{0}=50$, $T_{1}=200$ | 0.808 | 0.890 | 0.934 | 0.697 | 0.795 | 0.872
$T_{0}=100$, $T_{1}=200$ | 0.809 | 0.894 | 0.946 | 0.751 | 0.852 | 0.910
$T_{0}=200$, $T_{1}=200$ | 0.852 | 0.931 | 0.965 | 0.793 | 0.888 | 0.940
$\frac{m}{T_{0}}=.6$ |
$T_{0}=50$,$T_{1}=50$ | 0.513 | 0.626 | 0.705 | 0.529 | 0.643 | 0.716
$T_{0}=100$,$T_{1}=50$ | 0.531 | 0.650 | 0.757 | 0.521 | 0.657 | 0.752
$T_{0}=200$,$T_{1}=50$ | 0.584 | 0.721 | 0.807 | 0.599 | 0.717 | 0.802
$T_{0}=50$, $T_{1}=200$ | 0.680 | 0.791 | 0.865 | 0.648 | 0.759 | 0.835
$T_{0}=100$, $T_{1}=200$ | 0.724 | 0.826 | 0.897 | 0.697 | 0.801 | 0.886
$T_{0}=200$, $T_{1}=200$ | 0.789 | 0.889 | 0.947 | 0.760 | 0.865 | 0.929

  * •

Notes: All simulations are done with a thousand samples.

Acknowledgements: I am grateful for helpful comments and feedback from Adam
Mccloskey, Carlos Martins-Filho, Daniel Kaffine, and Taylor Jaworski.

## References

  * (1)
  * Abadie (2021) Abadie, Alberto. 2021\. “Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects.” Journal of Economic Literature 59 (2): 391–425.  10.1257/jel.20191450.
  * Abadie et al. (2010) Abadie, Alberto, Alexis Diamond, and Jens Hainmueller. 2010\. “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program.” Journal of the American Statistical Association 105 (490): 493–505.  https://doi.org/10.1198/jasa.2009.ap08746.
  * Abadie et al. (2011) Abadie, Alberto, Alexis Diamond, and Jens Hainmueller. 2011\. “Synth: An R Package for Synthetic Control Methods in Comparative Case Studies.” Journal of Statistical Software 42 (13): 1–17.  10.18637/jss.v042.i13.
  * Abadie et al. (2015) Abadie, Alberto, Alexis Diamond, and Jens Hainmueller. 2015\. “Comparative Politics and the Synthetic Control Method.” American Journal of Political Science, 59 (2): 495–510.  https://doi.org/10.1111/ajps.12116.
  * Abadie and Gardeazabal (2003) Abadie, Alberto, and Javier Gardeazabal. 2003\. “The Economic Costs of Conflict: A Case Study of the Basque Country.” The American Economic Review 93 (1): 113–132.  10.1257/000282803321455188.
  * Agarwal et al. (2023) Agarwal, Anish, Devavrat Shah, and Dennis Shen. 2023\. “Synthetic A/B Testing using Synthetic Interventions.”  10.48550/ARXIV.2006.07691.
  * Andrews (2003) Andrews, D. W. K. 2003\. “End-of-Sample Instability Tests.” Econometrica 71 (6): 1661–1694.  https://doi.org/10.1111/1468-0262.00466.
  * Andrews (1991) Andrews, Donald W. K. 1991\. “Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation.” Econometrica 59 (3): 817–858.  https://doi.org/10.2307/2938229.
  * Andrews (2002) Andrews, Donald W. K. 2002\. “Generalized Method of Moments Estimation When a Parameter Is on a Boundary.” Journal of Business and Economic Statistics 20 (4): 530–544.  https://doi.org/10.1198/073500102288618667.
  * Andrews and Guggenberger (2010) Andrews, Donald W. K., and Patrik Guggenberger. 2010\. “Asymptotic Size and a Problem with Subsampling and with the m out of n Bootstrap.” Econometric Theory 26 (2): 426–468.  https://doi.org/10.1017/S0266466609100051.
  * Andrews and Lu (2001) Andrews, Donald W.K., and Biao Lu. 2001\. “Consistent model and moment selection procedures for GMM estimation with application to dynamic panel data models.” Journal of Econometrics 101 (1): 123–164.  https://doi.org/10.1016/S0304-4076(00)00077-4.
  * Arellano and Honoré (2001) Arellano, Manuel, and Bo Honoré. 2001\. “Chapter 53 - Panel Data Models: Some Recent Developments.” Volume 5. of Handbook of Econometrics 3229–3296, Elsevier, .  https://doi.org/10.1016/S1573-4412(01)05006-1.
  * Arkhangelsky et al. (2021) Arkhangelsky, Dmitry, Susan Athey, David A. Hirshberg, Guido W. Imbens, and Stefan Wager. 2021\. “Synthetic Difference-in-Differences.” American Economic Review 111 (12): 4088–4118.  10.1257/aer.20190159.
  * Bai (2009) Bai, Jushan. 2009\. “Panel Data Models with Interactive Fixed Effects.” Econometrica 77 (4): 1229–1279.  https://doi.org/10.3982/ECTA6135.
  * Bai and Ng (2002) Bai, Jushan, and Serena Ng. 2002\. “Determining the Number of Factors in Approximate Factor Models.” Econometrica 70 (1): 191–221.  https://doi.org/10.1111/1468-0262.00273.
  * Bertrand et al. (2004) Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan. 2004\. “How Much Should We Trust Differences-In-Differences Estimates?.” The Quarterly Journal of Economics 119 (1): 249–275.  10.1162/003355304772839588.
  * Botosaru and Ferman (2019) Botosaru, Irene, and Bruno Ferman. 2019\. “On the role of covariates in the synthetic control method.” Econometrics Journal 22 117–130.  https://doi.org/10.1093/ectj/utz001.
  * Bradley and Tone (2017) Bradley, Richard C., and Cristina Tone. 2017\. “A Central Limit Theorem for Non-stationary Strongly Mixing Random Fields.” Journal of Theoretical Probability 30 655–674.  https://doi.org/10.1007/s10959-015-0656-2.
  * Chamberlain (1984) Chamberlain, Gary. 1984\. “Chapter 22 Panel data.” Volume 2. of Handbook of Econometrics 1247–1318, Elsevier, .  https://doi.org/10.1016/S1573-4412(84)02014-6.
  * Chernozhukov et al. (2021) Chernozhukov, Victor, Kaspar Wüthrich, and Yinchu Zhu. 2021\. “An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls.” Journal of the American Statistical Association 0 (0): 1–16.  10.1080/01621459.2021.1920957.
  * Chernozhukov et al. (2022) Chernozhukov, Victor, Kaspar Wüthrich, and Yinchu Zhu. 2022\. “A t-test for synthetic controls.” Working Paper.
  * Doudchenko and Imbens (2017) Doudchenko, Nikolay, and Guido W. Imbens. 2017\. “Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis.”  10.3386/w22791.
  * Feenstra et al. (2023) Feenstra, Robert C., Robert Inklaar, and Marcel P. Timmer. 2023\. “The Next Generation of the Penn World Table.”Technical Report 10, American Economic Review.  10.15141/S5Q94M, Accessed: 2022-01-19 [Dataset].
  * Ferman (2021) Ferman, Bruno. 2021\. “On the Properties of the Synthetic Control Estimator with Many Periods and Many Controls.” Journal of the American Statistical Association 116 (536): 1764–1772.  https://doi.org/10.1080/01621459.2021.1965613.
  * Ferman and Pinto (2021) Ferman, Bruno, and Cristine Pinto. 2021\. “Synthetic controls with imperfect pretreatment fit.” Quantitative Economics 12 (4): 1197–1221.  https://doi.org/10.3982/QE1596.
  * Ferman et al. (2020) Ferman, Bruno, Cristine Pinto, and Vitor Possebom. 2020\. “Cherry Picking with Synthetic Controls.” Journal of Policy Analysis and Management 39 (2): 510–532.  https://doi.org/10.1002/pam.22206.
  * Hahn and Shi (2017) Hahn, Jinyong, and Ruoyao Shi. 2017\. “Synthetic Control and Inference.” Econometrics 5 (4): 52.  https://doi.org/10.3390/econometrics5040052.
  * Hollingsworth and Wing (2020) Hollingsworth, Alex, and Coady Wing. 2020\. “Tactics for design and inference in synthetic control studies: An applied example using high-dimensional data.” Working Paper.  10.31235/osf.io/fc9xt.
  * Hsiao et al. (2012) Hsiao, Cheng, H. Steve Ching, and Shui Ki Wan. 2012\. “A Panel Data Approach For Program Evaluation: Measuring The Benefits of Political And Economic Integration of Hong Kong with Mainland China.” Journal of Applied Econometrics 27 (5): 705–740.  https://doi.org/10.1002/jae.1230.
  * Kaul et al. (2021) Kaul, Ashok, Stefan Klöbner, Gregor Pfeifer, and Manuel Schieler. 2021\. “Standard Synthetic Control Methods: The Case Of Using All Pre-Intervention Outcomes Together With Covariates.” Journal of Economic Literature 59 (2): 391–425.  https://doi.org/10.1080/07350015.2021.1930012.
  * Li (2020) Li, Kathleen T. 2020\. “Statistical Inference for Average Treatment Effects Estimated by Synthetic Control Methods.” Journal of the American Statistical Association 115 (532): 2068–2083.  https://doi.org/10.1080/01621459.2019.1686986.
  * Li and Bell (2017) Li, Kathleen T., and David R. Bell. 2017\. “Estimation of average treatment effects with panel data: Asymptotic theory and implementation.” Journal of Econometrics 197 (1): 65–75.  https://doi.org/10.1016/j.jeconom.2016.01.011.
  * Liang and Zeger (1986) Liang, Kung-Yee, and Scott L. Zeger. 1986\. “Longitudinal Data Analysis Using Generalized Linear Models.” Biometrika 73 (1): 13–22.  https://doi.org/10.2307/2336267.
  * Masini and Medeiros (2021) Masini, Ricardo, and Marcelo C. Medeiros. 2021\. “Counterfactual Analysis With Artificial Controls: Inference, High Dimensions, and Nonstationarity.” Journal of the American Statistical Association 116 (536): 1773–1788.  10.1080/01621459.2021.1964978.
  * Mehmet Caner and Lee (2018) Mehmet Caner, Xu Han, and Yoonseok Lee. 2018\. “Adaptive Elastic Net GMM Estimation With Many Invalid Moment Conditions: Simultaneous Model and Moment Selection.” Journal of Business & Economic Statistics 36 (1): 24–46.  10.1080/07350015.2015.1129344.
  * Moon and Weidner (2015) Moon, Hyungsik Roger, and Martin Weidner. 2015\. “Linear Regression for Panel with Unknown Number of Factors as Interactive Fixed Effects.” Econometrica 83 (4): 1543–1579.  https://doi.org/10.3982/ECTA9382.
  * Moon and Weidner (2017) Moon, Hyungsik Roger, and Martin Weidner. 2017\. “Dynamic Linear Panel Regression Models with Interactive Fixed Effects.” Econometric Theory 33 (1): 158–195.  https://doi.org/10.1017/S0266466615000328.
  * Newey and McFadden (1994) Newey, Whitney K., and Daniel McFadden. 1994\. “Chapter 36 Large sample estimation and hypothesis testing.” Volume 4. of Handbook of Econometrics 2111–2245, Elsevier, .  https://doi.org/10.1016/S1573-4412(05)80005-4.
  * Newey and West (1987) Newey, Whitney K., and Kenneth D. West. 1987\. “A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix.” Econometrica 55 (3): 703–708.  https://doi.org/10.2307/1913610.
  * Pesaran (2006) Pesaran, M. Hashem. 2006\. “Estimation and Inference in Large Heterogeneous Panels with a Multifactor Error Structure.” Econometrica 74 (4): 967–1012, http://www.jstor.org/stable/3805914.
  * Politis et al. (1999) Politis, Dimitris N., Joseph P. Romano, and Michael Wolf. 1999\. Subsampling. New York: Springer.
  * Poskitt (2020) Poskitt, Don S. 2020\. “On GMM Inference: Partial Identification, Identification Strength, and Non-Standard.” Monash Econometrics and Business Statistics Working Papers.
  * Rockafellar and Wets (1958) Rockafellar, R. Tyrrell, and Roger J. B. Wets. 1958\. Variational Analysis. New York: Wiley.
  * Shi et al. (2023) Shi, Xu, Kendrick Li, Wang Miao, Mengtong Hu, and Eric Tchetgen Tchetgen. 2023\. “Theory for identification and Inference with Synthetic Controls: A Proximal Causal Inference Framework.”
  * World Bank National Accounts Data and OECD National Accounts Data Files (2023) World Bank National Accounts Data and OECD National Accounts Data Files. 2023\. “GDP per capita growth (annual %).”Technical report, The World Bank, https://data.worldbank.org/indicator/NY.GDP.PCAP.KD.ZG, Accessed: 2022-09-21 [Dataset].
  * Xu (2017) Xu, Yiqing. 2017\. “Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models.” Political Analysis 25 (1): 57–76.  10.1017/pan.2016.2.
  * Zarantonello (1971) Zarantonello, Eduardo H. 1971\. Projections on Convex Sets in Hilbert Space and Spectral Theory: Part I. Projections on Convex Sets: Part II. Spectral Theory. 237–424, New York, NY: Elsevier, .  https://doi.org/10.1016/B978-0-12-775850-3.50013-3.