# On the implementation of Approximate Randomization Tests in Linear Models
with a Small Number of Clusters††thanks: We would like to thank Matthew Thomas
for excellent research assistance developing the R and Stata packages for this
paper. The research of the fourth author is supported by NSF Grant
SES-1530661.

Yong Cai
Department of Economics
Northwestern University
<EMAIL_ADDRESS>Ivan A. Canay
Department of Economics
Northwestern University
<EMAIL_ADDRESS>Deborah Kim
Department of Economics
Northwestern University
<EMAIL_ADDRESS>Azeem M. Shaikh
Department of Economics
University of Chicago
<EMAIL_ADDRESS>
###### Abstract

This paper provides a user’s guide to the general theory of approximate
randomization tests developed in Canay et al. (2017a) when specialized to
linear regressions with clustered data. An important feature of the
methodology is that it applies to settings in which the number of clusters is
small – even as small as five. We provide a step-by-step algorithmic
description of how to implement the test and construct confidence intervals
for the parameter of interest. In doing so, we additionally present three
novel results concerning the methodology: we show that the method admits an
equivalent implementation based on weighted scores; we show the test and
confidence intervals are invariant to whether the test statistic is
studentized or not; and we prove convexity of the confidence intervals for
scalar parameters. We also articulate the main requirements underlying the
test, emphasizing in particular common pitfalls that researchers may
encounter. Finally, we illustrate the use of the methodology with two
applications that further illuminate these points: one to a linear regression
with clustered data based on Meng et al. (2015) and a second to a linear
regression with temporally dependent data based on Munyo and Rossi (2015). The
companion R and Stata packages facilitate the implementation of the
methodology and the replication of the empirical exercises.

KEYWORDS: Randomization tests, linear regression, clustered data, time series

JEL classification codes: C12, C14

##  1 Introduction

This paper provides a user’s guide to the general theory of approximate
randomization tests (ARTs) developed in Canay et al. (2017a) when specialized
to linear regressions with clustered data. Here, clustered data refers to data
that may be grouped so that there may be dependence within each cluster, but
distinct clusters are approximately independent in a way to be made precise
below. Such data is remarkably common, including not only data that are
naturally grouped into clusters, such as villages or repeated observations
over time on individual units, but also data with weak temporal dependence, in
which pseudo-clusters may be formed using blocks of consecutive observations.
An important feature of the methodology is that it applies to commonly
encountered settings in which the number of clusters is small – even as small
as five. In this respect, the proposed methodology contrasts sharply and
meaningfully with many commonly employed methods for inference in such
settings. We briefly elaborate on this point in our discussion of related
literature below.

A principal goal of this paper is to make the general theory developed in
Canay et al. (2017a) more accessible by providing a step-by-step algorithmic
description of how to implement the test and construct confidence intervals
for the quantity of interest in these types of settings. In order to do so, we
develop three novel results concerning the methodology in Section 3. Our first
result shows that what we view as the most natural implementation of the test,
as described in Algorithm 2.1, is numerically equivalent to an alternative
implementation based on weighted scores (see Algorithm 3.1). Our second result
shows that when the parameter of interest is a scalar parameter, studentizing
or not the $t$-statistic entering the test does not affect the results of the
test or the associated confidence intervals. We therefore focus on the
unstudentized statistic in Algorithm 2.1. Finally, our third result shows that
the confidence sets for scalar parameters that are conceptually described by
test inversion are indeed a closed interval of the real line. This further
leads to a simple closed form expression for the lower and upper bound of the
confidence intervals (see Algorithm 3.1). These results are new to this paper
and play an important role in developing simple algorithms for the
implementation of ARTs.

We additionally provide a discussion of the main requirements underlying the
test in Section 4. These requirements essentially demand that the quantity of
interest is suitably estimable cluster-by-cluster. As discussed further in
Section 4, when this is not satisfied, a researcher need not conclude that it
is not possible to exploit the results in Canay et al. (2017a). Instead,
several remedies are possible, including clustering more coarsely or changing
the specification to ensure that this requirement is satisfied. We provide two
applications that further elucidate these points: one to a linear regression
with clustered data based on Meng et al. (2015) and a second to a linear
regression with temporally dependent data based on Munyo and Rossi (2015). The
required software to replicate these empirical exercises and to aid
researchers wishing to employ the methods elsewhere is provided in both R and
Stata.111The Stata and R packages ARTs can be downloaded from
http://sites.northwestern. edu/iac879/software/.

The methodology described in this paper is part of a large and active
literature on inference with clustered data. Following Bertrand et al. (2004),
researchers are acutely aware of the need to adjust inferences appropriately
to account for this sort of dependence. Many of the most commonly employed
methods for doing so, however, are inadequate for the unusually common
situation in which the number of clusters is small. Conventional wisdom
suggests that the number of clusters is small when it is less than forty. For
example, the method described in Liang and Zeger (1986), which has enjoyed
considerable popularity due to its availability in software packages such as
Stata, is widely acknowledged to perform poorly when this rule-of-thumb is not
satisfied. Similarly, the cluster wild bootstrap described in Cameron et al.
(2008) requires either a sufficiently large number of clusters or, as shown by
Canay et al. (2020), stringent homogeneity across clusters, to perform
reliably. As explained further in Section 4, the methods developed in Canay et
al. (2017a) and described in this paper, require neither a large number of
clusters nor such homogeneity across clusters. We note that the methods by
Ibragimov and Müller (2010, 2016), which are closely related to the ones
described here, also do not require such restrictions, but are generally less
powerful and permit testing a less rich variety of hypotheses. See Canay et
al. (2017a) for further discussion of these points as well as Conley et al.
(2018) for an insightful and thorough review of the related literature more
broadly.

The remainder of this paper is organized as follows. In Section 2, we first
formalize the setting and establish some notation. We then describe the
implementation of approximate randomization tests (ARTs) in an algorithmic
fashion, including how to use these tests to construct confidence intervals
for the quantity of interest. In Section 3 we present three results that play
an important role in developing these algorithms. In Section 4, we articulate
the main requirements underlying the tests and discuss remedies for cases
where these requirements are not satisfied. Our two empirical applications are
contained in Section 5. Finally, we provide some concluding remarks in Section
6.

##  2 Review of ARTs in regression models

We start by reviewing the inference approach proposed by Canay et al. (2017a)
in the context of a linear regression model with clustered data. In order to
do so, we index clusters by $j\in J\equiv\\{1,\ldots,q\\}$ and units in the
$j$th cluster by $i\in I_{n,j}\equiv\\{1,\ldots,n_{j}\\}$. We also denote by
$n=\sum_{j=1}^{q}n_{j}$ the total number of observations. The observed data
consists of an outcome of interest, $Y_{i,j}$, and a vector of covariates,
$Z_{i,j}\in\mathbf{R}^{d_{z}}$, that are related through the equation

$Y_{i,j}=Z_{i,j}^{\prime}\beta+\epsilon_{i,j}~{},$ (1)

where $\beta\in\mathbf{R}^{d_{z}}$ are unknown parameters and our requirements
on $\epsilon_{i,j}$ are explained below in Section 4. Our goal is to test

$H_{0}:c^{\prime}\beta=\lambda\quad\text{vs.}\quad
H_{1}:c^{\prime}\beta\neq\lambda~{},$ (2)

for given values of $c\in\mathbf{R}^{d_{z}}$ and $\lambda\in\mathbf{R}$, at
level $\alpha\in(0,1)$. An important special case of this framework is a test
of the null hypothesis that a particular component of $\beta$ equals a given
value, i.e.,

$H_{0}:\beta_{\ell}=\lambda\quad\text{vs.}\quad
H_{1}:\beta_{\ell}\neq\lambda~{},$

for some $\ell\in\\{1,\dots,d_{z}\\}$, by simply setting $c$ to be a standard
unit vector with a one in the $\ell$th component and zeros otherwise. More
generally, the approach we describe below extends immediately to the case
where the hypothesis of interest involves multiple elements of $\beta$, in
which case the test becomes

$H_{0}:R\beta=\Lambda\quad\text{vs.}\quad H_{1}:R\beta\neq\Lambda~{},$ (3)

for a given $p\times d_{z}$-dimensional matrix $R$ and $p$-dimensional vector
$\Lambda$, at level $\alpha\in(0,1)$.

ARTs were developed more generally in Canay et al. (2017a) and admit a variety
of different applications that go beyond the linear model considered here. For
example, the method accommodates non-linear models, non-linear hypotheses, or
even applications that go beyond inference with a small number of clusters
(e.g., Canay and Kamat (2018) develop a variation that applies to inference in
the regression discontinuity design). Here, we abstract away from the
generality of the method and focus on the steps needed to use ARTs to test the
null hypothesis in (2) in the context of the model in (1).

###  2.1 How to implement ARTs

The most straightforward way to test the hypotheses in (2) via ARTs is by
following the steps described in Algorithm 2.1 below.

######  Algorithm 2.1 (ARTs via within-cluster estimates).

This implementation of ARTs involves the following steps:

  1. Step 1: For each cluster $j\in J$, run an ordinary least squares regression of $Y_{i,j}$ on $Z_{i,j}$ using the $n_{j}$ observations in cluster $j$. Denote the corresponding estimators of $\beta$ by

$\\{\hat{\beta}_{n,j}:j\in J\\}~{}.$

  2. Step 2: For each $j\in J$, define the random variables

$S_{n,j}\equiv\sqrt{n_{j}}(c^{\prime}\hat{\beta}_{n,j}-\lambda)~{},$ (4)

and then construct the test statistic

$T_{n}=\Big{|}\frac{1}{q}\sum_{j=1}^{q}S_{n,j}\Big{|}~{}.$ (5)

  3. Step 3: Let $\mathbf{G}=\\{1,-1\\}^{q}$, so $g=(g_{1},\dots,g_{q})\in\mathbf{G}$ is simply a $q$-dimensional vector with elements $g_{j}$ being either $1$ or $-1$. For any element $g\in\mathbf{G}$, define

$T_{n}(g)=\Big{|}\frac{1}{q}\sum_{j=1}^{q}g_{j}S_{n,j}\Big{|}~{}.$ (6)

  4. Step 4: Compute the $1-\alpha$ quantile of $\\{T_{n}(g):g\in\mathbf{G}\\}$ as

$\hat{c}_{n}(1-\alpha)\equiv\inf\left\\{u\in\mathbf{R}:\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\\{T_{n}(g)\leq
u\\}\geq 1-\alpha\right\\}~{}.$ (7)

  5. Step 5: Compute the test as

$\phi_{n}\equiv I\\{T_{n}>\hat{c}_{n}(1-\alpha)\\}~{},$ (8)

where $T_{n}$ is as in (5) and $\hat{c}_{n}(1-\alpha)$ is as in (7). The
associated $p$-value is

$\hat{p}_{n}\equiv\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\\{T_{n}(g)\geq
T_{n}\\}~{},$ (9)

where $T_{n}(g)$ is as in (6).

Algorithm 2.1 involves five steps that are easy to implement from a
computational standpoint, but some of the steps deserve some clarification.
Step 1 involves $q$ within-cluster regressions that lead to $q$ estimates of
$\beta$. This essentially demands that the parameter $\beta$ is identified
cluster-by-cluster, and may fail to hold if some of the variables in the
vector $Z_{i,j}$ are constant within cluster. We discuss possible remedies for
this problem in Section 4 and illustrate their use in one of the applications
in Section 5. An important feature of the method is that from Step 2 onwards,
the original data is no longer needed as all the calculations only involve the
$q$ estimators of the parameter $\beta$ obtained in Step 1.

Step 2 defines a type of unstudentized $t$-statistic that is appropriate for
the null hypothesis in (2). We discuss the connection to its studentized
version in Section 3.2 below. If the null hypothesis of interest is the one in
(3), then a Wald-type test statistic could be used instead, i.e.,

$T_{n}^{\rm wald}\equiv
q\Big{(}\frac{1}{q}\sum_{j=1}^{q}S_{n,j}\Big{)}^{\prime}\Sigma_{S}^{-1}\Big{(}\frac{1}{q}\sum_{j=1}^{q}S_{n,j}\Big{)}~{},$
(10)

where

$S_{n,j}\equiv\sqrt{n}(R\hat{\beta}_{n,j}-\Lambda)\quad\text{ and
}\quad\Sigma_{S}\equiv\frac{1}{q}\sum_{j=1}^{q}S_{n,j}S_{n,j}^{\prime}~{}.$

Step 3 does not require one to recompute the estimates of $\beta$. It rather
uses the $q$ estimates from Step 1 and applies sign changes to the
$q$-dimensional vector $\\{S_{n,j}:j\in J\\}$. Since the cardinality of
$\mathbf{G}$ is $|\mathbf{G}|=2^{q}$, it exceeds $2000$ when $q>10$ and in
such cases it may be convenient to use a stochastic approximation. This may be
done while still controlling the rejection probability under the null
hypothesis (see Canay et al., 2017a, Remark 2.2). Formally, in this case we
let

$\hat{\mathbf{G}}\equiv\\{g^{1},\dots,g^{B}\\}~{},$ (11)

where $g^{1}=(1,\dots,1)$ is the identity vector and
$g^{b}=(g^{b}_{1},\dots,g^{b}_{q})$, for $b=2,\dots,B$, are i.i.d. Rademacher
random variables; i.e., each $g^{b}_{j}$ equals $\pm 1$ with equal
probability. We then implement Algorithm 2.1 with $\hat{\mathbf{G}}$ replacing
$\mathbf{G}$ everywhere and set $B=1000$ (or any other reasonably large number
chosen by the analyst).

Step 4 requires computing the $1-\alpha$ quantile of
$\\{T_{n}(g):g\in\mathbf{G}\\}$, which can be typically obtained by sorting
the values of $\\{T_{n}(g):g\in\mathbf{G}\\}$ and then taking the
$\lceil|\mathbf{G}|(1-\alpha)\rceil^{\rm th}$ highest element in the ordered
list. Thus, if we denote the ordered values of $\\{T_{n}(g):g\in\mathbf{G}\\}$
by

$T^{(1)}_{n}\leq T^{(2)}_{n}\leq\cdots\leq T^{(B)}_{n}~{},$

then we may define $\hat{c}_{n}(1-\alpha)$ in (7) as
$\hat{c}_{n}(1-\alpha)=T^{(\lceil|\mathbf{G}|(1-\alpha)\rceil)}$. This
representation suggests that the test may have trivial power for very low
values of $q$. For example, when $\alpha=10\%$, this problem arises if $q\leq
4$. For $q=5$ the test already has non-trivial power and is only slightly
conservative under the null. Similarly, when $\alpha=5\%$ the test has non-
trivial power for any $q\geq 6$.

Step 5 is straightforward and it provides both the test $\phi_{n}$ and the
$p$-value $\hat{p}_{n}$. Each of these correspond to the non-randomized
version of ARTs as opposed to their randomized counterparts (see Remark 2.4 in
Canay et al., 2017a) since practitioners often prefer tests that do not
involve exogenous randomness. In any case, the differences between the
randomized and non-randomized versions of the test have been found to be
minimal in simulations (see, e.g., Canay et al., 2017a).

###  2.2 How to compute confidence intervals

We now discuss how to compute confidence intervals for the parameter
$c^{\prime}\beta$ by developing a novel algorithm that exploits the properties
derived in Section 3.3. As before, a particularly important case is when $c$
selects the $\ell$th component of $\beta$ and then the confidence set is
simply a confidence interval for $\beta_{\ell}$. Conceptually we can simply
form the confidence set by collecting all values of $c^{\prime}\beta$ that
cannot be rejected by our test at level $\alpha$. That is, for the test
$\phi_{n}$ in (8) we define

$C_{n}=\\{\lambda\in\mathbf{R}:\phi_{n}=0\text{ when testing
}H_{0}:c^{\prime}\beta=\lambda\\}~{}.$ (12)

In an asymptotic framework where $n\to\infty$ while $q$ remains fixed, Canay
et al. (2017a) show that $\phi_{n}$ is asymptotically level $\alpha$ under
$H_{0}$. It follows from that result that, by construction, $C_{n}$ covers
$c^{\prime}\beta$ with probability at least equal to $1-\alpha$
asymptotically. In Section 3.3 we show that $C_{n}$ is indeed a closed
interval in $\mathbf{R}$ and so it takes the form

$C_{n}=[\lambda_{l},\lambda_{u}]~{},$ (13)

where $\lambda_{l}$ is the _smallest_ value of $\lambda$ that cannot be
rejected by $\phi_{n}$ and $\lambda_{u}$ is the _largest_ value of $\lambda$
that cannot be rejected by $\phi_{n}$. The analysis in Section 3.3 also
reveals that $\lambda_{l}$ and $\lambda_{u}$ admit simple closed-form
representations that we exploit to develop Algorithm 2.2 below.

######  Algorithm 2.2 (ART-based confidence intervals for $c^{\prime}\beta$).

For $\\{\hat{\beta}_{n,j}:j\in J\\}$ as defined in Step 1 of Algorithm 2.1,
the construction of the confidence interval involves the following steps:

  1. Step 1: For every $g\in\mathbf{G}$, compute the following objects,

$a(g)\equiv\frac{1}{q}\sum_{j=1}^{q}\sqrt{n_{j}}g_{j}~{},\quad
b(g)\equiv\frac{1}{q}\sum_{j=1}^{q}\sqrt{n_{j}}g_{j}c^{\prime}\hat{\beta}_{n,j}~{},\quad\text{and}\quad\lambda_{0}\equiv\frac{b(\iota)}{a(\iota)}~{},$
(14)

where $\iota=(1,\dots,1)\in\mathbf{G}$ is the vector with all ones.

  2. Step 2: For every $g\in\mathbf{G}$ define

$\lambda_{l}(g)\equiv\begin{cases}\frac{b(\iota)}{a(\iota)}\frac{|a(\iota)|}{|a(\iota)|+|a(g)|}+\frac{b(g)}{a(g)}\frac{|a(g)|}{|a(\iota)|+|a(g)|}&\text{
if }\frac{b(g)}{a(g)}\leq\lambda_{0}\text{ and }|a(g)|\neq 0\\\\[10.0pt]
\frac{b(\iota)}{a(\iota)}\frac{|a(\iota)|}{|a(\iota)|-|a(g)|}-\frac{b(g)}{a(g)}\frac{|a(g)|}{|a(\iota)|-|a(g)|}&\text{
if }\frac{b(g)}{a(g)}>\lambda_{0}\text{ and }|a(g)|\neq 0\\\\[10.0pt]
\frac{b(\iota)}{a(\iota)}-\frac{|b(g)|}{a(\iota)}&\text{ if
}|a(g)|=0\\\\[10.0pt] -\infty&\text{ if }g=\pm\iota\end{cases}~{}.$ (15)

and

$\lambda_{u}(g)\equiv\begin{cases}\frac{b(\iota)}{a(\iota)}\frac{|a(\iota)|}{|a(\iota)|+|a(g)|}+\frac{b(g)}{a(g)}\frac{|a(g)|}{|a(\iota)|+|a(g)|}&\mbox{
if }\frac{b(g)}{a(g)}\geq\lambda_{0}\text{ and }|a(g)|\neq 0\\\\[10.0pt]
\frac{b(\iota)}{a(\iota)}\frac{|a(\iota)|}{|a(\iota)|-|a(g)|}-\frac{b(g)}{a(g)}\frac{|a(g)|}{|a(\iota)|-|a(g)|}&\mbox{
if }\frac{b(g)}{a(g)}<\lambda_{0}\text{ and }|a(g)|\neq 0\\\\[10.0pt]
\frac{b(\iota)}{a(\iota)}+\frac{|b(g)|}{a(\iota)}&\mbox{ if
}|a(g)|=0\\\\[10.0pt] +\infty&\text{ if }g=\pm\iota\end{cases}~{}.$ (16)

  3. Step 3: Compute the lower bound $\lambda_{l}$ in the confidence interval (13) as the $\alpha$ quantile of $\left\\{\lambda_{l}(g)\,:\,g\in\mathbf{G}\right\\}$, i.e.,

$\lambda_{l}\equiv\inf\left\\{u\in\mathbf{R}:\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\\{\lambda_{l}(g)\leq
u\\}\geq\alpha\right\\}~{}.$ (17)

Compute the upper bound $\lambda_{u}$ in the confidence interval (13) as the
negative of the $\alpha$ quantile of
$\left\\{-\lambda_{u}(g)\,:\,g\in\mathbf{G}\right\\}$, i.e.,

$\lambda_{u}\equiv-\inf\left\\{u\in\mathbf{R}:\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\\{-\lambda_{u}(g)\leq
u\\}\geq\alpha\right\\}~{}.$ (18)

Report the confidence interval $C_{n}$ as in (13).

Algorithm 2.2 requires three steps that are straightforward to compute and
that exploit the results in Section 3.3. We refer the reader to that section
for the details on why $\lambda_{l}$ and $\lambda_{u}$ admit the expressions
in (17) and (18), respectively.

##  3 Three results on implementation of ARTs

Before we review the main requirement underlying ARTs, we present three
properties related to the implementation of ARTs that we believe practitioners
should be aware of and that are novel to this paper. The first property
establishes a connection between the implementation of ARTs as described in
Algorithm 2.1 and an alternative implementation based on weighted scores. The
second property establishes the numerical equivalence of ARTs for the null in
(2) when the test statistics in (5) is replaced by its studentized version.
The third and final result shows that ARTs confidence set for
$c^{\prime}\beta$ is indeed a closed interval in $\mathbf{R}$ and provides a
representation for the upper and lower bounds of the interval that lead to
Algorithm 2.2.

###  3.1 Equivalence with weighted scores

It turns out that ARTs can be implemented by an algorithm that does not
involve estimating the parameter $\beta$ within each cluster. This alternative
algorithm involves replacing Steps 1 and 2 in Algorithm 2.1 by the two
alternative steps described in Algorithm 3.1 below, while keeping Steps 3 to 5
unaffected.

######  Algorithm 3.1 (ARTs via within-cluster weighted scores).

This implementation of ARTs involves the following steps:

  * Step 1′: Run a full-sample least squares regression of $Y_{i,j}$ on $Z_{i,j}$ subject to the restriction imposed by the null hypothesis, i.e., $c^{\prime}\beta=\lambda$. Denote by $\hat{\epsilon}^{\rm r}_{i,j}$ the restricted residuals from this regression and by $\hat{\beta}^{\rm r}_{n}$ the restricted LS estimator of $\beta$.

  * Step 2′: For each cluster $j\in J$, define

$S_{n,j}\equiv
c^{\prime}\hat{\Omega}_{n,j}^{-1}\frac{1}{\sqrt{n_{j}}}\sum_{i\in
I_{n,j}}Z_{i,j}\hat{\epsilon}^{\rm r}_{i,j}~{},$ (19)

where

$\hat{\Omega}_{n,j}\equiv\frac{1}{n_{j}\textbf{}}\sum_{i\in
I_{n,j}}Z_{i,j}Z_{i,j}^{\prime}$ (20)

is a $d_{z}\times d_{z}$ matrix that is assumed to be full rank with inverse
$\hat{\Omega}_{n,j}^{-1}$.

  * Steps 3-5: Same as in Algorithm 2.1.

Note that Steps 3-5 remain unchanged given the alternative definition of
$S_{n,j}$ in Step 2′. When it comes to Steps 1 and 2, there are two
differences worth discussing. The first difference is that Step 1′ requires a
single full-sample restricted least squares estimator of $\beta$ as opposed to
the $q$ cluster-by-cluster estimators in Step 1 of Algorithm 2.1. The second
difference is that Step 2′ is based on within-cluster weighted scores as
opposed to the centered within-cluster estimates of $\beta$ in Step 2 of
Algorithm 2.1. Interestingly, these two implementations are numerically
equivalent and so implementing ARTs via Algorithm 2.1 or Algorithm 3.1 leads
to identical results. To see this formally, it is enough to show that
$S_{n,j}$ as defined in (4) and (19) are the same using the following
argument. For each $j\in J$,

$\displaystyle S_{n,j}$ $\displaystyle\equiv
c^{\prime}\hat{\Omega}_{n,j}^{-1}\frac{1}{\sqrt{n_{j}\textbf{}}}\sum_{i\in
I_{n,j}}Z_{i,j}\hat{\epsilon}^{\rm r}_{i,j}$
$\displaystyle=c^{\prime}\hat{\Omega}_{n,j}^{-1}\frac{1}{\sqrt{n_{j}}}\sum_{i\in
I_{n,j}}Z_{i,j}(Y_{i,j}-Z_{i,j}^{\prime}\hat{\beta}^{\rm r}_{n})$
$\displaystyle=c^{\prime}\hat{\Omega}_{n,j}^{-1}\frac{1}{\sqrt{n_{j}}}\sum_{i\in
I_{n,j}}Z_{i,j}Y_{i,j}-c^{\prime}\hat{\Omega}_{n,j}^{-1}\frac{1}{\sqrt{n_{j}}}\sum_{i\in
I_{n,j}}Z_{i,j}Z_{i,j}^{\prime}\hat{\beta}^{\rm r}_{n}$
$\displaystyle=\sqrt{n_{j}}(c^{\prime}\hat{\beta}_{n,j}-c^{\prime}\beta)-\sqrt{n_{j}}(c^{\prime}\hat{\beta}^{\rm
r}_{n}-c^{\prime}\beta)$
$\displaystyle=\sqrt{n_{j}}(c^{\prime}\hat{\beta}_{n,j}-\lambda)~{},$

where the fourth equality follows by adding and subtracting
$\sqrt{n_{j}}c^{\prime}\beta$ and the last equality holds because
$c^{\prime}\hat{\beta}_{n}^{\rm r}=c^{\prime}\beta=\lambda$ under the null
hypothesis in (2). It thus follows that $S_{n,j}$ in (4) and in (19) are
identical and so ARTs can be alternatively implemented via Algorithm 2.1 or
3.1.

###  3.2 Equivalence with studentized version of the _t_ -statistic

The ART defined in (8) of Algorithm 2.1 is based on the unstudentized test
statistic $T_{n}$ defined in (5). It may perhaps appear more desirable to
instead consider the studentized version of this test statistic as
studentization commonly improves performance in a variety of other settings.
Here, we prove that this is not the case for ARTs when the null hypothesis is
the one in (2) and that both versions of the test statistic lead to
numerically identical results.

To see this, start by defining the studentized version of the test statistic
in (5) as $T_{n}^{\rm s}\equiv T_{n}^{\rm s}(\iota)$, where for each
$g\in\mathbf{G}$,

$T_{n}^{\rm
s}(g)\equiv\sqrt{q}\frac{\Big{|}\frac{1}{q}\sum_{j=1}^{q}g_{j}S_{n,j}\Big{|}}{\hat{\sigma}_{\rm
s}(g)}\quad\text{ and }\quad\hat{\sigma}_{\rm
s}(g)\equiv\sqrt{\frac{1}{q}\sum_{j=1}^{q}\Big{(}g_{j}S_{n,j}-\frac{1}{q}\sum_{j=1}^{q}g_{j}S_{n,j}\Big{)}^{2}}~{}.$
(21)

Then note that

$\displaystyle\hat{\sigma}_{\rm s}^{2}(g)$
$\displaystyle=\frac{1}{q}\sum_{j=1}^{q}g_{j}^{2}S^{2}_{n,j}-\left(\frac{1}{q}\sum_{j=1}^{q}g_{j}S_{n,j}\right)^{2}=V_{n}-T_{n}^{2}(g)~{},$

where $V_{n}\equiv\frac{1}{q}\sum_{j=1}^{q}g_{j}^{2}S^{2}_{n,j}$ does not
depend on $g$ as $g_{j}^{2}=1$ for all $j\in J$. It follows that we can write
the studentized test statistic as

$\displaystyle T_{n}^{\rm
s}(g)=\sqrt{q}\frac{T_{n}(g)}{\sqrt{V_{n}-T_{n}^{2}(g)}}~{}.$

Since the function $x\mapsto\frac{x}{\sqrt{1-x^{2}}}$ is strictly increasing
for $x\in[0,1)$, it follows that $T_{n}^{\rm s}(g)$ is a strictly monotonic
transformation of $T_{n}(g)$ for each $g\in\mathbf{G}$. We conclude that
$I\\{T_{n}(g)\geq T_{n}(\iota)\\}=I\\{T_{n}^{\rm s}(g)\geq T_{n}^{\rm
s}(\iota)\\}$ for all $g\in\mathbf{G}$ and so the ART based on $T_{n}(g)$ and
$T_{n}^{\rm s}(g)$ leads to identical $p$-values.

###  3.3 Convexity of the confidence intervals

The ART-based confidence intervals for $c^{\prime}\beta$ defined in (12) can
be computed by test inversion. From a computational standpoint, however,
computing confidence sets by test inversion may be cumbersome and the
resulting set may not even be an interval. That is, it may not be closed and
convex. In this section we prove that this is not a concern for ART-based
confidence intervals for $c^{\prime}\beta$ and so such confidence intervals
could be easily computed by a standard bisection algorithm. In fact, our
results go even further. We derive closed form expressions for the lower and
upper bounds of the confidence interval that imply that computing ART-based
confidence intervals for $c^{\prime}\beta$ is straightforward from a
computational standpoint. In order to derive these results, we slightly change
our notation to make explicit the dependence on $\lambda$ of each of the
elements entering the test in (8).

In order to make the dependence of the test statistic in (5) on $\lambda$
explicit, let

$T_{n}(g,\lambda)\equiv\Big{|}\frac{1}{q}\sum_{j=1}^{q}g_{j}S_{n,j}(\lambda)\Big{|}\quad\text{
where }\quad
S_{n,j}(\lambda)=\sqrt{n_{j}}(c^{\prime}\hat{\beta}_{n,j}-\lambda)~{},$

and note that $T_{n}=T_{n}(\iota,\lambda)$. Using this notation, we can re-
write the confidence interval in (12) as

$C_{n}=\left\\{\lambda\in\mathbf{R}\,:\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\left\\{T_{n}(g,\lambda)\geq
T_{n}(\iota,\lambda)\right\\}\geq\alpha\right\\}~{},$

which is simply the values of $\lambda$ for which the $p$-value of the test,
as defined in (9), is not below $\alpha$. In order to show that this
confidence set is a closed interval, we claim that the $p$-value

$\hat{p}_{n}(\lambda)=\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\left\\{T_{n}(g,\lambda)\geq
T_{n}(\iota,\lambda)\right\\}$ (22)

is equal to $1$ for $\lambda_{0}\equiv b(\iota)/a(\iota)$, monotonically
increasing for any $\lambda<\lambda_{0}$, and monotonically decreasing for any
$\lambda>\lambda_{0}$. The next lemma formalizes this result.

######  Lemma 3.1.

Let $a(g)$, $b(g)$, and $\lambda_{0}$ be defined as in (14). The p-value in
(22) equals

$\hat{p}_{n}(\lambda)=\begin{cases}\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\left\\{\lambda\geq\lambda_{l}(g)\right\\}&\text{
for }\lambda<\lambda_{0}\\\\[3.0pt] 1&\text{ for
}\lambda=\lambda_{0}\\\\[3.0pt]
\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\left\\{\lambda\leq\lambda_{u}(g)\right\\}&\text{
for }\lambda>\lambda_{0}\end{cases}~{},$ (23)

where $\\{\lambda_{u}(g):g\in\mathbf{G}\\}$ and
$\\{\lambda_{l}(g):g\in\mathbf{G}\\}$ are defined in Algorithm 2.2.

###### Proof.

It is useful to re-write $T_{n}(g,\lambda)$ in terms of $a(g)$ and $b(g)$. To
this end, note that

$\displaystyle T_{n}(g,\lambda)$
$\displaystyle\equiv\Big{|}\frac{1}{q}\sum_{j=1}^{q}g_{j}S_{n,j}(\lambda)\Big{|}=\Big{|}\frac{1}{q}\sum_{j=1}^{q}g_{j}\sqrt{n_{j}}c^{\prime}\hat{\beta}_{n,j}-\lambda\frac{1}{q}\sum_{j=1}^{q}g_{j}\sqrt{n_{j}}\Big{|}$
$\displaystyle=|b(g)-\lambda a(g)|~{}.$ (24)

Given $g\in\mathbf{G}$ and $a(g)\neq 0$, $T_{n}(g,\lambda)$ is a “V-shaped”
function of $\lambda$ taking the value $0$ at $\frac{b(g)}{a(g)}$ and with
slope $-|a(g)|$ for all $\lambda<\frac{b(g)}{a(g)}$ and slope $|a(g)|\leq
a(\iota)$ for all $\lambda>\frac{b(g)}{a(g)}$. Figure 1 illustrates this for
three values of $g$.

First, note that $I\left\\{T_{n}(g,\lambda_{0})\geq
T_{n}(\iota,\lambda_{0})\right\\}=I\\{T_{n}(g,\lambda_{0})\geq 0\\}=1$ for all
$g\in\mathbf{G}$ and so it follows immediately that
$\hat{p}_{n}(\lambda_{0})=1$.

Second, restrict attention to the set
$\Lambda^{+}\equiv\\{\lambda\in\mathbf{R}:\lambda>\lambda_{0}\\}$ where
$T_{n}(\iota,\lambda)$ is linearly increasing. In order to prove that
$\hat{p}_{n}(\lambda)$ takes the form in (23) we prove that
$I\left\\{T_{n}(g,\lambda)\geq
T_{n}(\iota,\lambda)\right\\}=I\left\\{\lambda\leq\lambda_{u}(g)\right\\}$ for
each $g\in\mathbf{G}$ by dividing the argument into three cases.

Case 1: Consider $g\in\mathbf{G}$ such that $a(g)\neq 0$ and $|a(g)|\neq
a(\iota)$. Since $|a(g)|<a(\iota)$, it follows that $T_{n}(g,\lambda)$ and
$T_{n}(\iota,\lambda)$ intersect only once on $\Lambda^{+}$ and this holds
regardless of whether $\frac{b(g)}{a(g)}<\lambda_{0}$ or
$\frac{b(g)}{a(g)}\geq\lambda_{0}$ (see Figure 1 for a graphical intuition of
each of these cases). Denote the intersection point by $\lambda_{u}(g)$ and
note that $T_{n}(g,\lambda)\geq T_{n}(\iota,\lambda)$ for all
$\lambda_{0}<\lambda\leq\lambda_{u}(g)$ and
$T_{n}(g,\lambda)<T_{n}(\iota,\lambda)$ for all $\lambda>\lambda_{u}(g)$.
Conclude that on $\Lambda^{+}$,

$I\\{T_{n}(g,\lambda)\geq
T_{n}(\iota,\lambda)\\}=I\\{\lambda\leq\lambda_{u}(g)\\}~{}.$ (25)

Simple algebra shows that the intersection point $\lambda_{u}(g)$ takes the
form in (16).

Case 2: Consider $g\in\mathbf{G}$ such that $a(g)=0$. Note that
$b(\iota)-\lambda a(\iota)<0$ for $\lambda\in\Lambda^{+}$. It thus follows
that for $\lambda\in\Lambda^{+}$,

$I\\{T_{n}(g,\lambda)\geq
T_{n}(\iota,\lambda)\\}=I\\{|b(g)|\geq|b(\iota)-\lambda
a(\iota)|\\}=I\left\\{\lambda\leq\frac{b(\iota)}{a(\iota)}+\frac{|b(g)|}{a(\iota)}\right\\}~{},$

and so (25) holds in this case with
$\lambda_{u}(g)=\frac{b(\iota)}{a(\iota)}+\frac{|b(g)|}{a(\iota)}$, as defined
in (16).

Case 3: Consider $g\in\mathbf{G}$ such that $|a(g)|=a(\iota)$ and so
$g=\pm\iota$. If $g=\iota$, $I\\{T_{n}(g,\lambda)\geq
T_{n}(\iota,\lambda)\\}=1$ for all $\lambda\in\mathbf{R}$. We conclude that
(25) holds with $\lambda_{u}(g)=\infty$. If $g=-\iota$, then we have that
$a(-\iota)=-a(\iota)$ and $b(-\iota)=-b(\iota)$ so that
$\frac{b(-\iota)}{a(-\iota)}=\lambda_{0}$ and again
$I\left\\{T_{n}(-\iota,\lambda)\geq T_{n}(\iota,\lambda)\right\\}=1$ for all
$\lambda\in\mathbf{R}$. We conclude that (25) holds with
$\lambda_{u}(g)=\infty$, as defined in (16). This completes the proof of (23)
for the case $\lambda\in\Lambda^{+}$.

Finally, the construction for
$\lambda\in\Lambda^{-}\equiv\\{\lambda\in\mathbf{R}:\lambda<\lambda_{0}\\}$
parallels the one for $\lambda\in\Lambda^{+}$ so we omit the arguments here.
Putting all the cases together, (23) follows and this completes the proof.

$\lambda$$\lambda_{0}\
$$a(\iota)$$\frac{b(g_{1})}{a(g_{1})}$$|a(g_{1})|$$\frac{b(g_{2})}{a(g_{2})}$$-|a(g_{2})|$$\lambda_{u}(g_{1})$$\lambda_{u}(g_{2})$$T_{n}^{U}(g_{1},\lambda)$$T_{n}^{U}(g_{2},\lambda)$$T_{n}^{U}(\iota,\lambda)$$\lambda_{l}(g_{2})$$\lambda_{l}(g_{1})$

Figure 1: $T_{n}(g,\lambda)$ as functions of $\lambda$ for
$g\in\\{\iota,g_{1},g_{2}\\}$.

Figure 2 illustrates the $p$-value in (23) as a function of $\lambda$ for the
groups in Figure 1. Since $\hat{p}_{n}(\lambda)$ is right continuous and
increasing for $\lambda<\lambda_{0}$, we can define $\lambda_{l}$ as the
smallest value of $\lambda$ for which $\hat{p}_{n}(\lambda)\geq\alpha$. Such
value exists and is unique. Similar, since $\hat{p}_{n}(\lambda)$ is left
continuous and decreasing for $\lambda>\lambda_{0}$, we can define
$\lambda_{u}$ as the largest value of $\lambda$ for which
$\hat{p}_{n}(\lambda)\geq\alpha$. Such value exists and is again unique. This
argument leads to the representation of $C_{n}$ in (13), showing that ART-
based confidence intervals for $c^{\prime}\beta$ are indeed intervals in
$\mathbf{R}$. Furthermore, note that (23) implies that the smallest value of
$\lambda$ for which $\hat{p}_{n}(\lambda)\geq\alpha$ can be defined as

$\inf\left\\{\lambda\in\mathbf{R}:\frac{1}{|\mathbf{G}|}\sum_{g\in\mathbf{G}}I\left\\{\lambda\geq\lambda_{l}(g)\right\\}\geq\alpha\right\\}~{},$

which is just the definition of the $\alpha$ quantile of $\lambda_{l}(g)$, as
defined in Algorithm 2.2. A similar result holds for $\lambda_{u}$ and so
$C_{n}$ can be computed in closed form by Algorithm 2.2.

$\lambda$$\lambda_{0}\
$$\lambda_{u}(g_{1})$$\lambda_{u}(g_{2})$$\lambda_{l}(g_{2})$$\lambda_{l}(g_{1})$$\hat{p}_{n}(\lambda)$$1$
Figure 2: $\hat{p}_{n}(\lambda)$ as a function of $\lambda$.

##  4 What we need for ARTs to work

The main requirement underlying ARTs is Assumption 3.1 in Canay et al.
(2017a). This assumption guarantees that the test delivers rejection
probabilities under the null hypothesis that are close to the nominal level
$\alpha$ in an asymptotic framework where $n\to\infty$ and $q$ remains fixed.
In the context of the linear model in (1), this translates into the following
two conditions. The first condition is that the cluster-by-cluster estimators
of $\beta$ jointly converge in distribution at some (possibly unknown) rate;
i.e.,

$\left(\begin{array}[]{c}a_{n,1}(\hat{\beta}_{n,1}-\beta)\\\ \vdots\\\
a_{n,q}(\hat{\beta}_{n,q}-\beta)\\\
\end{array}\right)\overset{d}{\to}\left(\begin{array}[]{c}S_{1}\\\ \vdots\\\
S_{q}\\\ \end{array}\right)$ (26)

for a sequences $a_{n,j}\to\infty$ and random variables
$(S_{1},\dots,S_{q})^{\prime}$. The second condition requires that the
limiting random variables are invariant to sign changes, i.e.,

$(g_{1}S_{1},\dots,g_{q}S_{q})\overset{d}{=}(S_{1},\dots,S_{q})~{},$ (27)

for any $g$ in $\mathbf{G}$, where $\mathbf{G}$ is defined in Step 4 of
Algorithm 2.1.

Condition (26) holds, for example, when $Z_{i,j}$ and $\epsilon_{i,j}$ are
uncorrelated and the analyst assumes some form of weak dependence within
clusters that permits the application of an appropriate central limit theorem.
In such a case, (26) typically holds with $a_{n,j}=\sqrt{n_{j}}$ and each
$S_{j}$ being a mean-zero normal random variable. In fact, under the commonly
used assumption of independent clusters, it also follows that
$S_{j}\perp\\!\\!\\!\\!\perp S_{j^{\prime}}$ for any $j\neq j^{\prime}$. In
this case the normally distributed random variables may not be identically
distributed but are indeed independent. Condition (27), in turn, requires each
$S_{j}$ to be symmetrically distributed around zero and independent of each
other. This is immediately satisfied when each $S_{j}$ is a mean-zero normal
random variable and clusters are independent. Importantly, these assumptions
allow for the normally distributed random variables to have different
variances across clusters; a type of heterogeneity not allowed by the cluster
wild bootstrap approach popularized by Cameron et al. (2008) and later studied
formally by Canay et al. (2020).

######  Remark 4.1.

We focus our exposition on the case where $Z_{i,j}$ is exogenous but we
emphasize that the conditions in (26) and (27) typically hold in instrumental
variable (IV) models. Accommodating IV to ARTs then only requires modifying
Step 1 in Algorithm 2.1 so that the least squares regression is replaced with
the appropriate IV regression. Steps 2-6 remain unaffected.

An implicit requirement behind ARTs that deserves further comments lies in
Step 1 of Algorithm 2.1, which requires that the analyst runs cluster-by-
cluster regressions. This step implicitly assumes that the parameter $\beta$
is identified within each cluster. In practice, this means that the matrix
$\hat{\Omega}_{n,j}$ in (20) must be invertible and hence the same requirement
applies to Algorithm 3.1. This restriction may be substantially important in
some applications and so here we discuss common ways in which the problem may
manifest and two alternative remedies.

One case in which running least squares cluster-by-cluster is not feasible is
when the coefficient of interest is associated with a variable that only
varies across clusters. For example, consider the model in (1) and partition
$Z_{i,j}$ into a constant term, a scalar variable that only varies across
clusters, $Z^{(1)}_{j}$, and another variable that varies across and within
clusters, $Z^{(2)}_{i,j}$. That is,

$Y_{i,j}=\beta_{0}+Z^{(1)}_{j}\beta_{1}+Z^{(2)}_{i,j}\beta_{2}+\epsilon_{i,j}~{},$
(28)

where the analysts’ interest lies in the coefficient $\beta_{1}$, i.e.,
$c^{\prime}\beta=\beta_{1}$. Clearly, the regression in Step 1 of Algorithm
2.1 would not separately identify $\beta_{0}$ and $\beta_{1}$ as $Z^{(1)}_{j}$
is perfectly colinear with the constant term. The matrix $\hat{\Omega}_{n,j}$
in (20) is simply singular. This situation arises, for example, in the
empirical application considered by Canay et al. (2017b) where $j\in J$
indexes schools and the variable of interest is a treatment indicator at the
school level. A natural remedy in a situation like this is clustering more
coarsely (e.g., by combining clusters) to obtain variation within the re-
defined clusters. This is possible for ARTs since the validity of the method
does not rely on having a large number of clusters and thus it can afford to
work with coarser clustering. In fact, in certain settings combining clusters
may be quite natural. For example, Canay et al. (2017b) re-defined clusters as
“pairs” of schools (as opposed to just schools) given that the treatment
assignment mechanism of the experiment was a matched pairs design and so the
pairs used at the randomization stage represented natural groupings. In other
settings where it is less clear how to group clusters, any grouping that
satisfies the requisite identification condition leads to a valid test, but it
may be further desirable to combine such tests to limit concerns about “data
snooping” across groupings. To this end, results in DiCiccio et al. (2020) on
combining tests may be relevant.

######  Remark 4.2.

A quick inspection of (28) may lead the analyst to believe there is a
workaround that does not involve combining clusters if one instead uses some
estimator of $\beta_{0}$ from a full sample regression. For example, the full
sample least squares estimator $\hat{\beta}_{n,0}$ from the regression in (1).
Then, assuming for simplicity that $Z_{j}^{(1)}\neq 0$ for all $j\in J$, one
may consider modifying Step 1 in Algorithm 2.1 by running a regression of
$Y_{i,j}$ on an intercept and $Z^{(2)}_{i,j}$ (not including $Z_{j}^{(1)}$)
and then redefining $\\{\hat{\beta}_{n,j}:j\in J\\}$ as the difference between
the within cluster intercept estimates, $\hat{\beta}_{j,0}$ and the full
sample estimate $\hat{\beta}_{n,0}$, i.e.,
$\hat{\beta}_{n,j}=\hat{\beta}_{j,0}-\hat{\beta}_{n,0}$. Such strategies
unfortunately introduce dependence between the $q$ estimators of $\beta$ (as
they all depend on $\hat{\beta}_{n,0}$) and thus end up violating one of the
two main conditions needed for ARTs to be asymptotically valid; mainly
condition (27).

Another case where the lack of identification within cluster may manifest is
when the variable of interest actually varies within clusters but the model
specification involves other variables that are collinear with some other
variable (including the variable of interest or the constant term) within
clusters. For example, consider the model in (1) where instead of individuals
indexed by $i\in I_{n,j}$, units within cluster are indexed over time $t\in
T$. Partition $Z_{j,t}$ into the variable of interest, $Z^{(1)}_{j,t}$, and
time fixed effects $\delta_{t}$. That is,

$Y_{j,t}=Z^{(1)}_{j,t}\beta_{1}+\sum_{\tilde{t}\in
T}I\\{\tilde{t}=t\\}\delta_{\tilde{t}}+\epsilon_{j,t}~{}.$ (29)

It then follows that, within each cluster $j\in J$, the time fixed effect
$\delta_{t}$ absorbs all the variation in $Z^{(1)}_{j,t}$ and so $\beta_{1}$
is not identified. In cases like this the analyst could again combine clusters
to obtain variation within the re-defined clusters. An alternative remedy is
to change the specification by, for example, replacing the time fixed effect
with a cluster-specific time trend. Such specification is more restrictive
than the time fixed effect in the sense that it imposes a linear trend but, at
the same time, is more general as it allows for heterogeneity across clusters
in the linear trend. We illustrate this approach in the application we
consider in Section 5.1.

The need to identify $\beta$ within each cluster is in our view the main
limitation of ARTs, but a limitation that needs to be dealt with in certain
settings. One may then wonder why not simply use some other inference method
that is valid when the number of clusters is small and that does not rely on
estimating $\beta$ cluster-by-cluster. Perhaps the most popular approach in
that category is the cluster wild bootstrap popularized by Cameron et al.
(2008) and recently studied formally by Canay et al. (2020). While not having
to estimate $\beta$ within each cluster represents an advantage over ARTs,
this additional flexibility comes at a cost in terms of the degree of
heterogeneity that the model can deal with. In particular, the results in
Canay et al. (2020) show that the cluster wild bootstrap is expected to work
well in settings with a small number of clusters _as long as_ the clusters are
“homogeneous,” in a sense made precise in Canay et al. (2020). Intuitively, it
is required that the variance covariance matrix $\hat{\Omega}_{n,j}$ defined
in (20) is the same across clusters (up to scalar multiplication). Such
stringent homogeneity condition is not required for ARTs to work well, as the
method allows clusters to be arbitrarily heterogeneous as long as
$\hat{\Omega}_{n,j}$ is invertible for $j\in J$.

######  Remark 4.3.

For ease of exposition, we have written the requirement in (26) in terms of
the differences $\hat{\beta}_{n,j}-\beta$, but it is possible to replace it
with the differences $c’\hat{\beta}_{n,j}-c’\beta$ (or
$R\hat{\beta}_{n,j}-R\beta$, depending on the null hypotheses of interest). In
most cases, re-writing the condition in this way is not useful, but it is in
cases where $c’\beta$ is identified within each cluster while $\beta$ is not.
For example, consider the model in (28) when the coefficient of interest is
$\beta_{2}$ as opposed to $\beta_{1}$, i.e, $c^{\prime}\beta=\beta_{2}$. In
that case the entire term $\beta_{0}+Z_{j}^{(1)}\beta_{1}$ may be absorbed
into a cluster-specific intercept without affecting the identification and
estimation of $c^{\prime}\beta=\beta_{2}$ within each cluster.

##  5 Empirical applications

In this section we apply ARTs as described in Algorithm 2.1 and ART-based
confidence intervals as described in Algorithm 2.2 in the context of two
distinct empirical applications. The R and Stata packages and codes required
to replicate the results in this section are available as part of the online
supplemental material.

###  5.1 Meng, Qian and Yared (2015)

Meng et al. (2015, MQY) argue that China’s Great Famine, from 1959 to 1961,
was the result of an inflexible food procurement policy by the central
government. To make this point, they show that food production and mortality
become positively correlated during the time of famine, when this coefficient
is otherwise negative or not significantly different from 0 in normal times.

MQY consider the following regression,

$Y_{j,t+1}=Z^{(1)}_{j,t}\beta_{1}+Z^{(2)}_{j,t}\beta_{2}+\delta_{t}+\epsilon_{j,t}$

where $j$ indexes provinces (ranging from 1 to 19) and $t$ indexes years
(ranging from 1953 to 1982). Here,

$\displaystyle Y_{j,t+1}$ $\displaystyle=\log(\text{number of deaths in
province $j$ during year $t+1$})$ $\displaystyle Z^{(1)}_{j,t}$
$\displaystyle=\log(\text{predicted grain production in province $j$ during
year $t$})$ $\displaystyle\qquad\times I\\{\text{$t$ is a famine year}\\}$
$\displaystyle Z^{(2)}_{j,t}$ $\displaystyle=\log(\text{predicted grain
production in province $j$ during year $t$})$ $\displaystyle\delta_{t}$
$\displaystyle=\text{time fixed effects}~{}.$

In this application the level of clustering is a province, and so in order to
apply ARTs as described in Section 2.1, one needs to estimate
$\beta=(\beta_{1},\beta_{2})^{\prime}$ and $\delta_{t}$ province-by-province.
This illustrates one of the situations where including time fixed effects
province-by-province is infeasible for the implementation of ARTs, given that
the only source of remaining variation within a province is indeed time. The
second identification problem described in Section 4 then arises. As we
discussed in that section, one way to deal with this issue consists of
replacing the time fixed effects with a cluster-specific time trend, i.e., in
Step 1 of Algorithm 2.1 estimate

$Y_{j,t+1}=Z^{(1)}_{j,t}\beta_{1}+Z^{(2)}_{j,t}\beta_{2}+\gamma_{j}t+\epsilon_{j,t}~{}.$
(30)

We will refer to this as Analysis #1. In addition, we also consider the
following alternative specifications studied by MQY:

  * •

Analysis #2: Repeating Analysis #1 using only data between 1953 and 1965.

  * •

Analysis #3: Repeating Analysis #1 using four additional autonomous provinces.

  * •

Analysis #4: Repeating Analysis #2 using four additional autonomous provinces.

  * •

Analysis #5: Repeating Analysis #1 using actual rather than constructed grain
production.

  * •

Analysis #6: Repeating Analysis #2 using actual rather than constructed grain
production.

As with Analysis #1, the above analyses differ from their MQY counterparts
only in that a linear time trend $\gamma_{j}t$ replaces time fixed effects
$\delta_{t}$. Table 1 summarizes the number of clusters and the number of
observations for each of these analyses.

Analysis | # of Clusters | Min. Size | Med. Size | Max. Size | Mean
---|---|---|---|---|---
#1, #5 | 19 | 29 | 30 | 30 | 29.95
#2, #6 | 19 | 12 | 13 | 13 | 12.95
#3 | 23 | 29 | 30 | 30 | 29.96
#4 | 23 | 12 | 13 | 13 | 12.96

Table 1: Cluster Information. ‘Min. Size’, ‘Med. Size’, ‘Max. Size’ denote the
minimum, the median, and the maximum size of clusters.

Meng et al. (2015) consider the following two null hypotheses of interest,

$H_{0}^{(1)}:\beta_{1}=0\quad\text{ and }\quad
H_{0}^{(2)}:\beta_{1}+\beta_{2}=0~{}.$ (31)

In Table 2 we replicate the main table in Meng et al. (2015) using cluster
robust standard errors (CCE) and also include the results associated with ARTs
for both $H_{0}^{(1)}$ and $H_{0}^{(2)}$ in (31). For $H_{0}^{(1)}$ we report
$p$-values and 95% confidence intervals, while for $H_{0}^{(2)}$ we just
report $p$-values following MQY. The authors note in footnote 33 that using
the cluster wild bootstrap led to similar results as those presented in their
main table so we do not include cluster wild bootstrap results here either.

| #1 | #2 | #3 | #4 | #5 | #6
---|---|---|---|---|---|---
LS Estimate: $\beta_{1}$ | 0.063 | 0.057 | 0.071 | 0.067 | 0.064 | 0.058
CCE: Province |  |  |  |  |  |
se | 0.007 | 0.007 | 0.007 | 0.008 | 0.007 | 0.007
$p$-value | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
95% CI | [0.050, 0.077] | [0.043, 0.071] | [0.057, 0.086] | [0.051, 0.083] | [0.051, 0.078] | [0.044, 0.071]
ART |  |  |  |  |  |
$p$-value | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000
95% CI | [0.032, 0.055] | [0.018, 0.047] | [0.038, 0.066] | [0.028, 0.067] | [0.032, 0.058] | [0.029, 0.050]
$\beta_{1}+\beta_{2}=0$ |  |  |  |  |  |
CCE $p$-value | 0.050 | 0.009 | 0.059 | 0.005 | 0.266 | 0.363
ART $p$-value | 0.098 | 0.571 | 0.096 | 0.487 | 0.080 | 0.001
Observations | 569 | 246 | 689 | 298 | 569 | 246
Short Sample | No | Yes | No | Yes | No | Yes
Auto. Region | No | No | Yes | Yes | No | No
Pred. Grain Prod. | Yes | Yes | Yes | Yes | No | No

Table 2: Results for Analyses #1-6, comparable to those in Table 2 of Meng,
Qian and Yared (2015). ‘LS Estimate’ denotes the full sample OLS estimate for
$\beta_{1}$. CCE refers to cluster-robust standard errors. ART $p$-values are
obtained using Algorithm 2.1. ART-based 95% confidence intervals are obtained
using Algorithm 2.2.

We comment on the following main features of Table 2:

  1. 1.

For the null hypothesis $H_{0}^{(1)}$ associated with the parameter
$\beta_{1}$, the ART $p$-values are of comparable magnitude to traditional CCE
$p$-values. Similarly, ART-based confidence intervals are of roughly the same
length as those obtained based on CCE although the ART-based confidence
intervals do not contain the LS estimates. This is because ART-based
confidence intervals are centered around the mean of the province-by-province
estimates, which may not necessarily be equal to the full sample LS estimate
of $\beta_{1}$.

  2. 2.

For the null hypothesis $H_{0}^{(2)}$ associated with the parameter
$\beta_{1}+\beta_{2}$, the ART $p$-value is sometimes higher and sometimes
lower than the CCE $p$-value depending on the specification. Given the
relatively small number of clusters in this application, the ART $p$-values
are likely to be more reliable than those associated with CCE as CCE is known
to perform poorly when the number of clusters is not sufficiently large.

###  5.2 Munyo and Rossi (2015)

Munyo and Rossi (2015) study criminal recidivism of former prisoners by
looking at the relationship between the number of inmates released from
incarceration on a given day and the number of offenses committed on the same
day. They claim that the liquidity constraints that inmates face on the day of
release increase the likelihood of recidivism on the same day. Using data of
2631 days between January 1st 2004 and March 15 2011 collected from the
criminal incidents reports in Montevideo in Uruguay, they estimate the
following linear model by least squares

$\displaystyle Y_{t}=Z_{t}^{\prime}\beta+\epsilon_{t}$

where $t$ indexes days and

$\displaystyle Y_{t}$ $\displaystyle=\text{the total number of offenses on day
}t$ $\displaystyle Z_{t}$ $\displaystyle=\text{the total number of inmates
released, temperature, rainfall, hours of sunshine}$ on day $t$, a dummy for
holidays, a dummy for December 31st and a yearly trend.

We refer to this as Analysis #1. Munyo and Rossi (2015) additionally consider
the following four analyses:

  * •

Analysis #2: $Z_{t}$ includes a daily trend in place of a yearly trend.

  * •

Analysis #3: $Z_{t}$ includes a monthly trend in place of a yearly trend.

  * •

Analysis #4: $Z_{t}$ includes an intra-month daily trend, month- and year-
level fixed effects and their interactions in place of a yearly trend.

  * •

Analysis #5: $Z_{t}$ includes month- and year- level fixed effects and their
interactions in place of a yearly trend.

Analysis #5 is their preferred specification. Munyo and Rossi (2015) report
the results of these analyses in Table 2 in their paper. They report least
squares estimates of $\beta$ with Newey-West heteroskedasticity-
autocorrelation-consistent (HAC) standard errors. In addition, they report ART
$p$-values as described in Algorithm 2.1 for the null hypothesis that
$H_{0}:c^{\prime}\beta=0$ as in (2), where $c$ selects the coefficient on the
total number of inmates released on day $t$.

In this application the level of clustering is not naturally determined by the
data, but pseudo-clusters may be formed using blocks of consecutive
observations under the assumption of weak temporal dependence. In order to
apply ARTs as described in Algorithm 2.1 we then form $q$ pseudo-clusters by
dividing the data into $q$ consecutive blocks of size $b_{n}=\lfloor
n/q\rfloor$ where $n=2631$ is the number of total observations. More
concretely, we define the $j$th pseudo-cluster as

$\displaystyle
X^{(n)}_{j}=\\{(Y_{t},Z^{\prime}_{t})^{\prime}:t=(j-1)b_{n}+1,\cdots,jb_{n}\\}\quad\text{where}\quad
j=1,\cdots,q-1~{},$

and let the last $q$th pseudo-cluster contain all the remaining $n-b_{n}(q-1)$
observations. Note that in this application the number of pseudo-clusters $q$
is a tuning parameter that the analyst must specify. Munyo and Rossi (2015)
set $q=10$. We repeat their analyses with alternative values of $q$ and
investigate how sensitive the results are to this choice. The relevant cluster
information is given in Table 3.

# of Clusters (q) | Cluster Size
---|---
8 | 328
10 | 263
16 | 164
Table 3: Pseudo-cluster size for different values of $q$. Specification | #1 | #2 | #3 | #4 | #5
---|---|---|---|---|---
LS Estimate | 0.225 | 0.260 | 0.259 | 0.225 | 0.234
HAC |  |  |  |  |
se | 0.124 | 0.123 | 0.123 | 0.096 | 0.096
$p$-value | 0.068 | 0.034 | 0.034 | 0.019 | 0.015
95% CI | [-0.017, 0.468] | [0.02, 0.5] | [0.019, 0.5] | [0.038, 0.413] | [0.046, 0.421]
ART: q=8 |  |  |  |  |
$p$-value | 0.008 | 0.023 | 0.023 | 0.102 | 0.102
95% CI | [0.124, 0.429] | [0.035, 0.391] | [0.035, 0.391] | [-0.07, 0.397] | [-0.067, 0.418]
ART: q=10 |  |  |  |  |
$p$-value | 0.002 | 0.014 | 0.014 | 0.063 | 0.053
95% CI | [0.141, 0.603] | [0.068, 0.446] | [0.068, 0.458] | [-0.023, 0.431] | [-0.003, 0.452]
ART: q=16 |  |  |  |  |
$p$-value | 0.002 | 0.006 | 0.006 | 0.027 | 0.010
95% CI | [0.131, 0.444] | [0.097, 0.369] | [0.087, 0.371] | [0.02, 0.324] | [0.056, 0.367]
Observations | 2631 | 2631 | 2631 | 2631 | 2631
Time Trend | Year | Day | Month | Intra-month Day | None
Time Fixed Effect | No | No | No | Yes | Yes
Controls | No | No | No | No | No
Table 4: Results for Analyses #1-5, comparable to those in Table 2 of Munyo
and Rossi (2015). ‘LS Estimate’ denotes the full sample LS estimate of
$\beta$. HAC refers to the heteroskedasticity and autocorrelation consistent
standard error. ART $p$-values are obtained using Algorithm 2.1. ART-based 95%
confidence intervals are obtained using Algorithm 2.2.

Table 4 shows LS estimates of $\beta$, $p$-values for the hypothesis in (2),
and $95\%$ confidence intervals for each analysis. Following Munyo and Rossi
(2015), we report results based on HAC standard errors. The table also shows
ART $p$-values as described in Algorithm 2.1 and ART-based 95% confidence
intervals as described in Algorithm 2.2 for $q=8$, $q=10$, and $q=16$.

We summarize the main findings of the results in Table 4 as follows:

  1. 1.

The choice of $q$ is important for the results of ARTs but currently there is
no theory developed to choose this tuning parameter according to some data
dependent criteria. The smaller $q$ is, the more observations are available
within each cluster. Having more observations per cluster is important for one
of the requirements behind ARTs, mainly (26). A small value of $q$, however,
tends to affect the power of ARTs despite not really affecting the control of
the rejection probability under the null hypothesis. This feature can be seen
in Table 4, where ARTs $p$-values are decreasing in $q$ across different
specifications. In this application, where there are still over a hundred
observations when $q=16$, a larger value of $q$ like $q=10$ or $q=16$ may be
preferable to smaller values, like $q=8$, based on power considerations. Note,
however, that except in Analyses #4–5, where the choice of $q$ determines
whether the null hypothesis is rejected at a given significance level, the
results for Analyses #1–3 are in all agreement at a $5\%$ level.

  2. 2.

Overall, the test results based on standard $t$-test with HAC standard errors
are consistent to those of ARTs when $q=16$. Both methods reject the null
hypothesis $H_{0}:c^{\prime}\beta=0$ at a $10\%$ nominal level across
different specifications. The results support the authors’ argument that the
release of inmates from incarceration increase the chance of re-offenses on
the day of release.

##  6 Concluding remarks

The goal of this paper is to make the general theory developed in Canay et al.
(2017a) more accessible by providing a step-by-step algorithmic description of
how to implement the test and construct confidence intervals in linear
regression models with clustered data, as well as clarifying the main
requirements and limitations of the approach. The main two takeaways are the
following. First, ARTs-based confidence intervals for scalar parameters in
linear regression models can be characterized in closed form and thus are
straightforward to implement in practice. Algorithms 2.1 and 2.2 provide a
clear explanation of how to take ARTs to the data, and the companion Stata and
R packages available as part of the supplemental material are intended to
facilitate adoption of ARTs. Second, our discussion on the main requirements
behind ARTs hopefully show that understanding the trade-offs between ARTs and
other popular alternatives for inference with a small number of clusters, like
the cluster wild bootstrap, is fundamental for practitioners to choose a
method that aligns well with the features of their application. In particular,
while ARTs essentially demand that the parameter of interest is suitably
estimable cluster-by-cluster without imposing restrictions on the degree of
heterogeneity across clusters, the cluster wild bootstrap requires the
clusters to be sufficiently homogeneous (see Canay et al., 2020) without
demanding identification of the parameter of interest cluster-by-cluster.

## References

  * Bertrand et al. (2004) Bertrand, M., Duflo, E. and Mullainathan, S. (2004).  How much should we trust differences-in-differences estimates?  The Quarterly Journal of Economics, 119 249–275.
  * Cameron et al. (2008) Cameron, A. C., Gelbach, J. B. and Miller, D. L. (2008).  Bootstrap-based improvements for inference with clustered errors.  The Review of Economics and Statistics, 90 414–427.
  * Canay and Kamat (2018) Canay, I. A. and Kamat, V. (2018).  Approximate permutation tests and induced order statistics in the regression discontinuity design.  The Review of Economic Studies, 85 1577–1608.
  * Canay et al. (2017a) Canay, I. A., Romano, J. P. and Shaikh, A. M. (2017a).  Randomization tests under an approximate symmetry assumption.  Econometrica, 85 1013–1030.
  * Canay et al. (2017b) Canay, I. A., Romano, J. P. and Shaikh, A. M. (2017b).  Supplement to ‘Randomization tests under an approximate symmetry assumption’.  Econometrica Supplemental Material, 85.  http://dx.doi.org/10.3982/ECTA13081.
  * Canay et al. (2020) Canay, I. A., Santos, A. and Shaikh, A. M. (2020).  The wild bootstrap with a “small” number of “large” clusters.  The Review of Economics and Statistics.  Forthcoming.
  * Conley et al. (2018) Conley, T., Gonçalves, S. and Hansen, C. (2018).  Inference with dependent data in accounting and finance applications.  Journal of Accounting Research, 56 1139–1203.
  * DiCiccio et al. (2020) DiCiccio, C. J., DiCiccio, T. J. and Romano, J. P. (2020).  Exact tests via multiple data splitting.  Statistics & Probability Letters, 166 108865\.
  * Ibragimov and Müller (2010) Ibragimov, R. and Müller, U. K. (2010).  t-statistic based correlation and heterogeneity robust inference.  Journal of Business & Economic Statistics, 28 453–468.
  * Ibragimov and Müller (2016) Ibragimov, R. and Müller, U. K. (2016).  Inference with few heterogeneous clusters.  Review of Economics and Statistics, 98 83–96.
  * Liang and Zeger (1986) Liang, K.-Y. and Zeger, S. L. (1986).  Longitudinal data analysis using generalized linear models.  Biometrika, 73 13–22.
  * Meng et al. (2015) Meng, X., Qian, N. and Yared, P. (2015).  The institutional causes of china’s great famine, 1959–1961.  The Review of Economic Studies, 82 1568–1611.
  * Munyo and Rossi (2015) Munyo, I. and Rossi, M. A. (2015).  First-day criminal recidivism.  Journal of Public Economics, 124 81–90.


# ALERTA-Net: A Temporal Distance-Aware Recurrent Networks for Stock Movement
and Volatility Prediction

Shengkun Wang1, YangXiao Bai2, Kaiqun Fu2, Linhan Wang1,Chang-Tien Lu1, Taoran
Ji3
1Department of Computer Science, Virginia Tech
2Department of Computer Science, South Dakota State University
3Department of Computer Science, Texas A&M University - Corpus Christi
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

For both investors and policymakers, forecasting the stock market is essential
as it serves as an indicator of economic well-being. To this end, we harness
the power of social media data, a rich source of public sentiment, to enhance
the accuracy of stock market predictions. Diverging from conventional methods,
we pioneer an approach that integrates sentiment analysis, macroeconomic
indicators, search engine data, and historical prices within a multi-attention
deep learning model, masterfully decoding the complex patterns inherent in the
data. We showcase the state-of-the-art performance of our proposed model using
a dataset, specifically curated by us, for predicting stock market movements
and volatility.

###### Index Terms:

stock market prediction, twitter, google trends, sentiment analysis,
macroeconomic data

††publicationid: pubid:  Permission to make digital or hard copies of part or
all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or commercial advantage and
that copies bear this notice and the full citation on the first page.
Copyrights for third-party components of this work must be honored. For all
other uses, contact the Owner/Author. ASONAM ’23, November 6-9, 2023,
Kusadasi, Turkey
© Copyright is held by the owner/author(s).
ACM ISBN 979-8-4007-0409-3/23/11.
https://doi.org/10.1145/3625007.3627488

##  I Introduction

Significantly influencing other business sectors [1], the stock market serves
as a vital mechanism and is crucial for companies to raise capital. With U.S.
stock holdings expected to hit $40 trillion in 2023, equating to 1.5 times the
nation’s GDP, it stands as a major portion of the entire economy, highlighting
the stock market’s pivotal position as a benchmark for the U.S. economic
landscape. Our research centers on blue-chip stocks111Blue chip stocks are
shares issued by financially robust, well-established companies with stellar
reputations., which mirror the broader dynamics of the stock market.

We’ve selected 41 blue-chip stocks from 10 Global Industry Classification
Standard (GICS)222GICS classifies companies into specific economic sectors and
industry groups that most accurately represent their business operations.
Sectors for our financial market study. Each of these stocks is considered
investment-worthy333Companies rated Baa or higher by Moody’s and Standard &
Poor’s (S&P) are considered to be of high quality and deemed investment-
worthy. by both Moody’s and S&P. Recognizing the intrinsic challenge in
accurately predicting stock prices as highlighted by Nguyen et al.[2], we use
blue-chip stocks in our research to anticipate upcoming stock price movements
and volatility trends, as indicated by Feng et al.[3] and Xu et al. [4].

In the domain of stock market research, two primary methodologies prevail:
technical analysis and fundamental analysis. Technical analysis utilizes past
stock prices to predict future trends[5]. However, its heavy dependence on
historical data can sometimes overlook sudden market changes due to unexpected
events. Assuming a uniformly rational market behavior, this methodology can
inadvertently create an echo chamber. This effect can cause trading signals to
amplify themselves, eventually becoming disconnected from the actual economic
context. Conversely, fundamental analysis integrates both price features and
external information, including data from social media [6] and search engines
[7]. Mao et al.[8] demonstrated an enhanced accuracy in forecasting the S&P
500 closing price when integrating Twitter444Despite the recent rebranding of
Twitter to “X”, this article retains the use of its original name, “Twitter”.
data into their model. While these data sources frequently reflect not only
the financial market but also vital economic indicators, the prevailing
research in fundamental analysis tends to emphasize the financial market,
neglecting the symbiotic relationship between the broader economy and the
stock market. Moreover, while existing models mainly center on forecasting
trend shifts[9], they often neglect the importance of the scale of these
changes. In the realm of stock behavior, the magnitude of these shifts holds
significant weight.

In this paper, we propose ALERTA-Net: Attentional TemporaL DistancE AwaRe
RecurrenT NeurAl Networks. To our best knowledge, it is the first paper to use
the combination of social media, macroeconomic data and search engine
information to predict both stock price movement and volatility. Our
contributions can be summarized as follows:

• Proposing a framework enabling the fusion of social media, macroeconomic
factors and search engine data for stock movements and volatility. By
integrating above information with our method, we can not only predict stock
price movements but also efficiently extract information from stock market
volatility. This allows us to provide advance warning of any unusual
fluctuations in the stock market in the future.

• Formulating a temporal distance-aware, multi-attention mechanisms on multi-
view market data. The proposed ALERTA-Net shines in recognizing the dynamic,
temporal distance-based relationships inherent within difference hidden
states. Capitalizing the same day stock price movements, the model greatly
amplifies its precision in predicting stock market volatility.

• Validating the effectiveness and efficiency of the proposed model via
experiments and comparisons. We conduct experiments on one real-world
dataset555Our dataset is available at https://github.com/hao1zhao/ALERTA-Net.
Both conventional methods and deep learning based methods for stock market
movements and volatility are selected for comparisons. Evaluations of various
metrics are presented, illustrating the effectiveness of our proposed model.

##  II Related Work

Predictive methods for stock movement can be broadly categorized into two main
types: technical analysis and fundamental analysis. While technical analysis
relies exclusively on past price data to anticipate future trends, fundamental
analysis adopts a more comprehensive approach, considering not only historical
prices but also information from textual sources, economic indicators,
financial metrics, and a myriad of both qualitative and quantitative aspects.

Social Media: Numerous studies have explored stock market predictions using
social media data. Contemporary research combines sentiment analysis with
historical price data, extracting insights from platforms such as Yahoo’s
message board [10], blogs[11], Twitter[4], and Reddit[12]. This integration
has revealed correlations with stock market trends.

News and Search Engine: Exploring the influence of public news and user
browsing habits, research has probed into how traders respond to news events.
For instance, Xiong et al.[13] regard Google trends and market data as
catalysts for daily S&P 500 variations. Other research, works like Bordino et
al.[14] draw connections between search query volume and stock activity. More
sophisticated techniques, like the hierarchical attention mechanisms
introduced by Hu et al.[15], extract news sequences directly from textual
content to predict stock trends.

Macroeconomic Indicators: Numerous studies pinpoint various economic elements
influencing stock returns. Notably, Ferson et al. [16] highlight the central
role of interest rates in dictating stock returns. In addition, indicators
like the relative T-Bill rate and the consumption-wealth ratio are underscored
as crucial predictors by Jank et al. [17]. Beyond these, economic markers like
unemployment rates, inflation, and commodity prices also exert significant
influence on stock returns, as affirmed by sources like [18] and [19].

Figure 1: The architecture of ALERTA-Net is designed to predict the movement
$y_{m}^{t}$, and volatility $y_{v}^{t}$ on day $t$. In the data input and
preprocessing phase, we extract textual information from Twitter and convert
it into sentiment scores; Then, ALERTA-Net utilizes these scores, along with
other features, to make predictions, taking temporal distance into account.

##  III Problem Setup

Let $p=\\{p^{1},\ldots,p^{t}\\}$ denote the stock’s daily adjusted closing
price. We formulate the actual labels movement $y_{m}$ and volatility $y_{v}$
as follows:

$y_{m}^{t}=\mathbb{1}(p^{t}-p^{t-1}),$ (1)
$y_{v}^{t}=\mathbb{1}(p^{t}-p^{t-1})/p^{t-1}.$ (2)

The actual labels can be represented as
$y_{m}=\\{y_{m}^{1},\ldots,y_{m}^{t}\\}\in\mathbb{R}^{T}$ and $y_{v}\
=\\{y_{v}^{1},\ldots,y_{v}^{t}\\}\in\mathbb{R}^{T}$. Let
${X}^{T}=\\{x^{1},\ldots,x^{t}\\}\in\mathbb{R}^{D\times T}$ represents the
sequential input features (e.g., sentiment scores, stock adjusted prices) from
the previous $T$ time-steps, where $D$ signifies the dimension of the
features. Since our goal is to utilize the sequence of features ${X^{T}}$ to
predict the next time-step movements $\hat{y}_{m}$ and volatility
$\hat{y}_{v}$ of blue-chip stocks. We can define our prediction functions
$\hat{y}_{m}^{t}=f({X}^{T};{\Theta_{1}}),\hat{y}_{v}^{t}=g({X}^{T}$,
$\hat{y}_{m}^{t};{\Theta_{2}})$, where $f$ the function with parameters
${\Theta_{1}}$ aims to predict the movement of stock s at the next time-step
from the sequential features ${X}^{T}$ and $g$ aims to predict the unusual
fluctuation from both ${X}^{T}$ and $\hat{y}_{m}^{t}$.

In practical settings, by varying the time lag in extensive historical stock
data, we can often produce numerous training examples. However, for clarity in
presenting our proposed technique, we zero in on a distinct time lag for
predicting both movement and volatility. Additionally, our predictive models
have assimilated the concept of adjusted stock prices[20]. This ensures that
our models discern authentic stock value changes driven by market factors,
rather than getting influenced by artificial shifts arising from corporate
decisions.

##  IV Framework Components

In this paper, we introduce a new framework, ALERTA-Net, which incorporates
temporal distance-aware, multi-attention mechanisms for processing multi-view
stock data. The overall architecture is illustrated in Figure 1. The data
input & preprocessing layer transforms both temporal and textual information
into dense vectors. Then, our temporal distance-aware layer have the recurrent
representation identifies hidden dependencies within the current stock data,
based on past information. After that, distance-matrix context integrates
these historical dependencies within the sequence of features ${X}^{T}$.
Lastly, the predictions layer generates time-aware forecasts for the stock
movements and volatility in the next time-step, thereby providing a complete
and cohesive system for stock prediction.

Data input & preprocessing. We designate the textual data extracted from
Twitter as $\alpha$. To quantify the embedded sentiment within the Twitter
text, we utilize the roBERTa-base sentiment model in combination with TweetNLP
[21], facilitating the generation of tweet sentiment scores denoted as
$\hat{\alpha}$. Then, we concatenate $\hat{\alpha}$ with other relevant
historical data, yielding
$E^{T}=\\{e^{1},\ldots,e^{t}\\}\in\mathbb{R}^{D\times T}$, in which the
feature dimension is 17. Considering the fact that normalization provides a
uniform scale to all features, thereby precluding any specific feature from
dominating, we apply log normalization to $E^{T}$. For the purpose of avoiding
numerical instability, we add a small constant $\epsilon=1e-8$, the formula is
as follows:

${X}^{T}=\log{E^{T}+\epsilon}.$ (3)

Temporal distance aware (TDA). In this layer, we propose a temporal-distance
aware layer to enhance the modulation of conditional dependencies by allowing
the model to access and directly attend to previous hidden states. First,
given the proficiency in handling long-term dependencies, recurrent neural
network is extensively employed for sequential data processing [22]. The
general idea of recurrent unit is to recurrently project the input sequence
into a sequence of hidden representations. At each time-step, the recurrent
unit learns the hidden representations $h_{t}$ by jointly considering the
input $x_{t}$ and previous hidden representation $h_{t-1}$ to capture
sequential dependency. To capture the sequential dependencies and temporal
patterns in the historical stock features, an GRU[23] recurrent unit is
applied to map $\\{x^{1},\ldots,x^{t}\\}$ into hidden representations
$\\{h^{1},\ldots,h^{t}\\}\in\mathbb{R}^{U\times T}$, with the dimension of
$U$.

Instead of just using the immediate previous hidden state $h^{t}$ to update
the current hidden state $h^{t+1}$, we are now considering a weighted sum of
all previous hidden states. This not only enable the model to place greater
emphasis on the impact of recent events on the stock market but also to
consider a longer history. We utilized a temporal distance to represent the
interval between two time steps.

$w^{i}=\frac{1}{t-i+1},$ (4)

where the weight $w^{i}$ for a hidden state at time $i$ based on its temporal
distance from the current time $t$. By adding 1 to the denominator, we ensure
that even the most recent hidden state gets a weight, preventing division by
zero.

Then, we apply weights to the hidden states of the GRU:

$c^{t}=\text{GRU}\left(x^{t},\sum_{i=1}^{t}w^{i}\cdot h^{i}\right),$ (5)

each hidden state $h^{i}$ is multiplied by its respective weight $w^{i}$, and
the results are summed up. The result is a context state $c^{t}$ that captures
the weighted influence of all previous states.

Prediction Layer. Instead of directly predicting stock movement, denoted as
${y}_{m}^{t}$, and volatility ${y}_{v}^{t}$, we first concatenate the context
$c^{t}$ with the previous hidden state $h^{t}$ in the fusion layer. We then
use the cross-entropy function as the optimizer for stock price movement
prediction. Following this, we concatenate the last hidden state $h^{t}$,
context $c^{t}$, and the output ${y}_{m}^{t}$ to predict volatility
${y}_{v}^{t}$. For this prediction, we utilize the binary cross-entropy with
logits as the loss function.

##  V Experiment

###  V-A Dataset Description

Our dataset coalescest three main components: Historical price data, Twitter
data and macroeconomics data.

Yahoo Finance. We sourced historical data from Yahoo Finance, which monitored
the trajectory of 41 blue-chip stocks between June 1, 2020, and June 1, 2023.
To hone our prediction objectives, we designated a threshold range spanning
from -0.5% to 0.5% to filter out negligible shifts. While Baker et al.[24]
suggest daily stock price alterations exceeding 2.5% are deemed notable, our
model’s aim is to forecast atypical volatility. Aligning with Ding et al.’s
insights [25] on distinct stock fluctuation parameters, we’ve established a
loftier standard, recognizing a 5% swing as an outlier. As a result, we
categorize samples with variances below 5% as 0 and those at or above 5% as 1.

Twitter. During the same date range, we included approximately 7.8 million
tweets, gathered via Twitter’s official API at a sampling rate of 10%. We were
particular in our selection of tweets: they needed to contain at least one
cashtag and had to be posted within standard U.S. trading hours, from 9 am to
4:30 pm. We recognized the significant influence that Twitter volume has on
stock trading, a fact underscored by Cazzoli et al. [26]. Thus, we ensured
that our model’s input parameters included the daily count of processed
Twitter posts.

Google Trends & Federal Reserve Economic Data. We engaged in targeted searches
on Google and Federal Reserve Economic Data, using carefully curated keywords
originating from the “Outline of Economics” Wikipedia page. In light of the
disparate update intervals across various indicators, we broke down each data
pull into smaller segments, ensuring normalization of the data across these
windows.

###  V-B Baseline Methods and Evaluation Metrics

We evaluate our model’s effectiveness by comparing it with DP-LSTM[27], a
renowned stock movement prediction network by using financial data.

Other benchmarks employed in our study include Extreme Gradient Boosting[28],
Attention-based LSTM[29], and GRU[23]. Following similar procedures in (Xu et.
al [22]; Zhang et. al [9]), we report our results in terms of Accuracy (Acc.)
and Matthews Correlation Coefficient (MCC). Given that data points involving
stock price changes greater than 5% only constitute a minor portion of our
dataset, we’ve also chosen to utilize the Area Under the ROC Curve (AUC) as
our performance metric in order to achieve a more robust and realistic
evaluation.

TABLE I : Classification performance of ALERTA-NET and baseline methods,
measured with the accuracy (Acc.), the Matthews Correlation Coefficient (MCC)
and Receiver Operating Characteristic (ROC). The best is in bold, our ALERTA-
NET shows the best performance in all evaluation metrics.

Models | Movement | Volatility
---|---|---
Acc. |  MCC |  Acc. |  MCC |  AUC
XGBoost |  0.5030 |  0.01461 |  0.5240 |  -0.02275 |  0.5561
GRU |  0.5121 |  0.04217 |  0.5633 |  0.06241 |  0.5993
DP-LSTM |  0.5036 |  0.00741 |  0.5684 |  0.05297 |  0.5890
AT-LSTM |  0.5165 |  0.03779 |  0.5886 |  0.07028 |  0.5850
ALERTA-Net |  0.5238 |  0.05179 |  0.6136 |  0.07465 |  0.6197

TABLE II : An ablation study of ALERTA-Net, where ALERTA-Net(P) is baseline
without any tweet and macroeconomic data. ALERTA-Net(S) is only relies on
sentiment data. ALERTA-Net(W/O M) is without macroeconomic data. Both tweet
sentiment and macroeconomic data improve the performance of the framework.

Models | Movement | Volatility
---|---|---
Acc. |  MCC |  Acc. |  MCC |  AUC
ALERTA-Net(P) |  0.4988 |  0.0016 |  0.5238 |  0.07168 |  0.6019
ALERTA-Net(S) |  0.4952 |  0.0092 |  0.5150 |  0.03213 |  0.5753
ALERTA-Net(W/O M) |  0.5114 |  0.0378 |  0.5247 |  0.03992 |  0.5989
ALERTA-Net |  0.5238 |  0.05179 |  0.6136 |  0.07465 |  0.6197

###  V-C Results

The performances of our proposed models and the established baselines are
detailed in TABLE I. AT-LSTM is observed to be the superior baseline model in
terms of accuracy and MCC for movement prediction, while GRU shows an
outstanding performance in the AUC (Area Under the Curve) and MCC for
volatility prediction. ALERTA-Net surpasses both of these models by
significant margins. In terms of accuracy, ALERTA-Net achieves a score of
0.5238 and 0.6136, outperforming GRU and AT-LSTM by 1.4% and 4.2%
respectively. Additionally, for MCC, ALERTA-Net outperforms GRU and AT-LSTM by
22.8% and 6.2% respectively, and outshines both in AUC by a margin of 3.4%.
Overall, these results reinforce the effectiveness of our proposed model
ALERTA-Net.

###  V-D Ablation Study

To conduct an in-depth analysis of the core components of ALERTA-Net, we have
constructed three variations alongside the fully-loaded model. Each variant is
specifically tailored to handle certain types of input data: ALERTA-Net(P)
solely relies on closed price data, ALERTA-Net(S) exclusively processes
Twitter-derived sentiment data, while ALERTA-Net(W/O M) incorporates both
price and sentiment data, but omits macroeconomic information. As shown in
TABLE II, our ablation study revealed that incorporating macroeconomic data
significantly enhances the predictive capabilities of the model for stock
movement and volatility to varying degrees.

##  VI Conclusion

We introduced ALERTA-Net, a deep generative neural network architecture, to
showcase the efficacy of combining search engine data, macro-economy data, and
social media data when trying to predict stock movements and volatility. We
tested our model on a new comprehensive dataset and showed it performs better
than strong baselines. In future studies, we plan to enhance accuracy by
integrating a variety of text and audio sources, including earnings calls and
financial reports.

## References

  * [1] M. Billah, S. Waheed, and A. Hanifa, “Stock market prediction using an improved training algorithm of neural network,” in _2016 2nd International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE)_. IEEE, 2016, pp. 1–4.
  * [2] T. H. Nguyen, K. Shirai, and J. Velcin, “Sentiment analysis on social media for stock movement prediction,” _Expert Systems with Applications_ , vol. 42, no. 24, pp. 9603–9611, 2015.
  * [3] F. Feng, X. He, X. Wang, C. Luo, Y. Liu, and T.-S. Chua, “Temporal relational ranking for stock prediction,” _ACM Transactions on Information Systems (TOIS)_ , vol. 37, no. 2, pp. 1–30, 2019.
  * [4] Y. Xu and S. B. Cohen, “Stock movement prediction from tweets and historical prices,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 1970–1979. [Online]. Available: https://aclanthology.org/P18-1183
  * [5] W. Lu, J. Li, J. Wang, and L. Qin, “A cnn-bilstm-am method for stock price prediction,” _Neural Computing and Applications_ , vol. 33, pp. 4741–4753, 2021.
  * [6] J. A. Cookson and M. Niessner, “Why don’t we agree? evidence from a social network of investors,” _The Journal of Finance_ , vol. 75, no. 1, pp. 173–228, 2020.
  * [7] Z. Wang, Y. Huang, B. Cai, R. Ma, and Z. Wang, “Stock turnover prediction using search engine data,” _Journal of Circuits, Systems and Computers_ , vol. 30, no. 07, p. 2150122, 2021.
  * [8] Y. Mao, W. Wei, B. Wang, and B. Liu, “Correlating s&p 500 stocks with twitter data,” in _Proceedings of the First ACM International Workshop on Hot Topics on Interdisciplinary Social Networks Research_ , ser. HotSocial ’12. New York, NY, USA: Association for Computing Machinery, 2012, p. 69–72. [Online]. Available: https://doi.org/10.1145/2392622.2392634
  * [9] Q. Zhang, C. Qin, Y. Zhang, F. Bao, C. Zhang, and P. Liu, “Transformer-based attention network for stock movement prediction,” _Expert Systems with Applications_ , vol. 202, p. 117239, 2022.
  * [10] S. R. Das and M. Y. Chen, “Yahoo! for amazon: Sentiment parsing from small talk on the web,” _For Amazon: Sentiment Parsing from Small Talk on the Web (August 5, 2001). EFA_ , 2001.
  * [11] M. De Choudhury, H. Sundaram, A. John, and D. D. Seligmann, “Can blog communication dynamics be correlated with stock market activity?” in _Proceedings of the nineteenth ACM conference on Hypertext and hypermedia_ , 2008, pp. 55–60.
  * [12] S. Long, B. Lucey, Y. Xie, and L. Yarovaya, ““i just like the stock”: The role of reddit sentiment in the gamestop share rally,” _Financial Review_ , vol. 58, no. 1, pp. 19–37, 2023.
  * [13] R. Xiong, E. P. Nichols, and Y. Shen, “Deep learning stock volatility with google domestic trends,” _arXiv preprint arXiv:1512.04916_ , 2015.
  * [14] I. Bordino, S. Battiston, G. Caldarelli, M. Cristelli, A. Ukkonen, and I. Weber, “Web search queries can predict stock market volumes,” _PloS one_ , vol. 7, no. 7, p. e40014, 2012.
  * [15] Z. Hu, W. Liu, J. Bian, X. Liu, and T.-Y. Liu, “Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction,” in _Proceedings of the eleventh ACM international conference on web search and data mining_ , 2018, pp. 261–269.
  * [16] W. E. Ferson and C. R. Harvey, “The variation of economic risk premiums,” _Journal of political economy_ , vol. 99, no. 2, pp. 385–415, 1991.
  * [17] S. Jank, “Mutual fund flows, expected returns, and the real economy,” _Journal of Banking & Finance_, vol. 36, no. 11, pp. 3060–3070, 2012.
  * [18] J. D. Hamilton, “Causes and consequences of the oil shock of 2007-08,” National Bureau of Economic Research, Tech. Rep., 2009.
  * [19] P. J. Kehoe, P. Lopez, V. Midrigan, and E. Pastorino, “Asset prices and unemployment fluctuations,” National Bureau of Economic Research, Tech. Rep., 2019.
  * [20] B. Xie, R. Passonneau, L. Wu, and G. G. Creamer, “Semantic frames to predict stock price movement,” in _Proceedings of the 51st annual meeting of the association for computational linguistics_ , 2013, pp. 873–883.
  * [21] J. Camacho-collados, K. Rezaee, T. Riahi, A. Ushio, D. Loureiro, D. Antypas, J. Boisson, L. Espinosa Anke, F. Liu, E. Martínez Cámara _et al._ , “TweetNLP: Cutting-edge natural language processing for social media,” in _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Abu Dhabi, UAE: Association for Computational Linguistics, Dec. 2022, pp. 38–49. [Online]. Available: https://aclanthology.org/2022.emnlp-demos.5
  * [22] Y. Xu and S. B. Cohen, “Stock movement prediction from tweets and historical prices,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 1970–1979.
  * [23] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [24] S. R. Baker, N. Bloom, S. J. Davis, and M. C. Sammon, “What triggers stock market jumps?” National Bureau of Economic Research, Tech. Rep., 2021.
  * [25] Y. Ding, H. Zhang, and J. Zhang, “Private vs state ownership and earnings management: Evidence from chinese listed companies,” _Corporate Governance: An International Review_ , vol. 15, no. 2, pp. 223–238, 2007.
  * [26] L. Cazzoli, R. Sharma, M. Treccani, and F. Lillo, “A large scale study to understand the relation between twitter and financial market,” in _2016 third European network intelligence conference (ENIC)_. IEEE, 2016, pp. 98–105.
  * [27] X. Li, Y. Li, H. Yang, L. Yang, and X.-Y. Liu, “Dp-lstm: Differential privacy-inspired lstm for stock prediction using financial news,” _arXiv preprint arXiv:1912.10806_ , 2019.
  * [28] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 2016, pp. 785–794.
  * [29] Y. Wang, M. Huang, X. Zhu, and L. Zhao, “Attention-based lstm for aspect-level sentiment classification,” in _Proceedings of the 2016 conference on empirical methods in natural language processing_ , 2016, pp. 606–615.