# The effect of COVID-19 vaccinations on self-reported depression and anxiety
during February 2021

Max Rubinstein
Heinz College and Department of Statistics & Data Science
Carnegie Mellon University  The authors gratefully acknowledge invaluable
advice from discussions with Vinni Bhatia, Nate Breg, David Choi, Riccardo
Fogliato, Joel Greenhouse, Edward Kennedy, Samantha Patel, Alex Reinhart, and
Robin Mejia. Amelia Haviland
Heinz College
Carnegie Mellon University  Joshua Breslau
RAND Corporation

###### Abstract

Using the COVID-19 Trends and Impact survey, we find that COVID-19
vaccinations reduced the prevalence of self-reported feelings of depression
and anxiety, isolation, and worries about health among vaccine-accepting
respondents in February 2021 by 3.7, 3.3, and 4.3 percentage points,
respectively, with particularly large reductions among respondents aged 18 and
24 years old. We show that interventions targeting social isolation account
for 39.1% of the total effect of COVID-19 vaccinations on depression, while
interventions targeting worries about health can account for 8.3%. This
suggests that social isolation is a stronger mediator of the effect of
COVID-19 vaccinations on depression than worries about health. We caution that
these causal interpretations rely on strong assumptions.

Keywords: Mental health, COVID-19, vaccinations

##  1 Introduction

The COVID-19 pandemic and the associated social isolation, economic hardship
associated with job loss, and social uncertainty led to an increase
psychological distress ([24]). For example, [9] showed that the same number of
people experienced serious psychological distress in May 2021 as the entire
year prior to the pandemic. [27] estimated that the pandemic caused a 27.6%
increase in cases of major depressive disorder and a 25.6% increase in cases
of anxiety disorders globally. Younger adults have consistently been found to
be among the most adversely subgroups ([27], [28]). Secondary consequences of
these mental health impacts, such as suicide and drug overdoses, are also of
concern (see, e.g., [16], [13]).

If COVID-19 pandemic worsened mental health, then perhaps receiving a COVID-19
vaccination improved it. This might happen, for example, by decreasing worries
about health or social isolation. We investigate this hypothesis by estimating
the effects of COVID-19 vaccinations on the rates of depression and anxiety,
feelings of social isolation, and worries about health in February 2021. Our
analysis is based on the COVID-19 Trends and Impacts Survey (CTIS) ([26]), a
cross-sectional survey designed by the Delphi group at Carnegie Mellon
University and administered in collaboration with Facebook. We quantify the
average effect of vaccinations on these outcomes among a subset of vaccine-
accepting Facebook users during February 2021. By vaccine-accepting we mean
CTIS respondents who indicated that they either would receive a COVID-19
vaccine if offered one that day, or who indicated that they had already
received a COVID-19 vaccine. By depression and anxiety we mean self-reported
feelings, not a clinical diagnosis. A key assumption for the causal
interpretation of this analysis is that conditional on vaccine-acceptance and
observed covariates, vaccinated and unvaccinated respondents differed on
average only in vaccination status.

We also examine effect heterogeneity and mediation. For our heterogeneity
analysis, we consider both pre-specified and data-driven subgroups, using an
approach similar to [3]. For our mediation analysis, we consider a model where
COVID-19 vaccinations affect depression and anxiety through feelings of social
isolation, worries about health, and a direct effect capturing all other
channels. We allow the causal ordering of the mediators to be unknown;
however, we assume that they are determined prior to the outcome, and that no
other effects of vaccinations affect these mediators. Using the interventional
effect decomposition proposed by [30], we estimate the proportion of the total
effect on depression via each mediator. While interventional indirect effects
do not necessarily reflect mechanisms without stronger assumptions ([21]),
this is the first study we are aware of that attempts to estimate the effects
of COVID-19 vaccinations on depression via intermediate pathways.

Section 2 presents a brief review of the related literature and our
contributions. Section 3 describes our data and provides summary statistics.
Section 4 outlines our identification, estimation, and inferential strategy.
Section 5 presents our results. Section 6 considers the sensitivity of our
results to our causal assumptions and their robustness to analytic choices,
and Section 7 provides discussion and conclusion. Additional materials are
available in the Appendices.

##  2 Related Literature

Two concurrent studies examined the effect of COVID-19 vaccinations on mental
health: [23] and [1]. [23] use longitudinal data from the Understanding
America Survey and fixed-effects models to estimate that vaccines caused a 4%
reduction of the probability of being at least mildly depressed and a 15%
reduction in the probability of being severely depressed. When testing for
effect heterogeneity, they were unable to rule out that the effects were the
same across sex, race, and education level. [1] use repeated cross-sectional
survey data from Census Pulse. Using heterogeneity in state-level eligibility
requirements as an instrumental variable, they use two-stage least squares to
estimate that COVID-19 vaccination reduces anxiety and depression symptoms by
nearly 30%. They find larger effect sizes among individuals with lower
education levels, who rent their housing, who are unable to telework, and who
live with children.

Our identification strategy instead hinges on an assumption that sample
selection and COVID-19 vaccination are independent of potential outcomes given
vaccine-acceptance and observed covariates. This assumption precludes
confounding at the population-level and biases induced by conditioning on our
sample. We also exploit auxiliary data to identify potential high impact
subgroups hence mitigating the risk of false discoveries. Finally, we estimate
interventional indirect effects to better understand the mechanisms through
which COVID-19 vaccinations affect depression and anxiety. By contrast, the
previous studies only identify average effects or local average effects, do
not use sample-splitting for their heterogeneity analyses, and do not address
mediation beyond speculation. A key contribution of our paper is therefore our
mediation analysis. This analysis hinges on the idea that worries about health
and social isolation are the two primary pathways through which vaccinations
would change depression and anxiety. These are commonly cited as two major
ways that the pandemic has adversely impacted mental health ([25], [14],
[11]).

##  3 Data

Our primary dataset is the COVID-19 Trends and Impact Survey (CTIS), created
by the Delphi group at Carnegie Mellon University in collaboration with
Facebook. From April 2020 through June 2022, Facebook offered the CTIS survey
using a stratified random sample (strata defined by geographic region) of
approximately 100 million US residents from the Facebook Active User Base who
use one of the supported languages: English, Spanish, French, Brazilian
Portuguese, Vietnamese, and simplified Chinese. A link to participate in the
survey was displayed at the top of users’ News Feed. Users received this offer
at most once per month depending on their geographic location. The goal was to
obtain approximately one million monthly responses. The CTIS evolved over
time, but all waves of the survey instrument include questions about COVID
symptoms, health behaviors, mental health, and demographics. A complete list
of the questions over time is available online. [26] provides more details
about the survey.

We analyze the period from February 1-28, 2021111This period encompasses both
CTIS waves 7 and 8. Wave 8 was deployed on February 8, 2021, and incorporated
changes to some of the questions we use as covariates. These include the
addition of categories to the chronic health conditions and occupation
questions, and changing a question on the previous receipt of flu vaccine to
be defined from July 2020 instead of June 2020. We choose this timeframe to
increase the plausibility of our identifying assumptions, which we discuss
further in Section 4.4. Our primary outcome ($Y$) is whether a respondent
reported feeling depressed or anxious “most” or “all” of the time in the
previous five days. We hereafter refer to this variable as “depressed.” Our
treatment variable ($A$), is an indicator of whether the respondent ever
received a COVID-19 vaccine.222We recode the response “I don’t know” to be
grouped with people who indicated they did not already receive a COVID-19
vaccine. Importantly, the CTIS does not tell us when a respondent was
vaccinated. We also consider responses to how often a respondent felt isolated
from others in the previous five days ($M_{1}$), and how worried respondents
reported feeling about themselves or an immediate family member becoming
seriously ill from COVID-19 ($M_{2}$), each containing four levels.333The CTIS
questions specifically are “In the past 5 days, how often have you felt
isolated from others?” and “How worried are you that you or someone in your
immediate family might become seriously ill from COVID-19 (coronavirus
disease)?” We hereafter refer to these variables as “isolated” and “worries
about health,” respectively. We also create the variable $M$ as the sixteen
level joint variable $M_{1}\times M_{2}$. Finally, we dichotomize $M_{1}$ and
$M_{2}$ when analyzing them as outcomes. Specifically, we create indicators
for the responses “most” or “all” of the time for isolation, and “very
worried” for worries about health. Whenever referring to our outcomes
analyses, for simplicity we refer to any outcomes as $Y$.

We consider several covariates recorded in the CTIS pertaining to demographic,
household, employment, and health status. For demographics we consider each
respondent’s age category, race, ethnicity, gender, and highest education
attained. For household variables, we construct indicators of whether a
respondent lives alone; lives with a child attending school in-person full-
time, part-time, or not in school; or lives with an elderly (over 65)
individual. For employment, we record whether each respondent worked for pay
in the previous 30 days, and whether their work was at home or outside. Among
respondents who work, we record the respondent’s occupation type, including
healthcare, education, service-industry workers, protective services, other
employment sector. Finally, we record information regarding each respondent’s
health and health behaviors, including whether they have previously been
tested for and/or diagnosed with COVID-19, how many “high-risk” health
conditions they have ever been diagnosed with, (0, 1, 2, or three or more),
and whether they received a flu shot since April 2020 (included as a proxy for
health behaviors). We also record whether the respondent took the survey in
English or Spanish. For all of the CTIS survey-measured covariates, we recode
missingness as a separate category to account for item non-response. Lastly,
we augment the dataset with indicators of each respondent’s state and week of
response. The state indicators can control for fixed state-level policies
during the study period that may be associated with both vaccine access and
depression, including eligibility requirements, while the week indicators
control for time trends associated with both treatment and outcomes.

We also join several county-level variables using each respondents’ FIPS code.
Using data from the 2019 American Community Survey, we add the county-level
population, population density, percent living in poverty, percent uninsured,
and Gini index. Using the National Center for Health Statistics Urban-Rural
classification scheme, we classify the type of county each respondent lives in
(Large Central Metro, Large Fringe Metro, Medium Metro, Small Metro,
Micropolitan, and Non-core regions). These variables help control for
differential COVID-19 vaccine access. We also include the county the total
number of deaths per capita from COVID-19 in each county the final two weeks
in January, and divide this variable into sextiles (data obtained from Johns
Hopkins University). Finally, we include Biden’s county vote share at the
county-level (Biden minus Trump vote totals divided by total votes). We denote
all demographic and county-level covariates as $X$.

Our initial dataset from February includes 1,232,398 responses. We exclude
2,530 respondents whose Facebook user language was not English or Spanish,
3,690 responses from Alaska – Alaska did not report 2020 voting results at the
county-level – and 39,526 responses that did not provide FIPS code information
– preventing us from merging county-level covariates. We then subset our data
to only include respondents who say that they “probably” or “definitely” would
get the COVID-19 vaccine if offered to them today or who reported already
being vaccinated ($V=1$). This left 881,315 responses, excluding 229,894
vaccine-hesitant respondents and 75,443 respondents who did not indicate
whether or not they intended to receive a COVID-19 vaccine. Finally, 758,594
respondents on our primary sample, or 86.1 percent, provided answers to all
questions pertaining to vaccination and all three outcomes ($R=1$). This
defines our analytic sample. Additional details are available in Appendix 8.

###  3.1 Summary statistics

Figure 1 displays the weekly prevalence of depression among all CTIS
respondents. The plot displays a spike in May 2020 followed the police murder
of George Floyd and the associated protests, another spike in early November
2020 during the presidential election, and generally decreasing depression
afterwards. One exception occurred during the week of January 6th, when Donald
Trump instigated an insurrection on the U.S. Capitol following his loss in the
2020 election.

Figure 1: Trends in depression and anxiety, April 2020 - February 2021

The prevalence of depression varies substantially by age group. The left-hand
panel of Figure 2 shows the weekly prevalence from October 2020 through
February 2021, with rates increasing monotonically across seven age groups.
This may depict a real negative correlation between depression and age;
however, an alternative interpretation could be that younger individuals have
a different vocabulary around mental health than older individuals and respond
to these questions differently. While possible, Figure 2 displays similar
patterns for isolation and worries about health. Perhaps paradoxically, the
percentage of respondents who report being very worried about their health is
lowest among the older respondents and generally highest among 18-34 year old
respondents.444Similar patterns were observed in other surveys: for example,
an Economist/YouGov poll showed similar age patterns with respect to worries
about contracting COVID-19. [10] also found higher levels of social isolation
among younger individuals.

Figure 2: Mental health trends, October 2020 - February 2021

Each outcome declined in prevalence throughout February 2021, with
particularly large declines in worries about health. This may reflect seasonal
trends and the fact that the number of new COVID-19 infections was falling
substantially.555See, e.g., https://www.nytimes.com/interactive/2021/us/covid-
cases.html However, this timeframe also coincided with substantial numbers of
people receiving their COVID-19 vaccinations. Table 1 below illustrates number
of individuals and the percentage vaccinated within each age category among
all CTIS respondents in February 2021.666As has been observed elsewhere, the
CTIS over-represents vaccinated individuals compared to the U.S. population
([26]). This does not necessarily threaten the internal validity of our
analysis (see Section 4.3), though may threaten our ability to generalize our
results.

Table 1: Percent CTIS respondents vaccinated by age group, February 1-28th, 2021  Age group | Percent vaccinated | N
---|---|---
18-24 years | 12.1 | 41941
25-34 years | 19.0 | 123112
35-44 years | 21.1 | 156206
45-54 years | 21.6 | 166973
55-64 years | 21.5 | 201971
65-74 years | 47.4 | 201611
75+ years | 62.2 | 87256
Note: numbers excluded non-respondents to COVID-19 vaccination

Our study examines whether COVID-19 vaccinations in part account for these
improving outcomes during February 2021. We begin by examining the differences
in the prevalence of each outcome in vaccinated versus unvaccinated
respondents that month, taking these averages within separate subgroups.
Figure 3 displays these results.

Figure 3: Associations between mental health outcomes and vaccination,
February 2021

Given the high base-rates of depression and isolation among the youngest
respondents, perhaps not surprisingly we observe the largest declines in the
prevalence of each outcome among younger respondents relative to older ones.
We also see larger declines in rates of depression and isolation among White
relative to Black respondents, and non-Hispanic relative to Hispanic
respondents. When examining worries about health by each demographic category,
the declines appear lower among the the oldest age groups, higher among people
with three or more high-risk conditions, and higher among Hispanic versus non-
Hispanic respondents. Broadly speaking, patterns in depression and isolation
appear more similar to each other than to patterns in worries about health.

Of course, COVID-19 vaccinations were not randomly assigned, and we cannot
make causal claims from these associations. For example, individuals who were
most at-risk and who had the means were most likely to receive the
vaccination. If these same individuals were also more likely to be depressed
or anxious without the vaccine, then these associations would overstate the
absolute magnitude of any treatment effects. We therefore adjust these
estimates to control for the demographic, household, employment, health, and
county-characteristics described previously, as well as state and week
indicators, and outline the assumptions necessary for these adjusted
associations to reflect causal quantities. Additional summary statistics are
available in Appendix 8.

##  4 Methods

###  4.1 Preliminaries

Recall that $A$ indicates prior receipt of COVID-19 vaccination, $M_{1}$
isolation, $M_{2}$ worries about health, and $Y$ depression. $X$ indicates the
covariate vector described in Section 3, and $V$ indicates vaccine-acceptance.
We define $S$ as an indicator of responding to the survey, and
$(R_{Y},R_{A},R_{M_{1}},R_{M_{2}})$ as indicators of observing
$(Y,A,M_{1},M_{2})$, respectively. We observe $n$ independent samples from a
pool of active Facebook users residing in the United States of the random
variable
$O=(YR_{Y},AR_{A},M_{1}R_{M_{1}},M_{2}R_{M_{2}},R_{Y},R_{A},R_{M_{1}},R_{M_{2}},X,V,S=1)$.
We define two important functions of this distribution. First, we let
$R=R_{Y}R_{M_{1}}R_{M_{2}}R_{A}$. This is an indicator of whether we observe
all outcomes, mediators, and treatment assignment information (hereafter
“complete-cases”) for a respondent. Second, define $Z=RSV$. In other words,
$Z$ is an indicator of inclusion in our analytic sample.

###  4.2 Estimands

We next define causal estimands with respect to our outcomes and mediation
analyses using potential outcomes notation. Before defining these quantities,
we make the following assumptions. First, we invoke the SUTVA. This implies
that there is only one version of treatment and rules out interference between
individuals: specifically, each individual’s potential outcomes only depend on
their own vaccination status. Second, we assume no carryover effects. This
implies that an individual’s potential outcome having been vaccinated one week
prior to responding to the CTIS is identical to their potential outcome were
they vaccinated one month prior. Finally, we assume no anticipatory effects,
implying that future vaccination status does not affect one’s potential
outcomes at the time an individual responded to the CTIS.

Under these assumptions, we define the potential outcomes for each individual
as $Y_{i}^{a_{i}}$: that is, an individual’s outcome when setting their
vaccination status $A_{i}$ to level $a_{i}$. This quantity does not depend on
anyone else’s vaccination status (SUTVA), their past vaccination status (no
carryover effects), nor their future vaccination status (no anticipatory
effects). Finally, we define the function
$\mu_{a}(X_{i})=\mathbb{E}[Y_{i}^{a}\mid X_{i},Z_{i}=1]$, representing the
expected potential outcome under treatment $A_{i}=a$ for an individual given
their covariates, vaccine-acceptance, and being in our analytic sample.

####  4.2.1 Outcomes

We use $Y$ to denote any outcome considered in this section to ease notation,
recalling that $M_{1}$ and $M_{2}$ are dichotomized for this analysis. Assume
that there are $N$ adults living in the United States. A natural estimand is
the average treatment effect:

$\displaystyle\psi^{ate}=N^{-1}\sum_{i=1}^{N}[Y_{i}^{1}-Y_{i}^{0}]$ (1)

This reflects the mean difference in the prevalence of each outcome if
everyone were vaccinated contrasted against the case where no one was
vaccinated, averaging over the empirical covariate distribution of adults
residing in the United States.

This causal quantity is difficult to target for at least four reasons. First,
the U.S. adult population consists of two types of individuals: the vaccine-
hesitant and the vaccine-accepting. The vaccine-hesitant constitute a group of
“never-takers,” a group would never choose be vaccinated (at least during this
timeframe). We cannot estimate the treatment effect among this subgroup
without strong modeling assumptions. Second, the sample is taken over the
population of Facebook users who reside in the United States. This population
differs from the U.S. adult population, the arguably more interesting
population. Third, the average over the population of Facebook users is also
difficult to credibly estimate, as survey response rates in February 2021 were
approximately one percent. Unless we assume that survey non-response was
completely random, we cannot estimate this quantity without further assuming,
for example, that the selection process was based on pre-treatment
characteristics and then modeling this process. Finally, among the survey
respondents, we only observe the vaccination status and outcomes among those
who respond to these items. Similar to survey non-response, generalizing the
causal effects to include these individuals requires additional assumptions
and modeling.

We make several analytic choices to address these challenges. First, we
condition our analysis among the vaccine-accepting population. Second, our
primary estimands are conditional on our analytic sample. Specifically, we
define our target estimand as:777This expression slightly simplifies of our
true targeted estimand. As we explain in Section 3, we also remove those with
missing FIPS code information and those who live in Alaska.

$\displaystyle\psi$
$\displaystyle=\frac{1}{\sum_{i=1}^{N}Z_{i}}\sum_{i:Z_{i}=1}\mathbb{E}[Y_{i}^{1}-Y_{i}^{0}\mid
X_{i},Z_{i}=1]$ (2)
$\displaystyle=\mathbb{P}_{n}[\mu_{1}(X_{i})-\mu_{0}(X_{i})]$ (3)

This estimand is the average expected difference in the prevalence of the
outcomes given the empirical covariate distribution of the vaccine-accepting
complete-cases in our analytic sample. We abbreviate this empirical average
using $\mathbb{P}_{n}$ moving forward. We also examine all effects on both the
risk-ratio scale ($\mathbb{P}_{n}[\mu_{1}(X)]/\mathbb{P}_{n}[\mu_{0}(X)]$) and
averaged within distinct subgroups defined by the covariates. In Appendix 12.4
we consider estimands that do not deterministically set vaccination status,
but rather shift each individual’s probability of vaccination ([17]).

####  4.2.2 Interventional effects

We next decompose the effect of vaccinations on depression into four separate
channels: a direct effect, an indirect effect through social isolation, an
indirect effect through worries about health, and an indirect effect
reflecting the dependence of social isolation and worries about health on each
other, following [30]:

$\displaystyle\psi_{IDE,1}$
$\displaystyle=\mathbb{P}_{n}\left(\sum_{m_{1}}\sum_{m_{2}}(\mathbb{E}[Y^{1m_{1}m_{2}}\mid
X,Z=1]-\mathbb{E}[Y^{0m_{1}m_{2}}\mid X,Z=1])\right.$
$\displaystyle\left.\times P(M_{1}^{1}=m_{1},M_{2}^{1}=m_{2}\mid
X,Z=1)\right)$ $\displaystyle\psi_{M_{1},0}$
$\displaystyle=\mathbb{P}_{n}\left(\sum_{m_{1}}\sum_{m_{2}}\mathbb{E}[Y^{0m_{1}m_{2}}\mid
X,Z=1](P(M_{1}^{1}=m_{1}\mid X,Z=1)\right.$
$\displaystyle\left.-P(M_{1}^{0}=m_{1}\mid X,Z=1)]P(M_{2}^{0}=m_{2}\mid
X,Z=1)\right)$ $\displaystyle\psi_{M_{2},0}$
$\displaystyle=\mathbb{P}_{n}\left(\sum_{m_{1}}\sum_{m_{2}}\mathbb{E}[Y^{0m_{1}m_{2}}\mid
X,Z=1](P(M_{2}^{1}=m_{1}\mid X,Z=1)\right.$
$\displaystyle\left.-P(M_{2}^{0}=m_{2}\mid X,Z=1))P(M_{1}^{1}=m_{2}\mid
X,Z=1)\right)$ $\displaystyle\psi_{Cov,0}$
$\displaystyle=\psi-\psi_{IDE,1}-\psi_{M_{1},0}-\psi_{M_{2},0}$

As explained by [30], the interventional direct effect $(\psi_{IDE,1})$
represents the contrast in mean potential outcomes when everyone is vaccinated
versus no one is vaccinated, while drawing the mediators for each individual
randomly from the joint distribution of the mediators conditional on $A=0$
given their covariates $X$. By contrast the interventional indirect effect of
$M_{1}$ $(\psi_{M_{1},0})$ represents the average contrast between subject-
specific draws from the counterfactual distribution of feelings of social
isolation at $A=0$ versus $A=1$ given their covariates, while drawing worries
about health from the counterfactual distribution under $A=0$, given their
covariates and fixing vaccination status at level $A=1$. The interventional
indirect effect through $M_{2}$ $\left(\psi_{M_{2},0}\right)$ is defined
analogously. The covariant effect $(\psi_{Cov,0})$ is the difference between
the total effect and all three of these effects. This effect captures the
effect of the dependence of the mediators on each other, and would be equal to
zero, for example, if these mediators were conditionally independent. Finally,
a similar decomposition holds switching 0 for 1 in each estimand and reversing
the signs. We consider both decompositions in our analysis.

[30] describes $\psi_{M_{1},0}$ as capturing the pathways $A\to M_{1}\to Y$
and $A\to M_{2}\to M_{1}\to Y$, with $\psi_{M_{2},0}$ defined analogously.
This implies that $\psi_{M_{1},0}$ captures the effect of vaccinations on
depression via directly changing social isolation which then affects
depression; and via directly changing worries about health, which then affects
social isolation which affects depression. By contrast, $\psi_{M_{2},0}$ would
capture the effect of vaccinations on depression via direct changes in worries
about health which affects depression, and via changes in social isolation
which affect worries about health which affect depression. We refer to [30]
for more details on this decomposition and the interpretation of these
estimands.888We caution that recent results from [21] casts doubt on
mechanistic interpretations of interventional effects.

####  4.2.3 Other estimands

In Appendix 9 we describe three other estimands: the effect of the observed
distribution of vaccination, the effect of incremental interventions on the
propensity score, and the effect of the pandemic on mental health. The first
two estimands may correspond to more realistic interventions, requiring weaker
identifying assumptions than those noted below. The third estimand may be of
interest if we wish to use our effect estimates to learn about the impact of
the COVID-19 pandemic on mental health. We show in Appendix 10 that under some
assumptions, we may interpret our average effect estimates on either the risk-
differences or risk-ratio scale as a lower bound on this quantity using our
existing estimates of the effect of COVID-19 vaccinations on mental health.

###  4.3 Identification

We assume the following causal structure holds among vaccine-accepting
individuals.

XA(R, S)$M_{1}$$M_{2}$Y Figure 4: Assumed causal structure

Figure 4 clarifies assumptions about causal ordering of the variables. Two
particular features are worth noting: first, we assume no reverse causation
between depression and the mediators. Second, we allow the mediators to depend
on each other and require no assumptions about their causal ordering. Figure 4
also encodes the assumed relationships between the variables necessary to
identify our causal estimands in terms of our observed data. We instead
motivate these conditions using potential outcomes, and assume that for all
values of $(a,m_{1},m_{2})$:999The conditions in the DAG are necessary, but
not sufficient for our identification result (see Appendix 10). For example,
the DAG implies that $(R,S)\perp(Y,M_{1},M_{2})\mid(X,A,V=1)$, while we
instead invoke Assumption 3. Identification under alternative conditions is
possible.

######  Assumption 1 (No unmeasured confounding).

$\displaystyle(Y^{am_{1}m_{2}},M_{1}^{a},M_{2}^{a})\perp A\mid(X,V=1)$ (4)

######  Assumption 2 (Y-M ignorability).

$\displaystyle Y^{am_{1}m_{2}}\perp(M_{1},M_{2})\mid(A=a,X,V=1)$ (5)

Specifically, no unmeasured confounding states that among the vaccine-
accepting population, vaccinations are independent of the potential mediators
and outcomes conditional on covariates. Y-M ignorability states that the
mediators are jointly independent of potential outcomes conditional on
covariates and vaccination status. Finally, we we assume random sample
selection: this states that survey response and complete-case indicators are
random with respect to the potential outcomes given the covariates and
vaccination status.

######  Assumption 3 (Random sample selection).

$\displaystyle(S,R)\perp(Y^{am_{1}m_{2}},M_{1}^{a},M_{2}^{a})\mid(X,A,V=1)$
(6)

We additionally assume that the survey responses fully capture all population
relationships (“no measurement error”), that all vaccine-accepting individuals
in our sample have a probability of vaccination bounded away from zero and
one, and that the joint mediator probabilities are all bounded away from zero
(“positivity”). We formalize all identifying assumptions in Appendix 10 and
show that they imply that our causal parameters are identified in terms of the
observed data. For example,
$\mathbb{P}_{n}[\mu_{a}(X_{i})]=\mathbb{P}_{n}[\mathbb{E}[Y_{i}\mid
X_{i},A_{i}=a,Z_{i}=1]]$. We largely defer discussing possible violations of
these assumptions to Section 6.

###  4.4 Analytic timeframe

We choose to limit our causal analysis to February 2021 to strengthen the
credibility of our identifying assumptions. The challenge is that we only
observe cross-sectional data, and the causal assumptions required for our
analysis simplify a more complicated longitudinal process. Our identifying
assumptions arguably become less tenable over time. As one example, consider a
key unobserved variable: time since vaccination. Assuming no carryover effects
implies that this variable is not required either to define or identify our
causal estimands. Absent this assumption, time since vaccination would act as
an unmeasured confounder inducing a bias for our outcomes analysis that
plausibly increases with time from the first administered COVID-19 vaccines
(we discuss this point further in Appendix 10). Our mediation analysis
similarly becomes less tenable over time: for example, vaccinations may
improve economic conditions over time, which in turn may act as a post-
treatment confounder with respect to feelings of social isolation or worries
about health. We therefore limit our analysis to February, relatively soon
after the start of COVID-19 vaccination administrations.

###  4.5 Estimation and inference

We use influence-function based estimators throughout ([4], [17]), requiring
estimaion of several nuisance functions. For example, for the interventional
effects, we estimate the outcome model, the propensity score model, and the
joint mediator probabilities. For our primary results we use logistic
regression and multinomial logistic regression. We allow for separate models
for each treatment level for our outcomes analysis, and each treatment-
mediator level for our mediation analysis. Similarly, to estimate mediator
probabilities we allow separate models within each treatment level. Our
propensity score model includes only main effects for each covariate. These
estimators are multiply robust in the sense that the consistency of the
estimates do not require that all models are correctly specified, but rather
that some subset are ([4]). We generate 95% confidence intervals and conduct
hypothesis tests using the empirical variance of our estimated influence
function values and standard normal quantiles. These intervals and tests are
valid if all of our models are correctly specified. To alleviate concerns
about model misspecification, we also estimate all nuisance functions using
XGBoost and sample-splitting. Assuming the consistency of the influence-
function estimates and that that the nuisance functions are estimated at
$n^{-1/4}$ rates in the $L_{2}$ norm, these estimates are asymptotically
normal and root-n consistent with a simple expression for the variance ([4]).
Appendix 11 provides further details.

We also use a data-driven approach to identify subgroups with large or small
treatment effects. Specifically, we estimate the influence function values
using auxiliary data from March 1-14, 2021 using XGBoost. We regress these
pseudo-outcomes on a subset of the demographic covariates using depth-four
regression trees. After generating the trees, we use a “human-in-the-loop”
approach to determine whether the candidate subgroups are interpretable and
substantively interesting. We then conduct estimation and inference using on
these candidate hypotheses using the February data.101010This is similar to
the “honest” tree-based estimation approach proposed by [3], and is an example
of a DR-learner, discussed in [18] All t-tests referenced below set
$\alpha=0.05$ and only control the Type I error rate for each test marginally.

##  5 Results

###  5.1 Outcome analysis

Table 2 displays our primary results. We estimate comparable effect sizes for
each variable decreases of 3.7 (4.0, 3.5), 3.3 (3.5, 3.0), and 4.3 (4.6, 4.0)
in the prevalence of depression, isolation, and worries about health,
respectively. These effects translate to 19 percent, 16 percent, and 15
percent decreases in the prevalence of depression, social isolation and
worries about health, respectively.111111As noted in Appendix 10, we may
interpret these results as bounding the effect of the pandemic on depression.
For example, under some assumptions we can say that the pandemic increased
depression by at least 3.7 percentage points on average, and that the
percentage of depressed respondents was at least 24 percent higher on average
(100*($\frac{1}{1-.19}-1$)) than it would have been absent the pandemic among
respondents in our analytic sample. Table 3 displays our results when using
XGBoost to estimate the nuisance functions. These estimates are quite similar
to the GLM estimates.

Table 2: Effect estimates, GLM models (95% CI)  Outcome | Adjusted RD | Adjusted RR | Unadjusted RD | Unadjusted RR
---|---|---|---|---
Depressed | -3.72 (-3.98, -3.47) | 0.81 (0.79, 0.82) | -9.82 (-9.98, -9.66) | 0.55 (0.54, 0.55)
Isolated | -3.26 (-3.53, -2.99) | 0.84 (0.83, 0.85) | -8.39 (-8.57, -8.22) | 0.62 (0.62, 0.63)
Worried about health | -4.26 (-4.55, -3.96) | 0.85 (0.84, 0.86) | -11.35 (-11.55, -11.16) | 0.64 (0.63, 0.64)
RD indicates risk-differences; RR indicates risk-ratio
Table 3: Effect estimates, XGBoost models (95% CI)  Outcome | Adjusted RD | Adjusted RR | Unadjusted RD | Unadjusted RR
---|---|---|---|---
Depressed | -3.73 (-3.95, -3.52) | 0.81 (0.80, 0.82) | -9.82 (-9.98, -9.66) | 0.55 (0.54, 0.55)
Isolated | -3.20 (-3.42, -2.97) | 0.84 (0.83, 0.85) | -8.39 (-8.57, -8.22) | 0.62 (0.62, 0.63)
Worried about health | -4.14 (-4.39, -3.89) | 0.85 (0.85, 0.86) | -11.35 (-11.55, -11.16) | 0.64 (0.63, 0.64)
RD indicates risk-differences; RR indicates risk-ratio

###  5.2 Subgroup heterogeneity: pre-specified groups

Figure 5 displays the estimated effects across pre-specified subgroups.
Consistent with the unadjusted estimates, among those who reported their age,
vaccinations have the largest magnitude effect sizes among respondents between
18-34 years old for all outcomes. For example, we estimate that vaccinations
reduced depression by -5.9 (-6.8, -5.0) percentage points among respondents
aged 18-34, while only -2.5 (-2.8, -2.2) percentage points among respondents
aged 65 and older. We also find larger magnitude point estimates for females
relative to males and non-binary respondents for depression. Specifically, we
estimate the effect of vaccinations on depression as being -4.0 (-4.3, -3.7)
percentage points among female identifying respondents and -3.1 (-3.5, -2.7)
among male and non-binary respondents. Lastly, we see larger effect estimates
among respondents for worries about health among respondents with two or more
high-risk health conditions (-4.8 (-5.3, -4.3)) relative to those with none
(-3.7 (-4.2, -3.2)).121212Our estimates are somewhat large in absolute
magnitude among respondents who do not provide demographic information. For
example, among those who did not respond about their age, we estimate a -5.8
(-7.2, -4.3) percentage point reduction in depression and a -7.3 (-8.9, -5.6)
reduction in worries about health. Because non-response is highly correlated
across questions, we observe similar magnitude estimates for other non-
response categories, though the point estimates for isolation tend to be
comparable to other groups.

Figure 5: Outcomes analysis, February 2021

The XGBoost estimates are very similar. These results and results on the risk-
ratio scale are available in Appendix 12.2.

####  5.2.1 Data-driven subgroup heterogeneity

We identify three possibly heterogeneous subgroups using the March data with
respect to depression: respondents aged 18-24, non-Hispanic respondents aged
25 and older, and Hispanic respondents aged 25 and older. These categories
exclude non-respondents who to age, or to ethnicity among those 25 or older.
Figure 6 displays the effect sizes and confidence intervals estimated on the
February data.131313We present estimates using our XGBoost models: our GLM
estimates do not include interactions between the covariates, and are
therefore unlikely to correctly capture this heterogeneity. Consistent with
our hypotheses, we estimate effects of -12.2 (-13.7, -10.7) among respondents
aged 18-24, -3.6 (-3.8, -3.3) among non-Hispanic respondents aged 25 or older,
and -1.9 (-2.6, -1.2) among Hispanic respondents aged 25 or older. The
youngest respondents also appear to have larger magnitude treatment effects
for isolation and worries about health.141414We display significance levels
associated with the pairwise t-tests in Appendix 12.2.

Figure 6: Data-driven subgroup estimates

We examine additional subgroups within these three primary groups. Among
respondents aged 18-24, we estimate larger absolute magnitude effect sizes
among those who live with an elderly individual (-21.6 (-26.5, -16.8); N =
2,542) versus those who do not (-10.9 (-12.7, -9.2); N = 20,347). We estimate
effects for depression and isolation that are statistically indistinguishable
from zero among Hispanic respondents whose primary Facebook user language is
Spanish (N = 23,018). By contrast, all estimated effect sizes are
statistically significant and negative for those whose Facebook user language
is English (N = 57,236).151515The differences in the estimated effects are
both statistically significant at the $\alpha=0.05$ level. We test for
differences with respect to gender and race among respondents aged 18-24, and
with respect to age among Hispanic respondents aged 25 or older. We find
statistically significant differences with respect to race among the youngest
respondents for isolation and worries about health: the effect sizes appear
smaller in absolute magnitude for non-White versus White respondents. However,
our remaining hypotheses do not yield any statistically significant and
substantively meaningful differences on the risk-differences scale.

Finally, we identify candidate heterogeneous subgroups using isolation and
worries about health as the outcomes. Beyond the previously noted
heterogeneity with respect to age, we are largely unable to confirm additional
sources of heterogeneity. One exception is with respect to gender among non-
White respondents aged 25 or older: we estimate that female identifying
respondents had larger magnitude effects with respect to social isolation
(-3.2 (-4.0, -2.5)) than non-female identifying respondents (-1.5 (-2.4,
-0.7)). Additional results may be found in Appendix 12.2.

Our results reveal that being aged 18-24 is a key source of effect
heterogeneity, especially with respect to depression and social isolation.
Moreover, living with an elderly individual appears to have a strong
interaction effect among this same group. These patterns also hold on the
risk-ratio scale (see Appendix 12.2). Additional research would be valuable to
further investigate hypotheses with respect to other demographic
characteristics, including interactions between age, race, gender, and
ethnicity, though our results suggest that age would likely remain the factor
most strongly associated with effect heterogeneity.

###  5.3 Mediation analysis

Figure 7 displays results from our mediation analysis. The left-hand panels
displays results on the risk-differences scale while the right hand displays
each estimate as a proportion of the total effect. We estimate that shifting
the distribution of social isolation from what it would have been if everyone
were unvaccinated to a world where everyone were vaccinated would reduce
depression by 1.5 (1.6, 1.4) percentage points, while shifting the
distribution of worries about health similarly reduces depression by
approximately 0.3 (0.3, 0.3) percentage points. These effects would account
for 39.1 percent and 8.3 percent of the total effect of vaccinations on
depression, respectively. We estimate the covariant effect to be close to
zero. Figure 7 also displays the total indirect effect by both pathways,
simply the sum of these three effects. We reject the null hypothesis that the
mediated effects through isolation are equal to the mediated effects through
health ($p<0.001$). The effects are comparable when changing the reference
category in the estimands (see Appendix 12.3).

Figure 7: Mediation analysis: primary estimates

We also find evidence of a direct effect of vaccination on depression that
accounts for over half of the total effect. We interpret this as reflecting a
general sense of relief that vaccination provides, perhaps related to a belief
in a return to normalcy.

Figure 8 displays estimates of the proportion mediated across pre-specified
subgroups.161616We excluded Asian and Pacific Islander and Other racial
categories as the estimates are very imprecise. The patterns are consistent,
with social isolation appearing to account for a larger proportion of the
mediated effect relative to worries about health across all groups. Perhaps
not surprisingly, worries about health appear to account for a slightly
greater percentage of the total effect among people with two or more high-risk
health conditions compared to those with none (though we do not formally test
the statistical significance of these differences). The XGBoost results are
similar. Additional results are available in Appendix 12.3.

Figure 8: Mediation heterogeneity: proportion mediated by subgroup

##  6 Sensitivity, robustness, and limitations

We consider the sensitivity of our analyses to our causal assumptions, their
robustness to alternative analytic choices, and our ability to generalize our
effect estimates. We highlight the limitations of our analysis throughout, and
also provide heuristic arguments about how violations of our causal
assumptions may bias our estimates. We conclude that our outcomes analysis is
fairly robust, and that most possible violations likely bias our estimates
towards rather away from zero. We also argue that generalizing the effect
magnitudes of our estimates beyond our sample is not unreasonable. Finally, we
discuss the limitations of our mediation analysis; however, we do not conduct
formal sensitivity analyses.

###  6.1 Unobserved confounding and selection bias

####  6.1.1 Sensitivity analysis

Our outcomes analysis assumes no unmeasured confounding and random sample
selection. Together these assumptions imply that the potential outcomes are
independent of treatment assignment given covariates and vaccine-acceptance on
our sample. This assumption may be violated either assumption fails.
Unmeasured confounders may threaten this assumption at the population level,
or selection into our analytic sample may induce bias. We can relax these
assumptions and instead estimate obtain bounds on our target sample estimand
via a variation of a sensitivity analysis proposed by [20]. Let
$\bar{\mu}_{a}(a^{\prime},x)=\mathbb{E}[Y^{a}\mid X=x,A=a^{\prime},Z=1]$. We
then assume there exists a $\tau$ satisfying (7).

$\displaystyle\frac{1}{1-\tau}\geq\frac{\bar{\mu}_{a}(a^{\prime},x)}{\bar{\mu}_{a}(a,x)}\geq
1-\tau\qquad\forall(a,a^{\prime},x)$ (7)

For example, consider the left-hand side of (7) and let $a=1$ and
$a^{\prime}=0$. The bound specifies that within each covariate stratum, the
counterfactual probability of depression among unvaccinated survey respondents
when vaccinated is not greater than than $1/(1-\tau)$ times the observed
probability of depression among the vaccinated survey respondents. In Appendix
13, we show that (7) implies a bound on the treatment effect estimates using
only functions of the observed data and $\tau$. We then use influence-function
based estimators to estimate bounds across a range of values of $\tau$ until
we find values that could explain away our estimated effects.

We estimate that values of $\tau$ greater than or equal to 0.18, 0.15, and
0.14 would explain away our effect estimates for depression, isolation, and
worries about health, respectively. In other words, our effect estimates would
be statistically indistinguishable from zero if the ratios in (7) fell outside
of the (approximate) ranges of [0.83, 1.20], [0.86, 1.16], or [0.87, 1.15] for
each respective outcome.171717The results using the XGBoost models are
virtually identical. As a point of comparison, assume that no unmeasured
confounding on the complete-cases held when controlling for the full covariate
set, but that we controlled for none of them. This would yield values of
$\tau$ of approximately 0.34, 0.27, and 0.26 for each respective
outcome.181818These values reflect cases where the observed covariates are
assumed to deconfound the relationship on the observed sample. This does not
account for scenarios where bias is induced by selection on the observed or
potential outcomes. Our average effect estimates are therefore robust to
confounding variables that are approximately 53 to 55% (e.g. 0.18/0.34 for
depression) as associative with the outcomes as the entire observed covariate
set, suggesting that our results are at least somewhat robust to these
assumptions.

This analysis is quite conservative: for example, the bounds consider cases
where the biases work in opposite directions for $\mu_{1}$ and $\mu_{0}$.
However, the biases may work in the same direction. For example, if
$\bar{\mu}_{1}(1,x)-\bar{\mu}_{1}(0,x)\approx\bar{\mu}_{0}(1,x)-\bar{\mu}_{1}(0,x)$
for all values of $x$, then then our estimates of the causal risk-differences
would remain unbiased, despite biased estimates of the prevalence of each
potential outcome in our sample.

####  6.1.2 Heuristic reasoning: no unmeasured confounding

We can also reason about the sign of the bias induced by violations of no
unmeasured confounding at the population level. Two specific confounders we
may worry about are vaccine-access and motivation to be vaccinated. We attempt
to control for vaccine-access using a variety of county-level characteristics,
the respondents’ geographical location, employment status, and financial
stress. However, these covariates may not sufficiently capture differential
access across respondents. Similarly, vaccine-acceptance acts as a rough proxy
for motivation. Yet even among vaccine-accepting respondents, some likely have
higher motivation to be vaccinated than others. We reason that failing to
control for these variables likely induces biases that work in opposite
directions, resulting in an overall bias with no clear sign.

For example, if within every covariate stratum, the probability of being
vaccinated is lower for respondents with worse access, and the probability of
being depressed, isolated, or worried about health is higher for these same
respondents, our estimates will be negatively biased. On the other hand, if
motivated respondents are more likely to be vaccinated, and more likely to be
depressed, isolated, or worried about their health generally, our estimates
will be positively biased. Our data is consistent with these hypotheses: when
running our analyses without controlling for financial stress, employment
status, or occupation, our point estimates move further away from zero (see
also Section 6.4). On the other hand, when including the vaccine-hesitant in
our analytic sample, our estimates move closer to zero (see Table 36 in
Appendix 14). The total bias induced by these unmeasured confounders therefore
has no clear sign.

###  6.2 Generalizability

Assuming that our estimates among this sample of CTIS respondents are valid,
or more weakly that the magnitude of the effect estimates is correct, we may
wish to infer similar magnitude effects among all U.S. adults. Skepticism is
warranted for several reasons. First, CTIS respondents differ from the general
US population across several characteristics. For example, [12] found that the
CTIS survey over represents White individuals, those with higher education,
and those who have received COVID-19 vaccinations (see also [8]). Second, our
estimates may be subject to selection-bias: even if we knew the true sample
estimand, if sample selection were based on the potential or realized
outcomes, these effects may not generalize to a population of non-CTIS
respondents with identical covariates. Finally, our estimates make no claims
about the vaccine-hesitant, which is a substantial portion of the U.S.
population ([19]).

While caution is certainly warranted, we nevertheless argue that these
concerns should not entirely prohibit generalizations from our study. First,
assume that our conditional effect estimates are valid. Despite our sample’s
non-representative covariate distribution, we do not find evidence of
substantial sources of effect heterogeneity beyond age. Averages over other
mixes of vaccine-accepting populations are therefore plausibly similar.
Second, sample selection may in truth be a function of the observed or
potential outcomes. However, this does not imply our estimates are invalid:
our sample may not be representative of the prevalence of the potential
outcomes, but may still be representative of the risk-differences. Third, if
we wished to generalize across a population that includes a vaccine-hesitant
subset, we can simply assume that there is no treatment effect on average in
this subgroup. As long as the vaccine-hesitant comprise a minority of the
target population, similar magnitude effect sizes may therefore hold.191919As
a more formal analysis, in Appendix 13 we extend the previous sensitivity
analysis to evaluate how our effects might generalize across a population with
the same covariate distribution as our sample but include entirely cases where
$RS=0$. This analysis suggests that our ability to generalize our estimates
may be weak; however, it is also very conservative.

Such generalization may still strike some as implausible. We may instead wish
to consider the more limited goal of generalizing our estimates beyond the
complete-cases to the entire vaccine-accepting CTIS sample. We also run this
analysis and find that our estimated effect sizes are similar to our primary
analysis (see Appendix 14).

###  6.3 Interference, carryover, and anticipatory effects

Our analysis makes three restrictions on the dependence of the potential
outcomes across individuals and over time: SUTVA, no carryover effects, and no
anticipatory effects. While violations of these assumptions are possible, we
believe that such violations likely bias our effect estimates towards zero.

First consider violations of SUTVA. If the effect of receiving a COVID-19
vaccination depends on the vaccination status of other individuals in the
population, then we must redefine our causal estimand. First, assume that
these dependencies are limited to communities of individuals, which we index
from $c=1,...,C$. For simplicity, assume that each community has a fixed
number of individuals $p$, and that $p,C>1$. We can then define the following
terms: let $\mathbf{A}_{c-i}$ represent a random vector of vaccination
statuses for all individuals in a community $c$, omitting the index $ci$, and
let $S(\mathbf{A}_{c-i})$ be the average number of vaccinated individuals in
the random vector $\mathbf{A}_{c-i}$. Finally, let $\bar{1}_{c-i}$ be a vector
of $p-1$ ones and $\bar{0}_{c-i}$ be a vector of $p-1$ zeros (omitting the
$ci$-th index), so that, for example, $S(\bar{1}_{c-i})=1$.

Consider the following estimand:

$\displaystyle\vartheta=\mathbb{P}_{n}[Y^{1,\bar{1}}-Y^{0,\bar{0}}]$ (8)

This formalizes the difference in rates of depression in a world where
everyone is vaccinated versus a world where no one is vaccinated, allowing
each individual’s potential outcomes to depend on everyone else’s vaccination
status within their community. We can then consider how well have we estimated
$\vartheta$. Consider the model:

$\displaystyle P(Y^{A,\mathbf{A}}=1\mid
x)=\beta_{0x}+\beta_{1x}A_{ci}+\beta_{2x}S(\mathbf{A}_{c-i})$ (9)

Equation (9) implies that $\vartheta=\mathbb{P}_{n}(\beta_{1X}+\beta_{2X})$.
This estimand captures both a direct effect of vaccinations, captured by
$\mathbb{P}_{n}(\beta_{1X})$, and an indirect effect captured by
$\mathbb{P}_{n}(\beta_{2X})$. Assume that for all $x$,
$\beta_{1x},\beta_{2x}<0$ – that is, that both the direct and indirect effects
of vaccination are negative within all covariate strata. To make these
assumptions concrete, consider depression as the outcome. These assumption
would be reasonable if (1) vaccinations induce people to socialize, and
socializing decreases depression, and (2) socializing increases with the
number of vaccinated individuals within all covariate strata.

However, our estimates are not based on (9), but instead on the model:

$\displaystyle P(Y^{A,\mathbf{A}}=1\mid
x)=\tilde{\beta}_{0x}+\tilde{\beta}_{1x}A_{ci}$ (10)

Our estimators therefore target the biased quantity
$\tilde{\vartheta}=\mathbb{P}_{n}[\tilde{\beta}_{1X}]$. The omitted variable
bias formula implies that this estimand is positively biased with respect to
$\vartheta$.202020To see this, consider the model:
$\displaystyle\mathbb{E}[S(\mathbf{A}_{c-i})\mid
x]=\delta_{0x}+\delta_{1x}A_{ci}$ (11) Within each covariate stratum the bias
of our target quantity relative to our target is equal to
$\beta_{2x}(\delta_{1x}-1)$. The sample average of this bias across all
covariate strata is greater than zero: $\beta_{2x}<0$ for all $x$ by
assumption, and $\delta_{1x}\leq 1$ by the definition of
$S(\mathbf{A}_{c-i})$. As long as vaccinations aren’t perfectly correlated
within every community, there exists some $x^{\prime}$ in our sample where
$\delta_{1x^{\prime}}<1$, implying positive bias. If vaccinations were
independently assigned, $\delta_{1x}=0$, $\tilde{\beta}_{1x}=\beta_{1x}$, and
$\tilde{\vartheta}$ would correctly capture the direct effects of vaccination
on the outcomes (but not the total direct and indirect effects). The sign of
the bias of the direct effect is unclear and depends on $\delta_{1x}$. Whether
$\vartheta$ is the correct target of inference, or whether these assumptions
are realistic is debatable. Nevertheless, this logic provide justification for
believing that SUTVA violations would bias our effect estimates toward, rather
than away, from zero.212121[1] attempt to estimate spillover effects by
examining whether increases in community vaccination rates improve depression
and anxiety symptoms among the unvaccinated population. They are unable to
find evidence that this is the case, providing some empirical evidence that we
may need not worry about SUTVA violations affecting our estimates.

We also assume no carryover effects, precluding the potential outcomes from
depending on the time since receipt of vaccination. Were this violated, time
since vaccination would essentially be a confounder that is positively
associated with the observed treatment assignment indicator. Assuming that the
effects of COVID-19 vaccinations on mental health monotonically diminish over
time on average within each covariate stratum, violations of no carryover
effects would be bias our estimates towards zero. Finally, we assume no
effects in anticipation of receiving the vaccine. If anticipatory effects were
to improve mental health on average in our sample, violations would again bias
our effect estimates towards zero. In short, violations of SUTVA, no carryover
effects, and no anticipatory effects most plausibly bias our effect estimates
towards zero.

###  6.4 Bad controls

So-called “bad controls” may also bias our effect estimates ([2]). We argue
that these biases again plausibly move our estimates toward zero.

Specifically, we control for occupation, employment status, and worries about
finances. These variables are associated with vaccine-access, and – as noted
above – failing to control for such variable likely induces a negative bias in
our effect estimates. However, COVID-19 vaccinations may also affect these
variables, meaning that they act as a mediator on the causal pathway from
$A\to Y$, and are therefore “bad controls.” We argue that such effects were
weak in February 2021: acquiring or switching job can take considerable time,
and everyone in our sample was vaccinated for at most three months.
Additionally, any bias induced by over-controlling is likely positive. For
example, COVID-19 vaccinations likely boosted economic activity in aggregate
and increased employment. This would then likely reduce financial stress and
the associated mental health burdens. Controlling for variables along this
causal pathway therefore likely results in positive bias.

We examine these hypotheses in our data by rerunning our analyses without
controlling for worries about finances, occupation, or employment status. Our
point estimates move further away from zero (see Table 37 in Appendix 14).
This is consistent with either over-controlling moving our point estimates
closer to zero, or a negative bias induced by failing to control for these
variables. Because we prefer our estimates to be biased towards zero we
therefore control for these variables.

###  6.5 Measurement error

Measurement error may also bias our estimates. For example, our data is self-
reported, and may be subject to reporting bias. Relatedly, we assume no
unmeasured confounding assumption conditional on $X$, which includes both
observed covariate values and categories for missing data. The unobserved
values of these covariates may act as unmeasured confounders. We are unable to
fully address these issues. However, as one robustness check we run our
analysis while excluding respondents who provide implausible or unreasonable
answers. For example, several respondents indicate that they live with a
negative number of other individuals. Following [6], we consider a variety of
implausible answers to our questions and omit these responses when running our
analysis, removing 15,762 complete-cases. All results are consistent with our
primary analyses and are available in Appendix 14.

###  6.6 Mediation analysis

The validity of our mediation analysis requires strong assumptions. For
example, our model precludes a prominent possible pathway through which
COVID-19 vaccinations might affect depression: through household finances
([32], [5]). Our analysis instead assumes that vaccinations do not affect
depression by changing finances. This assumption may not strictly hold: at an
individual level, receipt of a COVID-19 vaccination might induce the
previously unemployed to seek employment, which might then change a person’s
personal finances. At an aggregate level, mass distribution of COVID-19
vaccinations may have boosted economic activity, which may affect any
individual’s employment status and therefore finances. As noted in Section
4.4, this is one reason we limit our analysis to February, a timeframe when
such effects are plausibly negligible.

Our analysis also requires that there exist no other post-treatment effects of
vaccination that confound the mediator-outcome relationship. For example, we
interpret the direct effect as reflecting the general feelings associated with
a return to normalcy. Our analysis requires that this feeling does not cause
social isolation or worries about health. However, causal arrows may exist
from social isolation or worries about health to this variable. In this case
both the direct and indirect pathways capture the effect of this
variable.222222[23] speculate that in addition to worries about health and
social isolation, “different work opportunities” is a possible pathway through
which COVID-19 vaccines may affect depression. While this pathway is related
to household finances, worries about household finances could remain the same
while one’s employment status may plausibly differ. More broadly our analysis
rules also out any pathways via employment changes. Limiting our analysis to
February again mitigates potential bias if we think people are unlikely to
change their employment within a short-time frame after being vaccinated.
These conditions and interpretations hold for any other hypothesized pathway
between COVID-19 vaccines and depression.

Finally, the interventional effects are not robust to reverse causation with
respect to $Y$. We must therefore interpret these results cautiously. While we
do not conduct a sensitivity analysis with respect any of these assumptions,
we compute all of the robustness checks noted previously and find comparable
results.

##  7 Discussion

We provide evidence that COVID-19 vaccinations reduced feelings of depression
and anxiety, social isolation, and worries about health, among vaccine-
accepting CTIS respondents in February 2021. Our results imply that not being
vaccinated decreased the prevalence of each outcome by 3.7, 3.3, and 4.3
percentage points, respectively, translating to 19, 16, and 15 percent
reductions in the prevalence of each outcome. We also examine effects by both
pre-specified and data-driven subgroups. We observe substantial heterogeneity
with respect to age, with younger age groups having larger magnitude effect
sizes. Our estimates on the risk-ratio scale suggest that some of this
heterogeneity may be driven by differences in base rates of each outcome by
age group. However, on both the risk-differences and risk-ratio scales, we
find that the total effects on each outcome are strongest among respondents
aged 18-24 relative to older respondents, with particularly large effects on
depression and isolation among those living with an elderly individual. We
also observe that those with multiple high-risk health conditions appear to
have slightly larger point estimates with respect to worries about health. Our
average effect estimates are moderately robust to violations to our causal
assumptions and analytic choices, and we argue that many violations would tend
to bias our effect estimates towards zero on the risk-differences scale.
Finally, while our estimates pertain only to our sample of CTIS respondents,
we argue that inferring similar magnitude effect sizes beyond our sample may
not be unreasonable.

We conclude by positing a model where vaccinations affect depression through
social isolation and worries about health. We then decompose the total effect
into a direct effect and indirect effects via interventions on the
distribution of social isolation and worries about health while holding
vaccination status fixed. We find that the interventional indirect effect via
social isolation is larger in absolute magnitude than the interventional
indirect effect via worries about health. We find evidence of a direct effect
that accounts for approximately half of the total effect on depression, which
we interpret as reflecting a pathway via a perceived return to normalcy (see
also [1]). These patterns are broadly consistent across our pre-specified
subgroups.

Our study has several implications. First, we find especially large effect
sizes among respondents aged 18-24. These results complement findings that the
pandemic had particularly negative effects on younger individuals ([27],
[28]), and suggest that COVID-19 vaccinations may have undone some of these
effects, at least temporarily. Second, our results suggest that effects on
social isolation play a larger mediating role than effects on worries about
health in explaining the effect of COVID-19 vaccinations on depression.
Finally, our results complement and support the findings from other studies
documenting and understanding this unintended beneficial outcome of COVID-19
vaccinations with respect to mental health ([23], [1]). For policy-makers
seeking to alleviate the mental health burden of the pandemic, our results
suggest targeting interventions that reduce feelings of social isolation,
particularly among younger individuals.

We conclude with three notes of caution. First, our results rely on strong
causal assumptions. Second, our primary statistical target of inference is the
treatment effect on a subset of vaccine-accepting CTIS-respondents, and not
the U.S. population more broadly. Finally, the CTIS measure of depression and
anxiety does not assess clinical criteria for mental disorders. Our measure,
like those used in other studies, likely captures a broad range of
psychological distress and demoralization than would be considered a
clinically significant disorder, and our findings may reflect improvements in
mental health status in a non-clinical range of severity ([22]; [29]). For
instance, symptoms of demoralization, such as helplessness, may be more
responsive to the introduction of a vaccine than symptoms of major depression,
such as anhedonia ([31]). While the results indicate improvements in mental
health status, these results should not be interpreted as indicating that
COVID-19 vaccinations had an impact on clinical disorders comparable to the
impact of psychotherapy or antidepressant medication. Regardless, future work
should continue to study mental health during the pandemic to seek to better
understand the extent of the challenges people are facing and to design
policies that can help alleviate suffering, particularly among society’s most
vulnerable populations.

## References

  * [1] Virat Agrawal, Jonathan H Cantor, Neeraj Sood, and Christopher M Whaley.  The impact of the covid-19 vaccine distribution on mental health outcomes.  Technical report, National Bureau of Economic Research, 2021.
  * [2] Joshua D Angrist and Jörn-Steffen Pischke.  Mostly harmless econometrics: An empiricist’s companion.  Princeton university press, 2009.
  * [3] Susan Athey and Guido Imbens.  Recursive partitioning for heterogeneous causal effects.  Proceedings of the National Academy of Sciences, 113(27):7353–7360, 2016.
  * [4] David Benkeser and Jialu Ran.  Nonparametric inference for interventional effects with multiple mediators.  Journal of Causal Inference, 9(1):172–189, 2021.
  * [5] Seth A Berkowitz and Sanjay Basu.  Unemployment insurance, health-related social needs, health care access, and mental health during the covid-19 pandemic.  JAMA internal medicine, 181(5):699–702, 2021.
  * [6] Alyssa Bilinski, Ezekiel Emanuel, Joshua A Salomon, and Atheendar Venkataramani.  Better late than never: Trends in covid-19 infection rates, risk perceptions, and behavioral responses in the usa.  Journal of general internal medicine, pages 1–4, 2021.
  * [7] Matteo Bonvini, Alec McClean, Zach Branson, and Edward H Kennedy.  Incremental causal effects: an introduction and review.  arXiv preprint arXiv:2110.10532, 2021.
  * [8] Valerie C Bradley, Shiro Kuriwaki, Michael Isakov, Dino Sejdinovic, Xiao-Li Meng, and Seth Flaxman.  Unrepresentative big surveys significantly overestimated us vaccine uptake.  Nature, 600(7890):695–700, 2021.
  * [9] Joshua Breslau, Melissa L Finucane, Alicia R Locker, Matthew D Baird, Elizabeth A Roth, and Rebecca L Collins.  A longitudinal study of psychological distress in the united states before and during the covid-19 pandemic.  Preventive medicine, 143:106362, 2021.
  * [10] Ruta Clair, Maya Gordon, Matthew Kroon, and Carolyn Reilly.  The effects of social isolation on well-being and life satisfaction during pandemic.  Humanities and Social Sciences Communications, 8(1), 2021.
  * [11] Gillian A Corbett, Sarah J Milne, Mark P Hehir, Stephen W Lindow, and Michael P O’connell.  Health anxiety and behavioural changes of pregnant women during the covid-19 pandemic.  European journal of obstetrics, gynecology, and reproductive biology, 249:96, 2020.
  * [12] Michael Daly, Andrew Jones, and Eric Robinson.  Public trust and willingness to vaccinate against covid-19 in the us from october 14, 2020, to march 29, 2021.  JAMA, 2021.
  * [13] Nancy Glober, George Mohler, Philip Huynh, Tom Arkins, Dan O’Donnell, Jeremy Carter, and Brad Ray.  Impact of covid-19 pandemic on drug overdoses in indianapolis.  Journal of Urban Health, 97(6):802–807, 2020.
  * [14] André Pereira Gonçalves, Ana Carolina Zuanazzi, Ana Paula Salvador, Alexandre Jaloto, Giselle Pianowski, and L de F Carvalho.  Preliminary findings on the associations between mental health indicators and social isolation during the covid-19 pandemic.  Archives of Psychiatry and Psychotherapy, 22(2):10–19, 2020.
  * [15] Guido W Imbens.  Nonparametric estimation of average treatment effects under exogeneity: A review.  Review of Economics and statistics, 86(1):4–29, 2004.
  * [16] Wolfram Kawohl and Carlos Nordt.  Covid-19, unemployment, and suicide.  The Lancet Psychiatry, 7(5):389–390, 2020.
  * [17] Edward H Kennedy.  Nonparametric causal effects based on incremental propensity score interventions.  Journal of the American Statistical Association, 114(526):645–656, 2019.
  * [18] Edward H Kennedy.  Towards optimal doubly robust estimation of heterogeneous causal effects.  arXiv preprint arXiv:2004.14497, 2020.
  * [19] Wendy C King, Max Rubinstein, Alex Reinhart, and Robin Mejia.  Covid-19 vaccine hesitancy january-may 2021 among 18–64 year old us adults by employment and occupation.  Preventive medicine reports, 24:101569, 2021.
  * [20] Alexander R Luedtke, Ivan Diaz, and Mark J van der Laan.  The statistics of sensitivity analyses.  2015\.
  * [21] Caleb H Miles.  On the causal interpretation of randomized interventional indirect effects.  arXiv preprint arXiv:2203.00245, 2022.
  * [22] JM Murphy.  Diagnosis, screening, and’demoralization’: epidemiologic implications.  Psychiatric developments, 4(2):101–133, 1986.
  * [23] Francisco Perez-Arce, Marco Angrisani, Daniel Bennett, Jill Darling, Arie Kapteyn, and Kyla Thomas.  Covid-19 vaccines and mental distress.  PloS one, 16(9):e0256406, 2021.
  * [24] Betty Pfefferbaum and Carol S North.  Mental health and the covid-19 pandemic.  New England Journal of Medicine, 383(6):510–512, 2020.
  * [25] Catherine E Robb, Celeste A de Jager, Sara Ahmadi-Abhari, Parthenia Giannakopoulou, Chinedu Udeh-Momoh, James McKeand, Geraint Price, Josip Car, Azeem Majeed, Helen Ward, et al.  Associations of social isolation with anxiety and depression during the early covid-19 pandemic: a survey of older adults in london, uk.  Frontiers in Psychiatry, 11, 2020.
  * [26] Joshua A Salomon, Alex Reinhart, Alyssa Bilinski, Eu Jing Chua, Wichada La Motte-Kerr, Minttu M Rönn, Marissa B Reitsma, Katherine A Morris, Sarah LaRocca, Tamer H Farag, et al.  The us covid-19 trends and impact survey: Continuous real-time measurement of covid-19 symptoms, risks, protective behaviors, testing, and vaccination.  Proceedings of the National Academy of Sciences, 118(51):e2111454118, 2021.
  * [27] Damian F Santomauro, Ana M Mantilla Herrera, Jamileh Shadid, Peng Zheng, Charlie Ashbaugh, David M Pigott, Cristiana Abbafati, Christopher Adolph, Joanne O Amlag, Aleksandr Y Aravkin, et al.  Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the covid-19 pandemic.  The Lancet, 398(10312):1700–1712, 2021.
  * [28] Elvira Sojli, Wing Wah Tham, Richard Bryant, and Michael McAleer.  Covid-19 restrictions and age-specific mental health—us probability-based panel evidence.  Translational Psychiatry, 11(1):1–7, 2021.
  * [29] L Tecuta, E Tomba, S Grandi, and GA Fava.  Demoralization: a systematic review on its clinical characterization.  Psychological medicine, 45(4):673–691, 2015.
  * [30] Stijn Vansteelandt and Rhian M Daniel.  Interventional effects for mediation analysis with multiple mediators.  Epidemiology (Cambridge, Mass.), 28(2):258, 2017.
  * [31] Marcus Wellen.  Differentiation between demoralization, grief, and anhedonic depression.  Current psychiatry reports, 12(3):229–233, 2010.
  * [32] Jenna M Wilson, Jerin Lee, Holly N Fitzgerald, Benjamin Oosterhoff, Baris Sevi, and Natalie J Shook.  Job insecurity and financial concern during the covid-19 pandemic are associated with worse mental health.  Journal of occupational and environmental medicine, 62(9):686–691, 2020.

##  8 Sample characteristics

Table 4 displays the participant flow into the analytic sample. Tables
LABEL:tab:sumtab1-LABEL:tab:sumtab3 display the demographic characteristics of
both the analytic sample and the entire February sample. Table
LABEL:tab:sumtab1 focuses on the treatment, outcomes, and mediators. Table
LABEL:tab:sumtab2 focuses on respondent-level demographic characteristics.
Table LABEL:tab:sumtab3 focuses on county-level characteristics, where we
divide the continuous variables into quintiles at the county-level and display
the proportion of respondents falling into each quintile. Each table
separately display the percentage and number of respondents in each subgroup,
and the percent vaccinated within each demographic subgroup. We display this
information for both the analytic sample and the entire February dataset.
Importantly, the percent vaccinated within each subgroup represents the
percent among those who responded to both the corresponding demographic
question and the COVID-19 vaccination question on the CTIS.

Table 4: Participant flow Sample subset | Total # | Non-hesitant #
---|---|---
Entire Sample | 1232398 | 991629
English or Spanish surveys | 1229868 | 989417
Alaska | 3690 | 2974
Missing FIPS | 39526 | 29685
Missing Intent | 75443 | 75443
Vaccine Hesitant | 229894 | 0
Non-hesitant sample | 881315 | -
Missing Depressed | 93355 | -
Missing Vaccinated | 5355 | -
Missing Isolated | 94575 | -
Missing Worried | 91156 | -
Complete-cases | 758594 | -
Table 5: Summary statistics: hesitancy, outcome, mediators, and treatment assignment  | Subset: vaccine-accepting complete-cases | Entire February sample
---|---|---
Group | % of subset (N) | % of group vaccinated | % of subset (N) | % of group vaccinated
Vaccine intent
Already vaccinated | 29.4 (278416) | 100 | 28.7 (319989) | 100
Yes, definitely | 38.3 (362403) | - | 37 (412120) | -
Yes, probably | 12.4 (117775) | - | 12.9 (143851) | -
No, probably not | 9.9 (93596) | - | 10.3 (114328) | -
No, definitely not | 9.9 (94135) | - | 10.3 (114270) | -
No response | 0.1 (1053) | - | 0.9 (9878) | -
Vaccinated
Not vaccinated | 70.6 (668962) | - | 71.3 (794447) | -
Vaccinated | 29.4 (278416) | 100 | 28.7 (319989) | 100
No response | 0 (0) | - | 0 (0) | -
Depressed
Not depressed or anxious | 81.2 (769435) | 31.9 | 71.3 (794576) | 31.9
Depressed or anxious | 18.8 (177943) | 18.3 | 16.6 (184752) | 18.4
No response | 0 (0) | - | 12.1 (135108) | 23.8
Worried about health
Worried | 25.1 (237887) | 23.3 | 22.3 (247971) | 23.5
Somewhat worried | 41.8 (396144) | 32.5 | 36.6 (407658) | 32.8
Not too worried | 22.8 (216225) | 33.4 | 20 (222949) | 33.8
Not worried at all | 10.3 (97122) | 22.5 | 9.1 (101314) | 23
No response | 0 (0) | - | 12.1 (134544) | 21.8
Isolated
None of the time | 43.8 (414960) | 31 | 38.4 (428360) | 31
Some of the time | 36.9 (349454) | 31.8 | 32.4 (361024) | 31.8
Most of the time | 13.2 (125013) | 23.7 | 11.5 (128698) | 23.7
All of the time | 6.1 (57951) | 15.5 | 5.4 (59796) | 15.6
No response | 0 (0) | - | 12.3 (136558) | 23.8
Note: all numbers exclude non-respondents to COVID-19 vaccination
Table 6: Summary statistics: individual-level characteristics  | Subset: vaccine-accepting complete-cases | Entire February sample
---|---|---
Group | % of subset (N) | % of group vaccinated | % of subset (N) | % of group vaccinated
Age group
18-24 years | 4.4 (41398) | 12.2 | 3.8 (41941) | 12.1
25-34 years | 12.7 (120509) | 19.1 | 11 (123112) | 19
35-44 years | 16 (151208) | 21.2 | 14 (156206) | 21.1
45-54 years | 16.9 (159671) | 21.8 | 15 (166973) | 21.6
55-64 years | 20.1 (190546) | 21.5 | 18.1 (201971) | 21.5
65-74 years | 19.8 (187875) | 47.5 | 18.1 (201611) | 47.4
75 or more years | 8.2 (78033) | 62.3 | 7.8 (87256) | 62.2
No response | 1.9 (18138) | 25.7 | 12.1 (135366) | 21.7
Race
Native American | 1.3 (12277) | 31.3 | 1.2 (13322) | 31.6
Asian or Pacific Islander | 2.3 (22154) | 30.6 | 2.1 (23466) | 31.1
Black | 6.7 (63327) | 24.5 | 6.2 (69637) | 25
White | 79.1 (749693) | 31.2 | 70.3 (783131) | 31.5
Other | 4.9 (46391) | 15.4 | 4.6 (51216) | 15.4
Multi-racial | 2.5 (23358) | 19.2 | 2.2 (24204) | 19.5
No response | 3.2 (30178) | 21.5 | 13.4 (149460) | 21.1
Ethnicity
Hispanic | 11.5 (109142) | 19.8 | 10.7 (119759) | 19.8
Not Hispanic | 85.9 (813430) | 30.8 | 76.4 (851381) | 31.1
No response | 2.6 (24806) | 26.3 | 12.9 (143296) | 22
Gender
Male | 31.9 (302520) | 28.2 | 28.6 (318320) | 28.4
Female | 64.4 (610049) | 30.6 | 57.7 (643546) | 30.8
Non-binary | 0.6 (5545) | 15.6 | 0.5 (5687) | 15.9
No response | 3.1 (29264) | 19 | 13.2 (146883) | 20.6
Facebook User Language
English | 96.8 (917134) | 30 | 96 (1069824) | 29.5
Spanish | 3.2 (30244) | 10 | 4 (44612) | 10.1
Other | 0 (0) | - | 0 (0) | -
Education level
High school, GED, or less | 19 (179567) | 18.3 | 17.7 (197564) | 18.8
Some college or two year degree | 35.3 (334843) | 26.8 | 31.6 (351924) | 27.3
Four year degree | 23.6 (223765) | 31.8 | 20.7 (231012) | 32.2
Masters | 13.9 (131363) | 41.5 | 12.2 (135543) | 41.8
JD or MD | 3.2 (29984) | 46.9 | 2.8 (31067) | 47.1
PhD | 2.2 (20724) | 43 | 1.9 (21410) | 43
No response | 2.9 (27132) | 25.8 | 13.1 (145916) | 22
Employment
Work outside home | 37.4 (354388) | 32 | 32.9 (366618) | 32
Work at home | 13.3 (126067) | 20.1 | 11.6 (129359) | 20.3
Does not work for pay | 45.8 (433452) | 30.2 | 41.8 (465898) | 30.6
No response | 3.5 (33471) | 26.1 | 13.7 (152561) | 22.2
Occupation
Healthcare | 8.6 (81621) | 70.1 | 7.6 (84363) | 70.1
Education | 5.6 (52691) | 37.3 | 4.8 (53993) | 37.4
Services | 3.6 (34052) | 13.9 | 3.2 (35576) | 14.3
Protective service | 0.7 (6314) | 43.1 | 0.6 (6504) | 43.3
Not working | 45.8 (433452) | 30.2 | 41.8 (465898) | 30.6
Other | 33 (312650) | 17.9 | 29 (323006) | 18
No response | 2.8 (26598) | 26.3 | 13 (145096) | 22.1
High-risk conditions
None | 36.5 (345704) | 24.6 | 35.2 (392333) | 24.5
One | 28.2 (267362) | 30.6 | 27.1 (301554) | 30.7
Two | 17.6 (166883) | 33.9 | 16.9 (188635) | 34.1
Three or more | 16.7 (158361) | 33.4 | 16.2 (180074) | 33.6
Missing | 1 (9068) | 21 | 4.7 (51840) | 12.5
Household: adults 65 and over
No | 47.3 (448427) | 21 | 46.5 (517715) | 20.3
Yes | 37.2 (352093) | 43.8 | 37.5 (418388) | 42.9
Missing | 15.5 (146858) | 20.4 | 16 (178333) | 19.9
Children
Children in school full time | 10.2 (96769) | 22.5 | 9 (100368) | 22.5
Children in school part time | 4.6 (43318) | 24 | 4 (44569) | 24
Children not in school | 21.4 (203211) | 24.3 | 23.6 (263008) | 22.8
No children | 47.6 (450498) | 31.9 | 46.4 (516689) | 31.4
No response | 16.2 (153582) | 34.6 | 17 (189802) | 33.9
Lives alone
Lives with others | 97.8 (926150) | 29.4 | 97.3 (1084376) | 28.8
Lives alone | 1.2 (10901) | 23.8 | 1.3 (14461) | 22.8
No response | 1.1 (10327) | 31.8 | 1.4 (15599) | 30.9
Worried about finances
Very worried | 18.8 (178283) | 14.6 | 17.1 (190407) | 14.8
Somewhat worried | 24.3 (230569) | 24.9 | 22 (244857) | 25.2
Not too worried | 28 (265430) | 32.5 | 24.9 (277803) | 32.9
Not worried at all | 28.7 (272223) | 39.8 | 25.5 (284565) | 40.2
No response | 0.1 (873) | 32.2 | 10.5 (116804) | 21
Received Flu Vaccine
Yes | 61.2 (580095) | 39.8 | 55 (612686) | 40.1
No or unsure | 37.6 (356036) | 12.7 | 33.6 (374011) | 12.8
No response | 0.4 (3811) | 26.7 | 10.7 (119721) | 21.2
Ever tested for COVID-19
Never tested | 44.3 (420020) | 26.8 | 44.4 (494688) | 26.1
Previously tested, not positive/unknown | 45 (426020) | 33.9 | 44.5 (495480) | 33.3
Previously tested positive | 10.5 (99225) | 20.6 | 10.9 (121059) | 20.3
No response | 0.2 (2113) | 39.8 | 0.3 (3209) | 38.7
Note: all numbers exclude non-respondents to COVID-19 vaccination
Table 7: Summary statistics: county-level characteristics  | Subset: vaccine-accepting complete-cases | Entire February sample
---|---|---
Group | % of subset (N) | % of group vaccinated | % of subset (N) | % of group vaccinated
County urban classification
Large central metro | 22.6 (214116) | 27.1 | 22.9 (255300) | 26.2
Large fringe metro | 22.1 (209298) | 27.6 | 21.9 (243660) | 27.1
Medium metro | 27.5 (260430) | 30.6 | 27.4 (305233) | 30
Small metro | 11.5 (108819) | 31.9 | 11.4 (127459) | 31.1
Micropolitan | 10 (94437) | 30.8 | 10 (111281) | 30
Non-core | 6.4 (60278) | 31.8 | 6.4 (71503) | 31.3
No response | 0 (0) | - | 0 (0) | -
County population per square mile quintile
1st quintile | 2 (18714) | 36.6 | 2 (21818) | 36.2
2nd quintile | 4 (37875) | 32.1 | 4 (44855) | 31.4
3rd quintile | 6.9 (65545) | 30.7 | 6.9 (77419) | 30
4th quintile | 14.7 (138973) | 29.9 | 14.7 (163554) | 29.2
5th quintile | 72.4 (686271) | 28.8 | 72.4 (806790) | 28.1
County population quintile
1st quintile | 1.1 (10233) | 35.3 | 1.1 (12134) | 35.1
2nd quintile | 2.8 (26960) | 32.5 | 2.9 (32000) | 31.7
3rd quintile | 5.7 (54136) | 30.5 | 5.8 (64338) | 29.8
4th quintile | 12.9 (122343) | 30.3 | 12.9 (143984) | 29.6
5th quintile | 77.4 (733706) | 29 | 77.3 (861980) | 28.3
County percent in poverty quintile
1st quintile | 30.2 (286253) | 28.7 | 29.6 (329680) | 28.3
2nd quintile | 21.4 (202372) | 29.7 | 21.2 (236068) | 29.1
3rd quintile | 23.7 (224853) | 28.9 | 23.9 (265980) | 28.1
4th quintile | 15.8 (149990) | 29.9 | 16.1 (179415) | 29
5th quintile | 8.9 (83910) | 31.3 | 9.3 (103293) | 30
County gini quintile
1st quintile | 9.4 (88737) | 28.5 | 9.2 (102709) | 28
2nd quintile | 14.9 (141454) | 29.6 | 14.7 (164232) | 29.1
3rd quintile | 17.9 (169122) | 29 | 17.7 (197780) | 28.4
4th quintile | 29.3 (277916) | 29.7 | 29.2 (325916) | 29.1
5th quintile | 28.5 (270149) | 29.4 | 29.1 (323799) | 28.6
County percent uninsured quintile
1st quintile | 31.1 (294470) | 29.1 | 30.5 (339515) | 28.7
2nd quintile | 20.8 (196935) | 29.2 | 20.7 (230724) | 28.5
3rd quintile | 18.5 (175327) | 29.4 | 18.6 (207344) | 28.7
4th quintile | 17.1 (162180) | 29.4 | 17.3 (192769) | 28.6
5th quintile | 12.5 (118466) | 30.3 | 12.9 (144084) | 29.3
County Biden vote lead quintile
1st quintile | 3 (28096) | 28 | 3 (33565) | 27.3
2nd quintile | 5.8 (54815) | 29.6 | 5.8 (64888) | 29.1
3rd quintile | 10.7 (101312) | 30 | 10.7 (119217) | 29.4
4th quintile | 21.3 (201454) | 30.3 | 21.2 (236127) | 29.6
5th quintile | 59.3 (561701) | 29 | 59.3 (660639) | 28.3
County deaths last two weeks of January sextile
1st sextile | 2.2 (20901) | 32.1 | 2.2 (24378) | 31.4
2nd sextile | 19.7 (186968) | 29.3 | 19.2 (213853) | 28.9
3rd sextile | 31.1 (295100) | 29.4 | 31.1 (346996) | 28.7
4th sextile | 23.9 (226849) | 29 | 24.2 (269917) | 28.2
5th sextile | 15.2 (143757) | 29.7 | 15.3 (170893) | 29
6th sextile | 7.8 (73803) | 29.4 | 7.9 (88399) | 28.5
No response | 0 (0) | - | 0 (0) | -
Note: all numbers exclude non-respondents to COVID-19 vaccination

##  9 Other estimands

As supplemental analyses we consider three additional estimands beyond those
included in the main paper.

###  9.1 Observed effects

Equation (2) attempts to answer the question: what would have happened in a
world where all respondents in our sample had been vaccinated in February 2021
contrasted against a world where no one had been vaccinated? Imagining this
former world in February 2021 is arguably challenging; since production and
distribution constraints likely prevented this from being possible. We can
instead ask, what was the effect of the observed distribution of vaccinations?
That is, how much did vaccine distribution in the factual world reduce
depression compared to a counterfactual world where no one was been vaccinated
in our analytic sample? Recalling that $Z=RSV$, equation 12 formalizes this:

$\displaystyle\psi^{obs}$
$\displaystyle=\mathbb{P}_{n}\left(\mathbb{E}[Y-Y^{0}\mid X,Z=1]\right)$ (12)

###  9.2 Incremental effects

We can generalize the above question to further ask: how would rates of
depression, isolation, and worries about health have changed were we able to
change everyone’s probability of being vaccinated in February 2021? To answer
this, we use the incremental intervention effects proposed by [17], formalized
in equation (13):

$\displaystyle\psi^{Q(\delta)}=\mathbb{P}_{n}\left[\frac{\delta\pi_{1}(X)Y^{1}+(1-\pi_{1}(X))Y^{0}}{\delta\pi_{1}(X)+1-\pi_{1}(X)}\right]$
(13)

where $\pi_{1}(X)=P(A=1\mid X,V=1)$ and $\delta$ represents a parameter that
multiplies everyone’s observed odds of being vaccinated. For example, take
$\delta=1.5$ and consider an individual with a 10 percent chance of being
vaccinated (or 1/9 odds). This intervention would increase this individual’s
odds to 3/18, leaving them with approximately 14 percent probability of being
vaccinated. As special cases of interest, $\delta=1$ corresponds to average
outcomes under the observed distribution of vaccinations (or
$\mathbb{P}_{n}[\mathbb{E}[Y\mid X,Z=1]]$), $\delta=0$ corresponds to the
average outcomes when no one is vaccinated ($\mathbb{P}_{n}[\mu_{0}(X)]$), and
$\delta\to\infty$ corresponds to the average outcomes when everyone is
vaccinated ($\mathbb{P}_{n}[\mu_{1}(X)]$).

We can also estimate contrasts setting different values of $\delta$ to see how
mental health would have changed under different interventions on the
probability of being vaccinated. Notice that (12) is an example of such a
contrast between $\delta=1$ and $\delta=0$. This highlights an important
distinction between (12) and (13): the former involves a causal contrast while
the latter reflects a counterfactual rate. We refer to [17] and [7] for more
details.

###  9.3 Pandemic effects

One reason we are interested in the effect of COVID-19 vaccinations on mental
health is because we believe that it may “undo” some of the effects of the
pandemic on mental health. We may therefore also wish to understand the
effects of the pandemic on mental health in a world without COVID-19
vaccinations. Let $C$ be an indicator of the COVID-19 pandemic. We can then
consider the analogue to our primary causal estimand ($\psi$):

$\displaystyle\psi^{c}$
$\displaystyle=\mathbb{P}_{n}[\mathbb{E}[Y^{C=1,A=0}-Y^{C=0,A=0}\mid X,Z=1]]$
(14)

We can also consider effects the risk-ratio scale. Under some assumptions, we
can use our estimates of $\psi$ to obtain lower bounds on both quantities (see
Appendix 10).

##  10 Identification

This section details our causal models. The first subsection outlines all
identifying assumptions. The second subsection provides expressions of our
causal estimands in terms of the observed data distribution, and provides
briefs proofs of these identification results. These results cover both the
estimands in the main paper and the additional estimands outlined in Appendix
9.

###  10.1 Causal assumptions

As noted previously, the CTIS is a repeated cross-sectional sample of Facebook
Users. Additionally, the CTIS only indicates whether a respondent was
vaccinated, but not when, and does not contain information on anyone’s
feelings of depression, isolation, or worries about health over time. While in
truth the causal process may be complex (see Remark 3), we make several
simplifying assumptions to identify the causal effects given this data.

Let $t=0,...,T$ index calendar days starting from December 1, 2020 (slightly
prior to the first administrations of COVID-19 vaccinations) and $T$ indexes
December 1, 2021. More generally we could consider any arbitrary start date
before COVID-19 vaccinations were first administered and afterwards. Define
$\mathbf{a}_{1t}$ to be a fixed vector of all vaccination-statuses across all
$N$ individuals in the U.S. population at time $t$. Assumption 4 precludes
carryover or anticipatory effects either for the outcome or the mediators
during February 2021. Specifically, we assume that, for any time-period $t$
within February 2021:

######  Assumption 4 (No carryover effects).

$\displaystyle
Y_{it}^{\mathbf{a}_{1}\mathbf{m}_{11}\mathbf{m}_{21},...,\mathbf{a}_{T}\mathbf{m}_{1T}\mathbf{m}_{2T}}=Y_{it}^{\mathbf{a}_{t}\mathbf{m}_{1t}\mathbf{m}_{2t}}$
(15) $\displaystyle
M_{1it}^{\mathbf{a}_{1},...,\mathbf{a}_{T}}=M_{1it}^{\mathbf{a}_{t}}$ (16)
$\displaystyle
M_{2it}^{\mathbf{a}_{1},...,\mathbf{a}_{T}}=M_{2it}^{\mathbf{a}_{t}}$ (17)

Assumption 5, or consistency, further restricts the dependence of the
potential outcomes (dropping the indices $it$ above for clarity). Consistency
implies that each unit has potential outcomes (potential mediator values)
equal to their observed outcomes (observed mediators) given their treatment
and mediator values (and their treatment values), and does not depend on the
treatment assignment or mediator value (or treatment assignment) of any other
individual.

######  Assumption 5 (Consistency).

$\displaystyle(A_{it}=a_{it},M_{1it}=m_{1it},M_{2it}=m_{2it})\implies
Y_{it}=Y_{it}^{a_{it}m_{1it}m_{2it}}=Y^{\mathbf{a}_{t}\mathbf{m}_{1t}\mathbf{m}_{2t}}$
(18) $\displaystyle A_{it}=a_{it}\implies
M_{1it}=M_{1it}^{a_{it}}=M_{1}^{\mathbf{a}_{t}},M_{2it}=M_{2it}^{a_{it}}=M_{2it}^{\mathbf{a}_{t}}$
(19)

To ease notation, we drop the indices $it$ moving forward. We again assume no
unmeasured confounding and M-Y ignorability, detailed in equations (4)-(5),
and reproduce these assumptions below for completeness.

######  Assumption 6 (No unmeasured confounding).

$\displaystyle(Y^{am_{1}m_{2}},M_{1}^{a},M_{2}^{a})\perp A\mid(X,V=1)$ (20)

######  Assumption 7 (Y-M ignorability).

$\displaystyle Y^{am_{1}m_{2}}\perp(M_{1},M_{2})\mid(A=a,X,V=1)$ (21)

No unmeasured confounding states that treatment assignment is independent of
the potential outcomes and potential mediators conditional on covariates and
vaccine-acceptance, while Y-M ignorability states that the potential outcomes
are independent of the mediators conditional on treatment assignment,
covariates, and vaccine-acceptance. This precludes the existence of other
causal descendants of $A$ that confound the mediator-outcome relationship.

######  Remark 1.

For our outcomes analysis we only require Assumption 6 for identification. By
contrast, our mediation analysis requires Assumption 7. We also implicitly
assume no reverse causation between the mediators and outcome: that is, that
depression does not cause feelings of isolation and worries about health.

######  Remark 2.

These assumptions reduce a potentially more complex longitudinal process into
a simpler structure that permits causal identification given our data. To
illustrate, we consider our outcomes analysis, and let
$W_{t}=[Y_{t},M_{1t},M_{2t}]$. For simplicity we assume Consistency and No
Anticipatory Effects. We also illustrate only four time points, though the
insights hold across a longer time-frame:

$X$$W_{-1}$$A_{0}$$W_{0}$$A_{1}$$W_{1}$$A_{2}$$W_{2}$ Figure 9: Causal
structure: unrestricted

This model allows for both vaccination status and the outcomes to depend on
the prior periods vaccination status and prior outcomes. This model further
implies the following conditional independence assumptions for any
$(a_{0},a_{1},a_{2})$:

$\displaystyle W_{0}(a_{0})\perp A_{0}\mid(X,W_{-1})$ $\displaystyle
W_{1}(a_{1},a_{0})\perp A_{1}\mid(X,W_{-1},W_{0},A_{0})$ $\displaystyle
W_{2}(a_{2},a_{1},a_{0})\perp A_{2}\mid(X,W_{-1},W_{0},W_{1},A_{0},A_{1})$

Because we do not observe each person’s outcomes or vaccination status over
time – only at the time they respond to the survey – we assume no carryover
effects and no unmeasured confounding. Figure 10 illustrates these exclusions
contrasted against Figure 9. First, we assume that vaccination status at any
time $t$ is not a function of any contemporaneous or prior outcomes. Second,
we assume that that the outcomes at time $t$ are only a function of that
period’s treatment status. We do allow that prior-period outcomes can affect
current outcomes; however, they cannot affect treatment assignment at any
point; otherwise, these variables would be a confounder.

$X$$W_{-1}$$A_{0}$$W_{0}$$A_{1}$$W_{1}$$A_{2}$$W_{2}$ Figure 10: Causal
structure: restricted

######  Remark 3.

Allowing for carryover effects would imply that only causal estimands at time
$t=0$ would be identified (again assuming $W_{-1}$ is not a confounder). This
motivates our decision to analyze data only from February 2021: this time-
period is “closer” to $t=0$. Heuristically, we may expect that the bias
induced from violations of this assumption are smaller when closer to this
“first” time-period and that this bias grows in magnitude over time. This may
occur, for example, when treatment effects change monotonically with time
since vaccination. As a related point, if our observed covariates are highly
correlated with time since vaccination, controlling for these variables may
reduce this bias.

######  Remark 4.

Modeling carryover effects may be of interest. For example, we may wish to
know whether the effects of COVID-19 vaccinations on mental health endure over
time. Unfortunately, even if we observed the differences in a pair of
potential outcomes for each individual sampled at time $t$, we could not
necessarily identify how effects change as a function of time since treatment
(“timevacc”). To illustrate this point, consider the model:

$\displaystyle
Y_{it}^{1}-Y_{it}^{0}=\beta_{0X}+\beta_{1X}t+\beta_{2X}timevacc_{it}$

where $timevacc_{it}=\max\\{0,\sum_{k=0}^{t}(A_{ik}-1)\\}$. $\beta_{0X}$
captures the initial effect of treatment, $\beta_{1X}$ represents
heterogeneity in the treatment effect as a linear function of the time index,
$\beta_{2X}$ captures the durability of the treatment effect over time, and
the coefficients are a function of each individual’s covariates $X$. Assume
that we observe $Y_{it}^{1}-Y_{it}^{0}$ for each individual when sampled at
time $t$, and let $Y_{it}^{1}-Y_{it}^{0}$ equal $\beta_{0X}+\beta_{1X}t$ for
individuals not yet vaccinated. That is, the treatment effect is assumed to
equal what would have happened were they vaccinated that same time period.

Consider the case where for all $x$, $\beta_{0x}>0$, $\beta_{1x}>0$,
$\beta_{2x}<0$, and $\lvert\beta_{1x}\rvert>\lvert\beta_{2x}\rvert$. That is,
there is some positive initial effect of treatment which increases with the
time index, but that decreases with time since treatment (i.e. the effects are
not “durable”). In this case we would observe that the average effects
increase with time; however, we would be wrong to infer that effects are
increasing with the length of treatment. This is because in this scenario we
cannot separately identify heterogeneity across time from carryover effects.

This illustrates the difficulty of modeling such effects generally. And in
fact, as noted in Remark 3, we must actually preclude the existence of
carryover effects identify our causal estimands given our data structure.
Therefore, our particular data makes it difficult, if not impossible, to learn
about effect durability, or carryover effects generally.

We next assume random sample selection to rule out collider-bias by
conditioning on our sample. This assumption is detailed in (6), but reproduced
below for completeness.

######  Assumption 8 (Random sample selection).

$\displaystyle(S,R)\perp(Y^{am_{1}m_{2}},M_{1}^{a},M_{2}^{a})\mid(A,X,V=1)$
(22)

where $R=R_{y}R_{a}R_{m_{1}}R_{m_{2}}$.

Finally, we assume positivity of the probability of treatment assignment, the
joint mediator probabilities, and being a complete-case for all values of
$(a,m_{1},m_{2})$ in the support of the data and $\epsilon>0$:232323Our
primary estimand is the complete-case estimator, which does not technically
require (26); however, this is required to estimate the effects over all non-
hesitant respondents where $V=1$.

######  Assumption 9 (Positivity).

$\displaystyle P\left\\{\min_{a}P(A=a\mid Z=1,X)>\epsilon\right\\}=1$ (24)
$\displaystyle P\left\\{\min_{a,m_{1},m_{2}}P(M_{1}=m_{1},M_{2}=m_{2}\mid
Z=1,A=a,X)>\epsilon\right\\}=1$ (25) $\displaystyle P\left\\{P(R=1\mid
V=1,S=1,X)>\epsilon\right\\}=1$ (26)

These assumptions are sufficient to identify our causal parameters.

####  10.1.1 Identification

We provide all identification results conditional on $X$, noting that the
target estimands are simply the empirical average of these expressions across
the analytic sample.

######  Proposition 1 (Mean potential outcomes).

Let $W^{a}=[Y^{a},M_{1}^{a},M_{2}^{a}]$. Under assumptions (4), (5) (6), (8),
and (9) $\mathbb{E}[W^{a}\mid X,Z=1]=\mathbb{E}[W\mid X,A=a,Z=1]$.

###### Proof of Proposition 1.

First, notice that
$Y^{a}=\sum_{m_{1}^{\prime},m_{2}^{\prime}}\mathds{1}(M_{1}^{a}=m_{1},M_{2}^{a}=m_{2})Y^{am_{1}m_{2}}$.
Therefore, $(Y^{am_{1}m_{2}},M_{1}^{a},M_{2}^{a})\perp A\mid
X\implies(Y^{a},M_{1}^{a},M_{2}^{a})\perp A\mid X$. Then:

$\displaystyle\mathbb{E}[W^{a}\mid Z=1,X]$
$\displaystyle=\sum_{a^{\prime}}\mathbb{E}[W^{a}\mid
Z=1,X,A=a^{\prime}]P(A=a^{\prime}\mid X,Z=1)$
$\displaystyle=\sum_{a^{\prime}}\mathbb{E}[W^{a}\mid
X,V=1,A=a^{\prime}]P(A=a^{\prime}\mid X,Z=1)$
$\displaystyle=\sum_{a^{\prime}}\mathbb{E}[W^{a}\mid
X,V=1,A=a]P(A=a^{\prime}\mid X,Z=1)$ $\displaystyle=\mathbb{E}[W^{a}\mid
X,V=1,A=a]$ $\displaystyle=\mathbb{E}[W^{a}\mid X,Z=1,A=a]$
$\displaystyle=\mathbb{E}[W\mid X,Z=1,A=a]$

where the first equality holds by the Law of Iterated Expectations, the second
by random sample selection, the third by no unmeasured confounding, the fourth
follows directly by the third, the fifth by no sample selection, and the sixth
by consistency. ∎

######  Proposition 2 (Joint counterfactual outcomes).

The following identification result holds:

$\displaystyle\mathbb{E}[Y^{am_{1}m_{2}}\mid Z=1,X]=\mathbb{E}[Y\mid
Z=1,X,A=a,M_{1}=m_{1},M_{2}=m_{2}]$

###### Proof of Proposition 2.

Let $M=M_{1}\times M_{2}$. Then:

$\displaystyle\mathbb{E}[Y^{am}\mid Z=1,X]$
$\displaystyle=\mathbb{E}[Y^{am}\mid X,V=1,A=a]$
$\displaystyle=\mathbb{E}[Y^{am}\mid X,V=1,A=a,M=m]$
$\displaystyle=\mathbb{E}[Y^{am}\mid X,A=a,M^{a}=m,V=1,RS=1]$
$\displaystyle=\mathbb{E}[Y\mid X,A=a,M=m,Z=1]$

where the first equality holds by LIE, no sample selection, and no unmeasured
confounding (following the first four equalities in Proof 10.1.1); the second
by Y-M ignorability, the third by consistency and the fact that
$(R,S)\perp(Y^{am},M^{a})\mid X,A,V=1\implies(R,S)\perp
Y^{am}\mid(M^{a}=m,X,V=1,A=a)$, and the final line by consistency. ∎

######  Proposition 3 (Interventional effects).

We provide the identifying expression for $\psi_{M_{1}}$, noting that the
other interventional effects can be derived similarly.

$\displaystyle\sum_{m_{1},m_{2}}\mathbb{E}[Y^{1m_{1}m_{2}}\mid
X,Z=1]\\{p(M_{1}^{1}=m_{1}\mid X,Z=1)$ $\displaystyle-p(M_{1}^{0}=m_{1}\mid
X,Z=1)\\}p(M_{2}^{0}=m_{2}\mid X,Z=1)$
$\displaystyle=\sum_{m_{1},m_{2}}\mathbb{E}[Y\mid
X,1,m_{1},m_{2},Z=1]\\{p(m_{1}\mid X,1,Z=1)$ $\displaystyle-p(m_{1}\mid
X,0,Z=1)\\}p(m_{2}\mid X,0,Z=1)$

###### Proof of Proposition 3.

The proof follows directly from Propositions LABEL:proposition:mpo and
LABEL:proposition:jointpo. ∎

######  Remark 5.

Equation 25 is technically stronger than necessary to identify $\psi_{M_{1}}$.
For example, for any $x$, we only need that $p(m_{1},m_{2}\mid 1,x)>0$
whenever $p(m_{2}\mid 0,x)>0$.

######  Proposition 4 (Incremental effects).

Assume (27)

$\displaystyle(R,S)\perp(A,Y^{a})\mid(X,V=1)$ (27)

In other words, missingness is independent of treatment assignment given
covariates and vaccine-acceptance. Then:

$\displaystyle\mathbb{E}[Y^{Q(\delta)}\mid X,Z=1]$ $\displaystyle=\frac{\delta
P(A=1\mid X,Z=1)\mathbb{E}[Y\mid A=1,X,Z=1]}{\delta P(A=1\mid X,Z=1)+P(A=0\mid
X,Z=1)}$ $\displaystyle+\frac{P(A=0\mid X,Z=1)\mathbb{E}[Y\mid
A=0,X,Z=1]}{\delta P(A=1\mid X,Z=1)+P(A=0\mid X,Z=1)}$

###### Proof of Proposition 4.

Under (27), $P(A=1\mid X,V=1)=P(A=1\mid X,Z=1)$. Moreover,

$\displaystyle\mathbb{E}[Y^{a}\mid X,Z=1]$ $\displaystyle=\mathbb{E}[Y^{a}\mid
X,V=1]$ $\displaystyle=\mathbb{E}[Y^{a}\mid X,V=1,A=a]$
$\displaystyle=\mathbb{E}[Y^{a}\mid X,Z=1,A=a]$
$\displaystyle=\mathbb{E}[Y\mid X,Z=1,A=a]$

where the first equality follows by (27), the second by no unmeasured
confounding, the third by (27), and the final line by consistency. The result
follows by combining these results. ∎

######  Remark 6.

This estimand requires no positivity assumption with respect to $\pi_{a}(X)$
((24)) for identification ([17]).

######  Remark 7.

Unlike our other analyses, we additionally assume (27). This precludes the
missingness process or sample selection process from being a function of
treatment assignment given the covariates and vaccine-acceptance. This
assumption may not hold: for example, [8] shows that the CTIS overrepresents
vaccine-uptake compared to the U.S. population, suggesting that perhaps there
is selection into the sample based on this variable. On the other hand, [26]
also shows that the CTIS overrepresents individuals with higher education, a
variable also associated with higher vaccination rates. As long as higher
vaccine-uptake among CTIS respondents can by fully explained by other observed
covariates, then (27) could hold.

Alternatively, we could imagine these shifts as the result of an intervention
that only targeted respondents to the CTIS, so that the intervention is on
$P(A=1\mid X,Z=1)$ rather than $P(A=1\mid X,V=1)$. However, no feasible real-
world intervention on the treatment assignment process would likely correspond
to such an intervention, rendering such an estimand arguably less useful to
consider.

######  Proposition 5 (Pandemic bounds).

Assume the following:

######  Assumption 10.

$\displaystyle\mathbb{E}[Y^{A=1}\mid X=x,V=1]\geq\mathbb{E}[Y^{C=0,A=0}\mid
X=x,V=1]\qquad\forall x$ (28)

It follows that:

$\displaystyle\mathbb{E}[Y^{A=0}-Y^{A=1}\mid
X=x,V=1]\leq\mathbb{E}[Y^{C=1,A=0}-Y^{C=0,A=0}\mid X=x,V=1]$ (29)
$\displaystyle\frac{\mathbb{E}[Y^{A=0}\mid X=x,V=1]}{\mathbb{E}[Y^{A=1}\mid
X=x,V=1]}\leq\frac{\mathbb{E}[Y^{C=1,A=0}\mid
X=x,V=1]}{\mathbb{E}[Y^{C=0,A=0}\mid X=x,V=1]}$ (30)

###### Proof of Proposition 5.

We use the identity that for all $x$:

$\displaystyle\mathbb{E}[Y^{A=0}\mid V=1,X=x]$
$\displaystyle=\mathbb{E}[Y^{C=1,A=0}\mid V=1,X=x]$

This holds by definition: our estimates all take place in a world in which
there is a COVID-19 pandemic. The remainder of the proof follows directly from
(28). ∎

######  Remark 8.

Equation 28 is perhaps better motivated by the following assumed inequalities:

$\displaystyle\mathbb{E}[Y^{A=1}\mid V=1,X=x]$
$\displaystyle=\mathbb{E}[Y^{C=1,A=1}\mid V=1,X=x]$
$\displaystyle\geq\mathbb{E}[Y^{C=0,A=1}\mid V=1,X=x]$
$\displaystyle=\mathbb{E}[Y^{C=0,A=0}\mid V=1,X=x]$

The first equality holds by definition. The second inequality imagines a
counterfactual world where everyone were vaccinated for COVID-19 and there
were no COVID-19 pandemic. While perhaps a strange counterfactual, we believe
this is reasonable: all else equal, rates of depression would on average be
lower absent a COVID-19 pandemic, regardless of COVID-19 vaccination status.
The final equality states that these rates are equal those in a world with no
COVID-19 pandemic and no COVID-19 vaccinations. This seems reasonable if we
believe that the effect of COVID-19 vaccines on mental health arguably only
operate during a COVID-19 pandemic. Assumption 10 then follows.

######  Remark 9.

If we further assume random sample selection, then the same the identifying
expressions for $\mathbb{E}[Y^{a}\mid X,Z=1]$ detailed Proposition
LABEL:proposition:mpo allows us to identify the bounds conditional on our
sample. This then allows us to interpret our sample-average effect estimates,
for example, as lower absolute magnitude bounds on the effects of the COVID-19
pandemic on mental health among respondents in our sample.

######  Remark 10.

Using potential outcomes in this setting is perhaps philosophically
challenging since “the COVID-19 pandemic” is arguably not a well-defined
exposure, nor is it manipulable. Nevertheless, in general terms we can
consider: what would have happened if COVID-19 and any of the associated
responses had never happened? Equation 10 simply states that expected mental
health in this world would be at least as “good” as in a version of our world
where COVID-19 emerged but everyone were vaccinated by February 2021. This
assumption implicitly precludes the possibility that this counterfactual world
would have realized some other large negative shock to mental health.

##  11 Estimation

This section provides additional details on our estimation approach. As noted
previously, we use influence-function based estimators
throughout.242424Technically, these estimators are derived treating the
covariates as if representative from some super-population with covariate
distribution $dP(X\mid Z=1)$. These estimators provide consistent estimates of
the sample average treatment effects and possibly conservative variance
estimates ([15]). Specifically, all of our estimators take the form:

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\frac{\mathds{1}(R_{i}=1,V_{i}=1)}{P(R_{i}=1,V_{i}=1\mid
S_{i}=1)}\hat{\varphi}(O)$

where $\hat{\varphi}(O)$ represents a scaled estimate of the uncentered
influence function for any of the functionals considered in Section 10. To be
precise, we use estimates of $\varphi$ for the average potential outcome under
treatment $A=a$, the incremental effects, and interventional effects via
$M_{1}$, respectively:

$\displaystyle\varphi_{1,a}(O)=\frac{\mathds{1}(A=a)}{\pi_{a}(X)}\left(Y-\mu_{a}(X)\right)+\mu_{a}(X)$

$\displaystyle\varphi_{2}(O)=\frac{\delta\pi_{1}(X)\varphi_{1,1}(Z)+\pi_{0}(X)\varphi_{1,0}(Z)}{\delta\pi_{1}(X)+\pi_{0}(X)}+\frac{\delta\\{\mu_{1}(X)-\mu_{0}(X)\\}(A-\pi_{1}(X))}{\\{\delta\pi_{1}(X)+\pi_{0}(X)\\}^{2}}$

$\displaystyle\varphi_{3}(O)$
$\displaystyle=\frac{\mathds{1}(A=a)}{\pi_{a}(X)}\frac{\\{p(M_{1}\mid
a,X)-p(M_{1}\mid a^{\prime},X)\\}p(M_{2}\mid a^{\prime},X)}{p(M_{1},M_{2},\mid
a,X)}(Y-\mu_{a}(M_{1},M_{2},X))$
$\displaystyle+\frac{\mathds{1}(A=a)}{\pi_{a}(X)}\\{\mu_{a,M_{2}^{^{\prime}}}(M_{1},X)-\mu_{a,M_{1}\times
M_{2}^{^{\prime}}}(X)\\}$
$\displaystyle-\frac{\mathds{1}(A=a^{\prime})}{\pi_{a^{\prime}}(X)}\\{\mu_{a,M_{2}^{^{\prime}}}(M_{1},X)-\mu_{a,M_{1}^{^{\prime}}\times
M_{2}^{^{\prime}}}(X)\\}$
$\displaystyle+\frac{\mathds{1}(A=a^{\prime})}{\pi_{a^{\prime}}(X)}\left(\mu_{a,M_{1}}(M_{2},X)-\mu_{a,M_{1}\times
M_{2}^{\prime}}(X)-(\mu_{a,M_{1}^{^{\prime}}}(M_{2},X)-\mu_{a,M_{1}^{^{\prime}}\times
M_{2}^{^{\prime}}}(X))\right)$ $\displaystyle+\mu_{a,M_{1}\times
M_{2}^{^{\prime}}}(X)-\mu_{a,M_{1}^{^{\prime}}\times M_{2}^{^{\prime}}}(X)$

where

$\displaystyle\pi_{a}(X)=P(A=a\mid X,Z=1)$
$\displaystyle\mu_{a}(X)=\mathbb{E}[Y\mid X,A=a,Z=1]$
$\displaystyle\mu_{a}(M_{1},M_{2},X)=\mathbb{E}[Y\mid X,M_{1},M_{2},A=a,Z=1]$
$\displaystyle p(.\mid a,X)=p(.\mid a,X,Z=1)$ $\displaystyle\mu_{a,M_{1}\times
M_{2}^{\prime}}(X)=\sum_{m_{1},m_{2}}\mu_{a}(m_{1},m_{2},X)p(m_{1}\mid
a,X)p(m_{2}\mid a^{\prime},X)$
$\displaystyle\mu_{a,M_{1}^{^{\prime}}}(M_{2},X)=\sum_{m_{1}}\mu_{a}(m_{1},M_{2},X)p(m_{1}\mid
a^{\prime},X)$

with other terms defined analogously. The result for $\varphi_{1}(O)$ is well-
known (see, e.g. [17]). The derivation of $\varphi_{2}(O)$ is contained in
[17] and for $\varphi_{3}(O)$ in [4]. We refer to [4] for the expressions for
the one-step estimators for the remaining interventional effects.

To obtain estimates of the effects across all vaccine-accepting respondents
(e.g. averaged over the entire stratum where $SV=1$), we multiply the bias-
correction terms (specifically, the terms involving inverse-probability
weights) by $\frac{\mathds{1}(R=1)}{P(R=1\mid X,V=1,S=1)}$ and average the
resulting function over all observations where $V=1$ (and $S=1$) to obtain our
estimates. Our variance estimates are estimated by taking the empirical
variance of the estimated influence functions across our analytic sample.

We use logistic regression and XGBoost to estimate all nuisance functions. In
the former case we estimate each function on the full-sample, and in the
latter we use cross-fitting. Additionally, for the XGBoost estimates, we
estimate twenty specifications across a grid of hyper-parameters using the
“xgboost”, “tidymodels”, and “tune” packages, and stack the resulting models
using the “stacks” package in R.

##  12 Additional results

This section contains additional results not displayed in the main paper. We
first present model diagnostics, and then present additional tables and
figures with respect to our outcomes analysis, mediation analysis, and
incremental effects analysis. Results from the incremental effects analysis
are only contained in this Appendix.

###  12.1 Model diagnostics

We next examine the distribution of propensity score (“pi”) and complete-case
weights (“eta”), noting again that the complete-case weights are only used for
our analyses in Appendix 14. Table 8 displays the quantiles of the
distribution for each of these models.

Table 8: Propensity score models: weight distribution Parameter | Model | 0% | 1% | 10% | 25% | 50% | 75% | 90% | 99% | 100%
---|---|---|---|---|---|---|---|---|---|---
Eta | GLM | 0.0005 | 0.0032 | 0.7122 | 0.9081 | 0.9587 | 0.9800 | 0.9893 | 0.9954 | 0.9988
Pi | GLM | 0.0043 | 0.0296 | 0.0827 | 0.1499 | 0.3162 | 0.5625 | 0.7608 | 0.9368 | 0.9972
Eta | XGboost | 0.0002 | 0.0002 | 0.6983 | 0.9278 | 0.9582 | 0.9701 | 0.9749 | 0.9782 | 0.9785
Pi | XGboost | 0.0678 | 0.0775 | 0.0970 | 0.1288 | 0.2500 | 0.5924 | 0.8297 | 0.9114 | 0.9297

Figure 11 displays the calibration curves with respect to our nuisance
functions for our outcomes analysis. We display results for both our GLM and
XGBoost models. $R$ reflects the calibration of the complete case model; $A$
represents the vaccination model; $RA$ represents the product of vaccination
model with the complete case model (and $(1-A)R$ analogously); $Y$ represents
the depression; $M1$ the (binary) isolation model; and $M2$ represents the
(binary) worries about health model. The complete-case model – used only for
the full sample estimates displayed in Appendix 14 – appear to under-predict
the probabilities of being a complete-case at the lower end of the estimated
probabilities. On the other hand, the product of the complete-case model and
the propensity score models are well-calibrated with respect to the observed
data.

Figure 11: Model calibration

###  12.2 Outcomes analysis

Figure 12 displays our heterogeneity analysis for pre-specified subgroups for
our outcomes analysis using our XGBoost estimates, analogous to Figure 5 in
Section 5.

The following tables display the subgroup estimates for the pre-specified
subgroups with respect to our outcomes analysis. The first table presents
results on the risk-differences scale and the second on the risk-ratio scale.
The second two tables are identical but using XGBoost. In general we see that
the results are quite similar except for the non-response category, which is
much smaller for the XGBoost than the logistic regression models.

Figure 12: Outcome heterogeneity: complete-case estimates (XGBoost estimates) Table 9: Heterogeneous effects, GLM estimates, risk-differences (95% CI)  Subgroup | N | Depressed | Isolated | Worried about health | N Full
---|---|---|---|---|---
Age
18-34 years | 114811 | -5.89 (-6.78, -5.01) | -5.66 (-6.54, -4.78) | -5.32 (-6.22, -4.42) | 117012
35-64 years | 392806 | -3.75 (-4.11, -3.38) | -3.25 (-3.63, -2.87) | -4.36 (-4.80, -3.93) | 412978
65 or more years | 238072 | -2.53 (-2.84, -2.22) | -2.13 (-2.51, -1.75) | -3.40 (-3.86, -2.95) | 261036
No response | 12905 | -5.75 (-7.16, -4.34) | -3.24 (-4.67, -1.82) | -7.28 (-8.92, -5.64) | 90289
Ethnicity
Hispanic | 85915 | -2.81 (-3.73, -1.88) | -2.59 (-3.59, -1.58) | -5.67 (-6.87, -4.47) | 95550
Not Hispanic | 654822 | -3.78 (-4.05, -3.52) | -3.34 (-3.62, -3.06) | -4.03 (-4.34, -3.73) | 689480
No response | 17857 | -5.94 (-7.29, -4.58) | -3.57 (-5.15, -1.99) | -5.61 (-7.26, -3.95) | 96285
Gender
Female | 491608 | -3.99 (-4.32, -3.66) | -3.38 (-3.72, -3.04) | -4.50 (-4.88, -4.12) | 522122
Not female | 249729 | -3.14 (-3.54, -2.74) | -3.10 (-3.55, -2.66) | -3.56 (-4.05, -3.06) | 264406
No response | 17257 | -4.50 (-6.35, -2.65) | -2.21 (-4.06, -0.35) | -7.41 (-9.65, -5.17) | 94787
High-risk conditions
None | 263250 | -3.63 (-4.07, -3.20) | -3.56 (-4.02, -3.09) | -3.73 (-4.24, -3.21) | 297477
One | 219199 | -3.53 (-3.98, -3.08) | -3.29 (-3.77, -2.80) | -4.27 (-4.82, -3.73) | 247411
Two or more | 269353 | -4.01 (-4.43, -3.58) | -2.99 (-3.44, -2.54) | -4.79 (-5.29, -4.28) | 306998
Missing | 6792 | -2.13 (-5.93, 1.67) | -1.83 (-5.30, 1.64) | -3.09 (-6.62, 0.44) | 29429
Race
White | 613930 | -3.91 (-4.18, -3.64) | -3.43 (-3.72, -3.14) | -4.06 (-4.37, -3.75) | 645144
Black | 19683 | -1.61 (-3.15, -0.07) | -1.73 (-3.17, -0.28) | -2.74 (-5.05, -0.42) | 20929
Asian or Pacific Islander | 45241 | -2.26 (-3.35, -1.16) | -2.35 (-3.41, -1.29) | -5.70 (-7.11, -4.29) | 50535
Other | 57888 | -3.43 (-4.64, -2.21) | -2.52 (-3.76, -1.28) | -4.54 (-6.02, -3.06) | 63659
No response | 21852 | -4.24 (-5.71, -2.77) | -3.85 (-5.53, -2.17) | -7.25 (-9.26, -5.24) | 101048
Table 10: Heterogeneous effects, GLM estimates, risk-ratios (95% CI)  Subgroup | N | Depressed | Isolated | Worried about health | N Full
---|---|---|---|---|---
Age
18-34 years | 114811 | 0.85 (0.82, 0.87) | 0.83 (0.80, 0.85) | 0.85 (0.82, 0.87) | 117012
35-64 years | 392806 | 0.81 (0.79, 0.83) | 0.84 (0.82, 0.86) | 0.86 (0.84, 0.87) | 412978
65 or more years | 238072 | 0.73 (0.70, 0.75) | 0.85 (0.83, 0.88) | 0.84 (0.82, 0.86) | 261036
No response | 12905 | 0.70 (0.63, 0.77) | 0.82 (0.75, 0.90) | 0.77 (0.72, 0.82) | 90289
Ethnicity
Hispanic | 85915 | 0.86 (0.82, 0.91) | 0.88 (0.83, 0.92) | 0.88 (0.86, 0.91) | 95550
Not Hispanic | 654822 | 0.80 (0.79, 0.81) | 0.83 (0.82, 0.85) | 0.84 (0.83, 0.86) | 689480
No response | 17857 | 0.68 (0.62, 0.75) | 0.81 (0.73, 0.89) | 0.81 (0.76, 0.87) | 96285
Gender
Female | 491608 | 0.81 (0.80, 0.83) | 0.85 (0.83, 0.86) | 0.85 (0.84, 0.86) | 522122
Not female | 249729 | 0.79 (0.76, 0.81) | 0.82 (0.79, 0.84) | 0.85 (0.83, 0.87) | 264406
No response | 17257 | 0.81 (0.73, 0.88) | 0.90 (0.81, 0.98) | 0.77 (0.71, 0.84) | 94787
High-risk conditions
None | 263250 | 0.80 (0.78, 0.82) | 0.81 (0.79, 0.84) | 0.86 (0.84, 0.88) | 297477
One | 219199 | 0.81 (0.79, 0.83) | 0.83 (0.81, 0.86) | 0.85 (0.83, 0.86) | 247411
Two or more | 269353 | 0.81 (0.79, 0.82) | 0.86 (0.84, 0.88) | 0.85 (0.83, 0.86) | 306998
Missing | 6792 | 0.91 (0.75, 1.07) | 0.92 (0.77, 1.07) | 0.89 (0.77, 1.01) | 29429
Race
White | 613930 | 0.80 (0.78, 0.81) | 0.83 (0.82, 0.85) | 0.85 (0.83, 0.86) | 645144
Black | 19683 | 0.88 (0.78, 0.99) | 0.88 (0.77, 0.98) | 0.92 (0.86, 0.99) | 20929
Asian or Pacific Islander | 45241 | 0.84 (0.77, 0.92) | 0.83 (0.76, 0.90) | 0.82 (0.78, 0.86) | 50535
Other | 57888 | 0.86 (0.81, 0.91) | 0.89 (0.84, 0.95) | 0.89 (0.86, 0.93) | 63659
No response | 21852 | 0.77 (0.69, 0.85) | 0.80 (0.71, 0.88) | 0.81 (0.76, 0.86) | 101048
Table 11: Heterogeneous effects, XGBoost estimates, risk-differences (95% CI)  Subgroup | N | Depressed | Isolated | Worried about health | N Full
---|---|---|---|---|---
Age
18-34 years | 114811 | -6.11 (-6.82, -5.40) | -5.80 (-6.48, -5.11) | -4.91 (-5.61, -4.21) | 117012
35-64 years | 392806 | -3.68 (-3.99, -3.36) | -3.14 (-3.46, -2.81) | -4.28 (-4.64, -3.91) | 412978
65 or more years | 238072 | -2.65 (-2.90, -2.39) | -2.06 (-2.37, -1.74) | -3.47 (-3.83, -3.10) | 261036
No response | 12905 | -4.48 (-6.09, -2.86) | -3.02 (-4.65, -1.40) | -5.35 (-7.23, -3.48) | 90289
Ethnicity
Hispanic | 85915 | -2.52 (-3.19, -1.84) | -2.41 (-3.11, -1.72) | -4.60 (-5.46, -3.74) | 95550
Not Hispanic | 654822 | -3.86 (-4.09, -3.64) | -3.30 (-3.54, -3.06) | -4.06 (-4.32, -3.80) | 689480
No response | 17857 | -4.87 (-6.22, -3.53) | -3.29 (-4.68, -1.90) | -4.71 (-6.30, -3.12) | 96285
Gender
Female | 491608 | -3.97 (-4.25, -3.69) | -3.36 (-3.64, -3.07) | -4.42 (-4.74, -4.11) | 522122
Not female | 249729 | -3.24 (-3.57, -2.92) | -2.88 (-3.24, -2.52) | -3.46 (-3.87, -3.06) | 264406
No response | 17257 | -4.16 (-5.71, -2.61) | -3.16 (-4.71, -1.61) | -5.69 (-7.40, -3.98) | 94787
High-risk conditions
None | 263250 | -3.47 (-3.83, -3.12) | -3.45 (-3.82, -3.07) | -3.56 (-3.97, -3.15) | 297477
One | 219199 | -3.53 (-3.92, -3.14) | -3.21 (-3.62, -2.80) | -4.21 (-4.67, -3.75) | 247411
Two or more | 269353 | -4.13 (-4.50, -3.77) | -2.93 (-3.31, -2.55) | -4.70 (-5.13, -4.27) | 306998
Missing | 6792 | -4.54 (-6.96, -2.12) | -3.49 (-5.89, -1.09) | -1.86 (-4.48, 0.76) | 29429
Race
White | 613930 | -3.97 (-4.21, -3.74) | -3.28 (-3.53, -3.04) | -4.13 (-4.40, -3.86) | 645144
Black | 19683 | -1.72 (-2.97, -0.47) | -2.19 (-3.45, -0.92) | -2.45 (-4.13, -0.77) | 20929
Asian or Pacific Islander | 45241 | -2.49 (-3.32, -1.67) | -2.86 (-3.67, -2.05) | -5.70 (-6.78, -4.61) | 50535
Other | 57888 | -3.06 (-3.93, -2.20) | -2.82 (-3.71, -1.93) | -2.97 (-4.01, -1.93) | 63659
No response | 21852 | -3.17 (-4.40, -1.94) | -3.36 (-4.61, -2.11) | -5.72 (-7.23, -4.21) | 101048
Table 12: Heterogeneous effects, XGBoost estimates, risk-ratio (95% CI)  Subgroup | N | Depressed | Isolated | Worried about health | N Full
---|---|---|---|---|---
Age
18-34 years | 114811 | 0.84 (0.82, 0.86) | 0.82 (0.80, 0.84) | 0.86 (0.84, 0.88) | 117012
35-64 years | 392806 | 0.81 (0.80, 0.83) | 0.85 (0.83, 0.86) | 0.86 (0.85, 0.87) | 412978
65 or more years | 238072 | 0.72 (0.70, 0.74) | 0.86 (0.84, 0.88) | 0.84 (0.82, 0.85) | 261036
No response | 12905 | 0.76 (0.68, 0.84) | 0.83 (0.75, 0.92) | 0.82 (0.77, 0.88) | 90289
Ethnicity
Hispanic | 85915 | 0.88 (0.85, 0.91) | 0.89 (0.85, 0.92) | 0.90 (0.89, 0.92) | 95550
Not Hispanic | 654822 | 0.80 (0.79, 0.81) | 0.84 (0.83, 0.85) | 0.84 (0.83, 0.85) | 689480
No response | 17857 | 0.74 (0.67, 0.81) | 0.82 (0.75, 0.89) | 0.84 (0.79, 0.89) | 96285
Gender
Female | 491608 | 0.81 (0.80, 0.83) | 0.85 (0.83, 0.86) | 0.86 (0.85, 0.87) | 522122
Not female | 249729 | 0.78 (0.76, 0.80) | 0.83 (0.81, 0.85) | 0.86 (0.84, 0.87) | 264406
No response | 17257 | 0.82 (0.76, 0.88) | 0.86 (0.79, 0.92) | 0.82 (0.77, 0.87) | 94787
High-risk conditions
None | 263250 | 0.81 (0.79, 0.83) | 0.82 (0.80, 0.84) | 0.86 (0.85, 0.88) | 297477
One | 219199 | 0.81 (0.79, 0.83) | 0.84 (0.82, 0.86) | 0.85 (0.83, 0.86) | 247411
Two or more | 269353 | 0.80 (0.78, 0.82) | 0.87 (0.85, 0.88) | 0.85 (0.84, 0.86) | 306998
Missing | 6792 | 0.81 (0.71, 0.90) | 0.85 (0.75, 0.95) | 0.93 (0.84, 1.02) | 29429
Race
White | 613930 | 0.79 (0.78, 0.81) | 0.84 (0.83, 0.85) | 0.84 (0.83, 0.85) | 645144
Black | 19683 | 0.88 (0.79, 0.96) | 0.85 (0.76, 0.93) | 0.93 (0.88, 0.98) | 20929
Asian or Pacific Islander | 45241 | 0.83 (0.77, 0.88) | 0.79 (0.74, 0.85) | 0.82 (0.79, 0.85) | 50535
Other | 57888 | 0.87 (0.84, 0.91) | 0.88 (0.85, 0.92) | 0.93 (0.91, 0.95) | 63659
No response | 21852 | 0.83 (0.76, 0.89) | 0.82 (0.76, 0.89) | 0.85 (0.81, 0.89) | 101048

This next set of tables display the results of the subgroup estimates for the
data-driven groups for our outcomes analysis. The first set of tables displays
the results for the groups that we expected to be heterogeneous with respect
to depression, which we discuss in the primary paper. In addition to this
analysis, we also present the results for the subgroups that we examined for
isolation and worries about health, though these results were largely
inconclusive. All estimates are using XGBoost. We also estimates these numbers
using logistic regression models. Many of these patterns in heterogeneity no
longer appear as prominent, likely in part because we do not include any
interactions between covariates in any of these models. These results are
available on request.

Table 13: All outcomes: data-driven subgroup heterogeneity, risk-differences
(XGBoost estimates)

| Depression | Isolation | Health |
---|---|---|---|---
Variable | Value | Estimate (95% CI) | T-test | Estimate (95% CI) | T-test | Estimate (95% CI) | T-test | N
All data
- | 18-24 years | -12.21 (-13.68, -10.74) | Ref | -10.73 (-12.14, -9.31) | Ref | -6.93 (-8.28, -5.59) | Ref | 28073
- | 25+ years, Hispanic | -1.89 (-2.58, -1.21) | *** | -1.87 (-2.59, -1.16) | *** | -4.71 (-5.60, -3.81) | ** | 80254
- | 25+ years, not Hispanic | -3.56 (-3.79, -3.33) | *** | -3.03 (-3.27, -2.79) | *** | -3.93 (-4.19, -3.66) | *** | 632157
- | Other | -4.88 (-6.21, -3.56) | *** | -3.38 (-4.76, -2.01) | *** | -4.62 (-6.20, -3.04) | * | 18110
18-24 years
Female | Not female | -13.76 (-16.07, -11.44) | Ref | -11.36 (-13.69, -9.04) | Ref | -6.29 (-8.50, -4.08) | Ref | 7956
Female | Female | -11.30 (-13.16, -9.44) |  | -10.35 (-12.13, -8.58) |  | -7.28 (-8.96, -5.61) |  | 19571
Female | Unknown | -22.25 (-33.68, -10.82) |  | -14.83 (-25.64, -4.02) |  | -3.80 (-14.78, 7.17) |  | 546
Lives with elderly | No | -10.94 (-12.67, -9.21) | Ref | -10.09 (-11.78, -8.41) | Ref | -5.22 (-6.81, -3.63) | Ref | 20347
Lives with elderly | Missing | -12.55 (-15.95, -9.15) |  | -10.21 (-13.46, -6.95) |  | -10.85 (-13.91, -7.80) | ** | 5184
Lives with elderly | Yes | -21.62 (-26.45, -16.79) | *** | -16.85 (-21.31, -12.40) | ** | -12.65 (-17.07, -8.24) | ** | 2542
White | White | -12.10 (-13.80, -10.39) | Ref | -11.55 (-13.18, -9.92) | Ref | -7.90 (-9.43, -6.37) | Ref | 21515
White | Not white | -12.24 (-15.28, -9.20) |  | -7.72 (-10.75, -4.69) | * | -3.70 (-6.66, -0.73) | * | 6019
White | Unknown | -16.22 (-24.48, -7.95) |  | -11.55 (-20.10, -3.00) |  | -4.68 (-12.59, 3.22) |  | 539
25+ years, Hispanic
Age | 25-44 years | -1.45 (-2.64, -0.26) | Ref | -1.38 (-2.56, -0.21) | Ref | -4.48 (-5.85, -3.10) | Ref | 33726
Age | 45 years or older | -2.22 (-3.03, -1.40) |  | -2.23 (-3.12, -1.34) |  | -4.87 (-6.04, -3.70) |  | 46528
Language | English | -2.52 (-3.39, -1.65) | Ref | -2.80 (-3.63, -1.97) | Ref | -4.96 (-6.02, -3.90) | Ref | 57236
Language | Spanish | -0.33 (-1.36, 0.69) | ** | 0.42 (-0.96, 1.81) | *** | -4.07 (-5.72, -2.43) |  | 23018

  * •

T-tests reflect test of difference in means between the category specified in
row and the reference category (Ref) within each Subset and Variable

  * •

*** indicates p ¡ 0.001; ** indicates p ¡ 0.01; * indicates p ¡ 0.05

  * •

Subgroups chosen using data from March 1-14th using specified outcome

Table 14: All outcomes: data-driven subgroup heterogeneity, risk-ratios
(XGBoost estimates)

| Depression | Isolation | Health |
---|---|---|---|---
Variable | Value | Estimate (95% CI) | T-test | Estimate (95% CI) | T-test | Estimate (95% CI) | T-test | N
All data
- | 18-24 years | 0.76 (0.73, 0.78) | Ref | 0.72 (0.69, 0.76) | Ref | 0.81 (0.77, 0.84) | Ref | 28073
- | 25+ years, Hispanic | 0.90 (0.86, 0.94) | *** | 0.91 (0.87, 0.94) | *** | 0.90 (0.88, 0.92) | *** | 80254
- | 25+ years, not Hispanic | 0.80 (0.79, 0.81) | ** | 0.85 (0.83, 0.86) | *** | 0.85 (0.84, 0.86) | * | 632157
- | Other | 0.73 (0.67, 0.80) |  | 0.82 (0.75, 0.89) | * | 0.84 (0.79, 0.90) |  | 18110
18-24 years
Female | Not female | 0.68 (0.63, 0.73) | Ref | 0.69 (0.63, 0.75) | Ref | 0.79 (0.72, 0.86) | Ref | 7956
Female | Female | 0.79 (0.75, 0.82) | *** | 0.74 (0.70, 0.78) |  | 0.81 (0.76, 0.85) |  | 19571
Female | Unknown | 0.66 (0.50, 0.82) |  | 0.70 (0.50, 0.90) |  | 0.91 (0.64, 1.17) |  | 546
Lives with elderly | No | 0.78 (0.75, 0.81) | Ref | 0.74 (0.70, 0.78) | Ref | 0.84 (0.80, 0.89) | Ref | 20347
Lives with elderly | Missing | 0.74 (0.68, 0.81) |  | 0.73 (0.65, 0.81) |  | 0.72 (0.65, 0.80) | ** | 5184
Lives with elderly | Yes | 0.60 (0.53, 0.68) | *** | 0.61 (0.52, 0.70) | * | 0.73 (0.65, 0.82) | * | 2542
White | White | 0.77 (0.74, 0.80) | Ref | 0.71 (0.67, 0.75) | Ref | 0.77 (0.73, 0.81) | Ref | 21515
White | Not white | 0.72 (0.66, 0.79) |  | 0.79 (0.71, 0.87) |  | 0.91 (0.83, 0.98) | ** | 6019
White | Unknown | 0.63 (0.48, 0.79) |  | 0.67 (0.46, 0.89) |  | 0.90 (0.75, 1.06) |  | 539
25+ years, Hispanic
Age | 25-44 years | 0.94 (0.89, 0.99) | Ref | 0.94 (0.89, 0.99) | Ref | 0.91 (0.88, 0.94) | Ref | 33726
Age | 45 years or older | 0.85 (0.80, 0.90) | * | 0.88 (0.83, 0.92) |  | 0.90 (0.87, 0.92) |  | 46528
Language | English | 0.88 (0.85, 0.92) | Ref | 0.86 (0.81, 0.90) | Ref | 0.89 (0.86, 0.91) | Ref | 57236
Language | Spanish | 0.97 (0.89, 1.06) |  | 1.02 (0.96, 1.08) | *** | 0.93 (0.90, 0.96) | * | 23018

  * •

T-tests reflect test of difference in means between the category specified in
row and the reference category (Ref) within each Subset and Variable

  * •

*** indicates p ¡ 0.001; ** indicates p ¡ 0.01; * indicates p ¡ 0.05

  * •

Subgroups chosen using data from March 1-14th using specified outcome

Table 15: Worries about health: data-driven subgroup heterogeneity (XGBoost
estimates)

| Risk-difference | Risk-ratio |
---|---|---|---
Strata | Estimate (95% CI) | T-test | Estimate (95% CI) | T-test | N
Female