# Total Error and Variability Measures for the Quarterly Workforce Indicators
and LEHD Origin-Destination Employment Statistics in OnTheMap

Kevin L. McKinney  Andrew S. Green  Lars Vilhuber  John M. Abowd Kevin
McKinney<EMAIL_ADDRESS>is Senior Economist, U.S. Census
Bureau; Andrew Green<EMAIL_ADDRESS>is Economist, OECD; Lars Vilhuber
<EMAIL_ADDRESS>is Senior Research Associate and Executive
Director of the Labor Dynamics Institute at Cornell University and Economist,
U.S. Census Bureau; John Abowd<EMAIL_ADDRESS>is Chief
Scientist and Associate Director for Research and Methodology, U.S. Census
Bureau, Edmund Ezra Day Professor of Economics, Statistics and Information
Science, Cornell University.

(July 21, 2020)

###### Abstract

We report results from the first comprehensive total quality evaluation of
five major indicators in the U.S. Census Bureau’s Longitudinal Employer-
Household Dynamics (LEHD) Program Quarterly Workforce Indicators (QWI): total
flow-employment, beginning-of-quarter employment, full-quarter employment,
average monthly earnings of full-quarter employees, and total quarterly
payroll. Beginning-of-quarter employment is also the main tabulation variable
in the LEHD Origin-Destination Employment Statistics (LODES) workplace reports
as displayed in OnTheMap (OTM), including OnTheMap for Emergency Management.
We account for errors due to coverage; record-level non-response; edit and
imputation of item missing data; and statistical disclosure limitation. The
analysis reveals that the five publication variables under study are estimated
very accurately for tabulations involving at least 10 jobs. Tabulations
involving three to nine jobs are a transition zone, where cells may be fit for
use with caution. Tabulations involving one or two jobs, which are generally
suppressed on fitness-for-use criteria in the QWI and synthesized in LODES,
have substantial total variability but can still be used to estimate
statistics for untabulated aggregates as long as the job count in the
aggregate is more than 10.

Keywords: Multiple imputation; Total quality measures; Employment statistics;
Earnings statistics; Total survey error; Input noise infusion SDL.

> Acknowledgements
>
> This paper is based in part on the technical report [17] and the appendices
> therein. The work was supported by the U.S. Census Bureau and by the
> National Science Foundation [Grants SES-0922005, BCS 0941226, TC-1012593,
> and SES-1131848 to Abowd and Vilhuber]. This research uses data from the
> U.S. Census Bureau’s Longitudinal Employer-Household Dynamics Program, which
> was partially supported by the National Science Foundation [Grants
> SES-9978093, SES-0339191 and ITR-0427889]; by the National Institute on
> Aging [Grant AG018854]; and by grants from the Alfred P. Sloan Foundation.
> Any opinions and conclusions expressed herein are those of the authors and
> do not represent the views of the U.S. Census Bureau. All results have been
> reviewed and released by the Disclosure Review Board with approval numbers
> CBDRB-FY19-CED002-B0017 and CBDRB-FY20-CED002-B0001. Online supplemental
> materials are available at [18].

##  1 Introduction

We undertake the first comprehensive analysis of the total error and
variability for two Longitudinal Employer-Household Dynamics (LEHD) products
from the U.S. Census Bureau: the Quarterly Workforce Indicators (QWI), which
are public-use tables displayed in QWI Explorer, and the workplace-based LEHD
Origin-Destination Employment Statistics (LODES), which are the public-use
tables displayed in OnTheMap (OTM) and OnTheMap for Emergency Mangement. The
Census Bureau produces these labor market indicators from a comprehensive
integrated administrative record system known as the LEHD Infrastructure File
System, which is based primarily on the linkage between employers and
employees provided by state-regulated Unemployment Insurance (UI) earnings
records. The theoretical universe to which these earnings records correspond
is all statutory jobs in the economy—private and public (excluding federal
employees).111At the time this evaluation was first undertaken, federal
employees were not covered in QWI and LODES, although they are now.

In principle, the published indicators are subject to errors from coverage,
record-level non-response, edit, imputation of item-level missing data, and
statistical disclosure limitation (SDL). The SDL error is due to employer-
level noise infused before tabulation. By addressing these sources of error in
our assessment of total variability, we have created comprehensive total
quality measures for these data.222[6] also identifies sampling,
specification, measurement, and data processing errors. The QWI data are not
sample-survey based; so, there is no sampling error in the traditional sense.
We do, however, model record-level missing data using concepts from sampling
theory.

The five indicators we study are published every quarter in the QWI,
stratified by ownership, sub-state geography, detailed industry, worker age,
gender, race, ethnicity, and education. The publication tables cross-classify
many of these same stratifiers. One of the five indicators, beginning-of-
quarter employment, is also the primary tabulation variable in LODES for
display in OnTheMap, which is released annually (reference date April
$1^{st}$) using many of the same stratifiers as in the QWI, and tabulated at
geographies as detailed as the census block. Overall, our comprehensive
measures of the total variability of QWI and LODES tabulations for these five
critical variables provide substantial evidence that the system is producing
reliable data.333By the standards in [24], section 13.7, at all levels of
stratification, QWI and LODES data are fit for use when the cells have at
least 3-9 jobs, with the exact cutoff dependant on the set of tabulation
characteristics. Cells with less than three jobs are not released in QWI,
where the full set of aggregations is always available. However, they are
released in LODES, with additional SDL, to permit construction of arbitrary
geographic aggregates that usually contain three or more jobs.

This study contributes to the growing body of literature on total survey
error. [6] defines total survey error as the “accumulation of all errors that
may arise in the design, collection, processing, and analysis of survey data.”
The total error estimates undertaken in this study address errors due to
coverage, record-level non-response, edit, imputation, and SDL.

This study also contributes to a recent, if more mature, literature that uses
administrative data to evaluate existing surveys, as well as an emerging
literature that assesses the total quality of administrative data themselves.
See [19], [20], and [9] for recent examples. Our final assessment adheres
closely to the best practices enumerated across many statistical agencies when
applied to current data products. See [12], [15], and [24] for examples of
total quality frameworks applied to other statistical products.

The remainder of this paper is organized as follows. Section 2 provides
important background on the LEHD data, including variable sources,
definitions, and characteristics related to employers and workers. Section 3
formally defines the universes, frames, estimands, and estimators that we
study for each of the five QWI statistics. Section 4 demonstrates that the
total error bias is zero. Section 5 provides formal models for estimating
total variability and its associated components in a manner that fully
incorporates the uncertainty due to the SDL procedures. Section 6 discusses
the detailed results and provides guidance for computing confidence intervals.
Section 7 concludes.

##  2 Important Features of the LEHD Data, QWI and LODES

The QWI and LODES are based on the LEHD Infrastructure File System. The
original production version of this system is documented in [3]. Enhancements
to the processing of characteristics are further documented in [17]. We focus
on five of the QWI indicators:

  * •

Total flow-employment, $M$, defined as the sum of all jobs with positive
earnings at any time in the quarter.

  * •

Beginning-of-quarter employment, $B$, defined as the sum of all jobs with
positive earnings in the current quarter as well as the previous quarter.

  * •

Full-quarter employment, $F$, defined as the sum of all jobs with positive
earnings in the current quarter in addition to the previous and subsequent
quarters.

  * •

Average monthly earnings of full-quarter jobs, $Z\\_{W3}$.

  * •

Total payroll, $W1$, defined as the total earnings at all active jobs ($M$) in
a quarter.

In LODES, the primary tabulation variable is $B$ using QWI definitions.444In
order to be consistent with the notation in most of the technical
documentation of the LEHD data, QWI, and LODES, we use the compact notation as
in [3]. A crosswalk between the notation used here and that found on the
public website can be found at [25].

UI earnings records are used to construct a job-based frame for the QWI and
LODES. An in-scope job occurs when a worker produces at least one dollar of
UI-covered earnings at a non-federal employer in a given quarter. The LEHD
Infrastructure File System combines this information with additional survey
and administrative data to associate individual characteristics (or features)
such as birth date, gender, place of birth, race, ethnicity, and education, as
well as workplace characteristics (or features), such as workplace address and
North American Industrial Classification System (NAICS) codes to all jobs in
the frame. The LEHD Infrastructure File System was developed using model-based
edit and imputation procedures. Every missing data element has been multiply-
imputed using an integrated set of models described in [3]. There are 10
implicates for every missing item, subscripted $l=1,\dots,L=10$. The missing
data models for most of the variables used in this paper, including birth
date, gender, race, ethnicity, education, workplace geography, workplace
NAICS, firm age, and firm size, have been substantially improved and modified
since the 2009 article was written. Because the LEHD Infrastructure File
System is rebuilt every quarter from all historical records, the analysis in
this paper incorporates all of those model improvements.

The five worker characteristics used in this paper are birth date, gender,
race, ethnicity, and education, each of which is part of an integrated
multiple-imputation model. We also evaluate two workplace characteristics;
NAICS sector and county. Both of the workplace characteristics have extremely
low missing data rates and are not multiply imputed, however, jobs at multi-
unit employers must be assigned to one of the employer’s establishments. The
assignment of a job to an establishment is multiply imputed and will
potentially introduce variation across implicates to the extent that a
workplace characteristic varies across workplaces (also called establishments)
within the same employing firm (UI account). Both processes are documented in
[17]. The statistical disclosure limitation applied to QWI and the workplace
component of LODES uses employer-level input noise infusion [3, 4].

All statistics in this paper are estimated for a given state-year-quarter. All
estimation uses the original micro-data from the production system, which
incorporates the missing data imputation models and provides values of all
micro-data elements with and without SDL applied. We re-estimate the actual
QWI statistics, then estimate total variability and its components. The
analysis covers the period 1995 through 2016 for the states AK, DC, DE, HI,
KY, ND, NH, RI, SD, VT, WV, and WY.555For simplicity, we include Washington,
D.C. when we say “state.” We restricted analysis to a subset of states to
address the comments from referees and the editor to incorporate the error due
to SDL. Using a subset of states reduces the computational burden. The longer
technical paper [17] uses data from all available states in the QWI release
labeled R2012Q4, which covers 1990:1 through 2012:1, but does not contain an
estimate of the contribution of the noise infusion to the total variability,
and does not use the formulas developed for this paper. The schema for the QWI
as of the R2012Q4 release are described at
https://lehd.ces.census.gov/data/schema/v3.5/. The schema is regularly
updated, and can be found at https://lehd.ces.census.gov/data/schema/.
Availability for each state varies both historically and at any point in time,
see https://lehd.ces.census.gov/doc/QWI_data_notices.pdf for available data
for each state.

Every job in the universe must have completed data for all the publication
variables. The LEHD Infrastructure File System has a fully-integrated
collection of probability models that generate multiply-imputed values for all
missing data items in the system. Most details are supplied in [3]—in
particular, the models for imputing missing demographic and workplace
characteristics.666[3] does not document the replacement to the demographic
variable imputation methods that were incorporated in 2010. Those methods are
documented in [17]. The system uses the methods first proposed by [21, 22] and
expanded in [16] for analyses using multiply-imputed missing data.

For each record in the component files of the LEHD data, variables used in the
computation of any QWI or LODES statistic are examined by an integrated set of
edit and imputation models. The system uses $L=10$ threads for these models.
Each thread generates its own posterior predictive samples, which are used to
apply the edits and imputations to each record. Calculations are done for all
time periods $t$, but for notational convenience, and without loss of
generality, we drop the $t$ subscript for most of our discussion below.

The data tabulation system infuses permanent multiplicative noise into the
employer-level data used to produce tabular output for QWI and LODES [3,
4].777A similar method is now used in many other Census Bureau economic data
publications [7]. The multiplicative noise factors $\delta_{j}$ for each
establishment $j$ are drawn from a two-sided symmetric ramp distribution
centered at unity. The draws from the distribution distort the original input
by at least a minimum percentage, and by no more than a maximum percentage.
Both of these values are Census confidential. This system is a substantial
generalization of the method originally developed by [13]. The SDL system
provides protection to both an employer and all of its workplaces—all
establishments for the same employer within a given state have input noise-
infusion factors on the same side of unity. In addition, the release
statistics are dynamically consistent—the same noise factor is used for an
employer or workplace in every quarter of data.888Most states do not code the
workplace or establishment on their UI earnings records. Consequently, it is
the employer-level noise-infusion factors that are incorporated into the
estimators in this paper. The establishment-level noise factors are only
salient when the workplace identifier does not have to be imputed on the UI
earnings record. In that case an establishment and a firm are equivalent, and
we call them both employers.

Multiplicative input noise infusion provides confidentiality protection in the
following sense. The originally reported values of the tabulation variables
are never used in the formation of the magnitudes (employer-level counts and
dollar sums) and ratios that are tabulated. The input noise infusion insures
that for every micro-data record tabulated, there is a strictly non-zero
percentage difference between the value used in tabulation and the true
confidential value. Tabulations based upon a small number of establishments
(at the limit one) or a small number of employees (at the limit one) contain
uncertainty induced by the distribution of the noise factor. This uncertainty
limits a user’s ability to infer attributes to within a range that is
confidential. Finally, the physical location of a workplace is not treated as
confidential because it is defined as the location where an employee must
report for work, and is therefore public. While the protection system is not
formally private in the sense of [11], it does satisfy the necessary
conditions in [10] for resistance to database reconstruction attacks. See [14]
for a formal privacy analysis of this protection mechanism.

##  3 Definition of Estimands and Estimators

In this section we define the statistical framework for the universes, frames,
estimands, and estimators used in the QWI and LODES data analyzed in this
paper. Because our focus is the estimation of total error in the spirit of
[6], the estimands defined here match the production system at the Census
Bureau, but the estimators have been slightly simplified to permit estimation
of their variance components.

###  3.1 Complete Data Estimands

The theoretical universe for QWI/LODES is all statutory jobs. While this
universe is conceptually easy to understand, implementing it in a frame via
administrative records only—that is, without independent field work to verify
the existence of businesses—is very tricky, as we will discuss in section 3.2.
Because the universe includes statutory employers whether or not they have
positive UI-covered jobs (_i.e._ , statutory employment) in a given quarter,
it is possible that a particular sub-universe will have no statutory
employment in a particular quarter, but nonetheless be at risk for positive
employment because statutory employers exist in that sub-universe. This is a
sampling zero. In contrast, some sub-universes will never have any statutory
jobs because there are no statutory employers in that sub-universe, and we
denote these as structural zeros because the probability of observing positive
employment in these sub-universes is zero.

To make all of the sources of uncertainty relative to the complete data
estimands clear, we define each component of the micro-data records
completely.999Although these definitions can be found in [3], repeating them
here helps properly distinguish the consequences of record and item missing
data. For individual $i$ employed by business $j$, a UI-covered statutory job
exists in quarter $t$, if $i$ has a UI-earnings at $j$, $w1_{i,j,t}$, with at
least $\$1$ of earnings in quarter $t$; that is, $w1_{i,j,t}\geq 1$. In this
case we say $m_{i,j,t}=1$, otherwise $m_{i,j,t}=0$. In quarter $t$, there is a
vector of characteristics associated with $i$ and $j$, which we call
$x_{i,j,t}$. These characteristics include worker features (gender, age, race,
ethnicity, and education) and employer features (workplace location, workplace
industry, employer size, and employer age). We define the sample space of
$x_{i,j,t}$ as $\chi$.

Partitions of $\chi$ are denoted by the set $\\{\Omega_{k}\\}$ having the
properties

$\Omega_{k}\cap\Omega_{k^{\prime}}=\varnothing,k\neq k^{\prime}$ (1)

and

$\cup_{\forall k}\Omega_{k}=\chi.$ (2)

Hence, a partition of $\chi$ is a mutually exclusive, exhaustive
stratification of the characteristics in the population. Although QWI and OTM
use multiple partitions, the properties of all estimators can be understood by
examining those defined on an arbitrary partition. When it is clear from the
context, we will call one element of this partition “cell $k$,” which means
all statutory jobs in the universe that belong in partition $\Omega_{k}$.

The finite-population estimands of $M_{k}$, $B_{k}$, $F_{k}$, $Z\\_W3_{k}$,
and $W1_{k}$ are defined here for the partition $\\{\Omega_{k}\\}$. For the
remainder of the paper, the subscript $t$ is suppressed unless it is needed to
define a primitive for quarter $t$.

The estimand for $M_{k}$ is

$M_{k}=\sum_{j=1}^{J}{\sum_{i=1}^{M_{j}}{I[x_{i,j}\in\Omega_{k}]m_{i,j}}},$
(3)

where $M_{j}=\sum_{i}m_{i,j}$, $I[A]$ is the indicator function taking the
value 1 when $A$ is true (0, otherwise), and $J$ is the number of employers in
the universe.

To define the estimand for $B_{k}$, we define $b_{i,j,t}=1$ when $m_{i,j,t}=1$
and $m_{i,j,t-1}=1$, otherwise $b_{i,j,t}=0$; then,

$B_{k}=\sum_{j=1}^{J}{\sum_{i=1}^{M_{j}}{I[x_{i,j}\in\Omega_{k}]b_{i,j}}}.$
(4)

To define he estimand for $F_{k}$, we define $f_{i,j,t}=1$ when
$m_{i,j,t+1}=1$, $m_{i,j,t}=1$ and $m_{i,j,t-1}=1$, otherwise $f_{i,j,t}=0$;
then,

$F_{k}=\sum_{j=1}^{J}{\sum_{i=1}^{M_{j}}{I[x_{i,j}\in\Omega_{k}]f_{i,j}}}.$
(5)

To define the estimand for $Z\\_W3_{k}$, we use $w1_{i,j,t}$ divided by 3, to
define average monthly earnings for a statutory job in quarter $t$. Then,

$Z\\_{W3}_{k}=\frac{1}{F_{k}}\sum_{j=1}^{J}{\sum_{i=1}^{M_{j}}{I[x_{i,j}\in\Omega_{k}]f_{i,j}\frac{w1_{i,j}}{3}}}.$
(6)

Note that the multiplication by $f_{i,j}$ inside the summation selects the
correct employees—those employed for the full quarter.

Finally, the estimand for $W1_{k}$ is

$W1_{k}=\sum_{j=1}^{J}{\sum_{i=1}^{M_{j}}{I[x_{i,j}\in\Omega_{k}]w1_{i,j}}}.$
(7)

Note that multiplication by $m_{i,j}$ is not required, since $m_{i,j,t}=1$ if,
and only if, $w1_{i,j,t}\geq 1$. There are no rows in the universe where
$m_{i,j,t}=0$, by construction.

###  3.2 Estimators for Each Estimand

If the LEHD data contained a record for every statutory job in the universe
with no item missing data, then the finite-population estimators for $M_{k}$,
$B_{k}$, $F$, $Z\\_W3_{k}$, and $W1_{k}$, in the absence of SDL, would be
identical to the estimands in equations (3)-(7). Because the frame must be
constructed dynamically, and because there are missing records and missing
items in the file system, and, finally, because the released data are subject
to SDL, we carefully construct the estimator appropriate for each estimand in
this section.

We cannot tell by observing a single quarter of data whether the absence of
employers in cell $k$ means that there were no statutory jobs in that cell for
that quarter (sampling zero) or no possibility of jobs in that cell because
there were never any employers (structural zero). Therefore, to develop a
frame for estimation, we must adopt a definition of the effective universe
that spans a broad time period.101010[1] show how the use of long sequences of
mandatory tax information returns, collected in a distinct administrative
operation from the data used for the employment statistics, can substitute for
independent field work in modeling the birth and death of employers in a
dynamic business frame problem similar to the one studied in this paper. We
define the frame to include an employer if there is any evidence of statutory
jobs anywhere in $\chi$ for that employer between the earliest quarter in the
database and the most recent quarter in the database–1990 to 2016, in this
paper. Note that the tabulation quarters in this paper–1995-2016–are a
temporally contiguous subset of the universe used to construct the frame. This
is also how the employer frame is defined in the production QWI and OTM.

The LEHD program receives two employer-level reports of employment and
earnings every quarter. The first is the Quarterly Census of Employment and
Wages (QCEW). The second is the state Unemployment Insurance (UI) earnings
data. The QWI/LODES employment and earnings measures are constructed from the
state UI earnings records, but the QCEW data also provide an employment
measure that we combine with the UI records to construct a proper frame. Most
statutory jobs are defined by state law. At the state level, any employer with
positive statutory employment in the QCEW or positive statutory employment in
the UI earnings data, as long as the UI employer appears at least once in the
QCEW in the frame window, contributes to the population of jobs.111111We
permanently exclude all UI accounts that appear in the UI earnings data but
never appear in the QCEW because they are very likely to be duplicates due to
identifier mismatch [5]. Using this definition of the frame, structural zeros
occur in cells $k$ that never have any employers in the period 1990 to 2016.
Note that structural zeros limit the sample space, $\chi$, of $x_{i,j,t}$ but
are only possible for employer features, not for individual features. For
example, a particular county can have a structural zero for a particular NAICS
sector because there are no employers in the frame who ever had jobs in that
county and NAICS sector. When we report statistics for partitions of the
sample space, partitions that are structural zeros are excluded because they
have neither employers nor jobs with probability one ex ante. Therefore, there
is no error associated with the structural zeros. This is exactly the same as
the treatment of geographic areas or NAICS sectors where there are no employer
firms in other economic data.

Record-level incomplete data occur because either the QCEW quarterly
employment and earnings summaries for the employer or _all_ the job-level UI
earnings records for an employer are missing due to failure to file or late
filing. The QCEW summary and UI individual earnings data are both collected as
part of the administration of the state UI system. They share employer-level
identifiers called SEINs in the LEHD data and UI account numbers in the state
systems. The statistical problem in determining total frame employment in
quarter $t$ is reconciling QCEW employment definitions (measured for the pay
period that includes the $12^{th}$ calendar day of the month for the first,
second and third months of each quarter) and the available employment
definitions in the QWI (first day of the quarter for $B$ and $F$ or any day in
the quarter for $M$). The QCEW data are edited following procedures laid out
by the BLS, and employment counts generated via the QCEW for month-one of the
quarter are most comparable to the $B$ measure in the QWI. There is no
employment measure in the QCEW data comparable to $F$ or $M$ in the QWI data.
QCEW total quarterly earnings are exactly comparable to $W1$ in the QWI.

To account for records missing from the LEHD system, we form a composite total
frame employment measure based on a precedence ordering. We use QCEW month-one
employment, if available. If not, we use QCEW month-two or month-three, in
that order, if available. If no QCEW data are available, we use UI-defined $B$
employment (applying equation (4) directly to the raw micro-data), if
available. If not, we look forward one quarter for $B$, if available. If not,
we use $M$ (again applying equation (3) directly to the raw micro-data), which
is always available if a UI record exists in quarter $t$. Thus, for the
purpose of determining the total frame employment, the UI data substitute for
QCEW data when QCEW data are not available in quarter $t$, as long as a QCEW
record exists for that employer in at least one other quarter in the frame
window (1990-2016). If no QCEW data ever exist in the frame window, the UI
earnings records for that employer are discarded from the frame.

For each quarter, the total employment calculated from this composite measure
is the finite job population, which we call $N_{B}$ because it is based on the
$B$ definition of employment. The employers with observed UI earnings data in
that quarter are the “sampled jobs” from this finite population, $N_{UB}$.
Call this set of employers $S_{t}$. $N_{UB}=\sum_{j}\sum_{i}b_{i,j}$ for $j\in
S_{t}$. The weight is $w=\frac{N_{B}}{N_{UB}}$ and the observed fraction of
jobs is $f=\frac{1}{w}$.121212We apologize for the abuse of notation in
defining the weight, $w$, and the sampled fraction, $f$, using symbols that
also appear with subscripts in the text. Whenever $w$ and $f$ are used without
subscripts, they always mean the weight and the sampled fraction.

As can be seen from the definitions, the weight is the ratio of total
composite $B$-based employment for all employers to total composite
$B$-employment for employers that appear in both the QCEW and UI earnings
data. Using a consistent set of employment reports in both the numerator and
the denominator ensures that an employer always receives a positive weight not
less than 1. The median employer-level weight is 1.007. The $75^{th}$
percentile employer-level weight is 1.020. The $95^{th}$ percentile employer-
level weight is 1.092.

The weight that we construct from the composite $B$-based employment measure
is calculated separately each quarter for private and public-sector employers.
The variation in weights between employers is due entirely to the private-
public classification. Public-sector weights are typically larger than
private-sector weights with a median of 1.182. Both Hawaii and DC have
unusually large public-sector weights (median of 6.409, max of 7.402). The
$95^{th}$ percentile for the public-sector weight is 1.278, while the
$95^{th}$ percentile for the private-sector weight is 1.039.

There is a single weight for all private employers and a different single
weight for all public-sector employers for each state and quarter. There is no
between-employer variance in the weight within the private or public sectors.
The weight, thus, assumes that employers, and thus UI earnings records, not
found in the LEHD file system are missing completely at random, given the
public/private sector of the employer. We recognize that this is a strong
assumption. Given the very small fraction of implied missing UI earnings
records, on average 1.67%, and the importance of having implementable formulas
for certain components of total variability as derived in Section 5, we think
it is a good working approximation.

The public/private status of an employer is permanent (non-time-varying) and
never missing. Therefore, we can unambiguously permanently stratify the frame
into public and private employers. The statistical analysis that follows is
identical for these two strata except that the numerical value of the total
job population is different in each stratum. For that reason, we do the
theoretical analysis assuming the frame has a single stratum. This keeps the
formulas simpler. Aggregation of the stratum-level statistics is via addition
for $M$, $B$, $F$, and $W1$ and weighted averages using the proportion of the
total job population in each stratum as weights for $Z\\_W3$.

The LEHD micro-data also contain the permanent noise-infusion factor,
$\delta_{j}$, for each establishment $j$ and the $L$ implicates from the
multiple imputation of missing characteristics. Given $w$, $\delta_{j}$ and
the implicates $l$, we can define the QWI/LODES estimators $\hat{M}_{k}$,
$\hat{B}_{k}$, $\hat{F}_{k}$, $\hat{Z}\\_W3_{k}$, and $\hat{W}1_{k}$ as
follows.

$\hat{M}_{k}=w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}I[x_{i,j}^{(l)}\in\Omega_{k}]m_{i,j},$
(8)

where $x_{i,j}^{(l)}$ is the $l^{th}$ implicate of $x_{i,j}$ from the multiple
imputation system. Note that $m_{i,j}$ cannot be missing if employer $j$ is in
the sampled set $S$ for the quarter being estimated. The upper limit of the
summation over $i$ is therefore unaffected by item missing data, and the
entire $i$ summation is multiplied by zero when UI records for employer $j$
are missing.

$\hat{B}_{k}=w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}I[x_{i,j}^{(l)}\in\Omega_{k}]b_{i,j}.$
(9)

Notice that there is no multiple imputation of the variable $b_{i,j}$, just as
there was not for $m_{i,j}$, because it is always possible to define $b_{i,j}$
without reference to any variable except $m_{i,j}$, which is never missing
when employer $j$ is in the observed sample. The variation due to item missing
data occurs only because the features in $x_{i,j}$, which may be missing,
determine whether the job record belongs in cell $k$.

$\hat{F}_{k}=w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}I[x_{i,j}^{(l)}\in\Omega_{k}]f_{i,j}.$
(10)

Notice, again, that $f_{i,j}$ is never missing for the same reasons as
$b_{i,j}$ and $m_{i,j}$. To define the estimator for $Z\\_W3$, we also need an
estimator for $F$ that does not have SDL, for use in the denominator:

$\hat{F}_{k,noSDL}=w\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}I[x_{i,j}^{(l)}\in\Omega_{k}]f_{i,j}.$

Then, the estimator for $Z\\_W3_{k}$ is

$\hat{Z}\\_W3_{k}=\frac{1}{\hat{F}_{k,noSDL}}w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}I[x_{i,j}^{(l)}\in\Omega_{k}]f_{i,j}\frac{w1_{i,j}}{3}.$
(11)

Notice, again, that $w1_{i,j}$ is never missing when employer $j$ is in the
observed sample. Finally, the estimator for $W1_{k}$ is

$\hat{W}1_{k}=w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}I[x_{i,j}^{(l)}\in\Omega_{k}]w1_{i,j}.$
(12)

##  4 The Bias Component of Total Error

Under the maintained assumptions that UI records are missing completely at
random and that item missing data imputed via the multiple imputation system
are ignorable, we can directly compute the bias in $B$. We evaluate

$\displaystyle E[\hat{B}_{k}-B_{k}|w,\rho]=$ $\displaystyle
w\sum_{j=1}^{J}E[I[j\in
S]|w]E[\delta_{j}|\rho]\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}E[I[x_{i,j}^{(l)}\in\Omega_{k}]]b_{i,j}$
$\displaystyle-\sum_{j=1}^{J}{\sum_{i=1}^{M_{j}}{I[x_{i,j}\in\Omega_{k}]b_{i,j}}},$

where $w$ is given, and $\rho$ are the parameters of the SDL system. $E[I[j\in
S]|w]=f=\frac{1}{w}$ because of the assumption that UI earnings records are
missing completely at random. By design, the SDL random variation is
independent of all other sources of error, and $E[\delta_{j}|\rho]=1$ for all
$\rho$ by properties of the symmetric ramp distribution used for the input
noise infusion. Finally,
$E[I[x_{i,j}^{(l)}\in\Omega_{k}]]=I[x_{i,j}\in\Omega_{k}]$, because of the
ignorable item missing data assumption and the independence of the SDL random
variable. Substituting yields

$E[\hat{B}_{k}-B_{k}|w,\rho]=\frac{w}{w}\sum_{j=1}^{J}\left[\sum_{i=1}^{M_{j}}\frac{L}{L}{I[x_{i,j}\in\Omega_{k}]b_{i,j}}-\sum_{i=1}^{M_{j}}{I[x_{i,j}\in\Omega_{k}]b_{i,j}}\right]=0.$
(13)

There is no comparable proof of unbiasedness for $\hat{M}_{k}$, $\hat{F}_{k}$,
or $\hat{Z}\\_{W3}_{k}$, because the frame can only be constructed using the
job definition reflected in $B$ (beginning-of-quarter employment) due to
limitations of date information on UI earnings records in comparison with the
date information on the QCEW data. In principle, however, the same procedures
used to compute $w$ based on $B$ could be used to compute a separate weight
based on $W1$ because the QCEW and QWI concepts and dating conventions are
identical. The alternative frame would be constructed using methods comparable
to those used in Section 3.2. In order to keep our analysis of statistics
related to $W1$ as useful as possible in understanding the properties of the
published QWI data, which use a single weight across all statistics, we do not
to re-weight $W1$.

##  5 Components of Total Variability

In this section, we exploit the structure of the LEHD data to develop
appropriate variance decomposition formulas. We proceed in three steps. First,
we show how earnings records missing due to incomplete reporting of employers
in the frame introduce simple random sampling uncertainty into the estimators
for each QWI statistic. For this component, we exploit the assumption that
these records are missing completely at random, leading to the conventional
estimand for the variance of finite-population totals and means. We implement
the conventional estimator for this estimand. Second, we show how noise
infusion introduces a multiplicative random error into the employer-level
component of each QWI statistic. The estimand does not have a simple estimator
with a closed-form solution; hence, we exploit the independence of the noise-
infusion process from all other components of total error to implement a
simulation-based estimator. These two components of error would occur even if
the data on characteristics of the jobs were complete. Finally, we show how
multiple imputation of the missing job characteristics implies the
conventional Rubin estimator for the total variation, when implemented with
the standard ignorability assumptions. In developing this estimator, we use
the first component of variance (due to randomly missing employers) as the
within-implicate estimator. The contribution of the noise infusion occurs only
at the employer level. We estimate this contribution with a simulation
estimator and add it to the between-implicate variance estimator.131313In
previous versions of this work, including [17], we developed formulas for
total variability and its components that also incorporated SDL in the
estimators for the variance component estimands that were consistent with the
noise infusion developed for the QWI/LODES publication system. This was done
to permit release of sub-state total variability statistics based on our
formulas. We are grateful to the editors and referees for urging us to
incorporate the SDL uncertainty into our overall analysis, and to purge the
total variability estimators of SDL components introduced exclusively to meet
our original goal of releasing estimates at sub-state levels. The Census
Bureau Disclosure Review Board cleared the release of the summary statistics
in this paper, which are all aggregated across 12 states, but would not have
approved the release of sub-state versions of the statistics because in this
paper we replace the required noise-infusion SDL with legacy rounding and
cell-size rules. The DRB did allow publication of the SDL component of error,
which is an important contribution to data analysis in the presence of non-
ignorable disclosure limitation [2].

We develop our analysis of the components of total variability for the
statistic $B_{k}$ first, because that measure corresponds to the central job
concept used to create the frame and generate the weights, as described in
Section 3.2. We distinguish between the variance component estimand and its
estimator. Finally, we describe how the estimator is implemented in our
variance decomposition. For the other four QWI statistics, we describe only
the changes to the components of the estimands and estimators necessary to
implement our analysis.

###  5.1 Variance Components for $B_{k}$

We begin by noting that in the absence of missing employers, disclosure
limitation, and missing job characteristics, equation (4) is identical to
equation (9), and there is no error due to the statistical processing. As
noted in Section 3.2, there may still be error in the raw data production,
ingestion and curation due to features of the Unemployment Insurance program
administration that are not modeled in this paper.

Consider the randomness due to employers in the frame not reporting UI
earnings records in a particular quarter. In this case equation (9) becomes:

$\hat{B}_{1k}=w\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]b_{i,j},$ (14)

where the subscript 1 indicates that this equation refers to the randomness
induced by the missing UI earnings records; _i.e._ , the first component of
variation. Define the complete data estimand for cell proportions as

$p_{i,j,k}=\frac{I[x_{i,j}\in\Omega_{k}]b_{i,j}}{N_{B}}.$ (15)

Then, the finite population estimator for $B_{1k}$ can be rewritten as

$\hat{B}_{1k}=N_{B}w\sum_{j=1}^{J}I[j\in S]\sum_{i=1}^{M_{j}}p_{i,j,k},$ (16)

and the estimator for the proportion of the jobs in cell $k$ is

$\hat{P}_{1k}=\frac{\hat{B}_{1k}}{N_{B}}.$ (17)

Under simple random sampling, implied by the assumption that job records are
missing completely at random, the proportions $\hat{P}_{1k}$ are unbiased
estimators of $\frac{B_{1k}}{N_{B}}$, with variance given by [8, p.51]:

$V[\hat{P}_{k}]=\frac{1}{fN_{B}}\left(\frac{B_{k}}{N_{B}}\right)\left(1-\frac{B_{k}}{N_{B}}\right)\frac{(1-f)N_{B}}{N_{B}-1}.$
(18)

The conventional unbiased estimator of $V[\hat{P}_{1k}]$ is [8, p.52]:

$\hat{V}_{1}[\hat{P}_{1k}]=\frac{1}{fN_{B}-1}\hat{P}_{1k}\left(1-\hat{P}_{1k}\right)(1-f),$
(19)

where $(1-f)$ is the finite population correction, and

$\hat{V}_{1}[\hat{B}_{1k}]=N_{B}^{2}\hat{V}_{1}[\hat{P}_{1k}].$ (20)

The second component of variance is due to employer-level input noise
infusion. Including SDL in equation (16) yields

$\hat{B}_{2k}=N_{B}w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}p_{i,j,k},$ (21)

where the subscript 2 indicates that this equation includes the randomness
induced by missing UI earnings records and SDL. Rewriting equation (21) gives

$\displaystyle\hat{B}_{2k}$
$\displaystyle=\hat{B}_{1k}+N_{B}w\sum_{j=1}^{J}I[j\in
S](\delta_{j}-1)\sum_{i=1}^{M_{j}}p_{i,j,k}$
$\displaystyle=\hat{B}_{1k}+SDL[\hat{B}_{2k}],$

where $SDL[\hat{B}_{2k}]$ is the component of $\hat{B}_{2k}$ due to SDL. By
the independence of $I[j\in S]$ and $\delta_{j}$, the expectation of
$SDL[\hat{B}_{2k}]$ is zero. $V[\hat{B}_{2k}]=V[\hat{B}_{1k}]$ \+
$V[SDL[\hat{B}_{2k}]]$ \+ $2Cov[\hat{B}_{1k},SDL[\hat{B}_{2k}]]$. By the same
independence, $Cov[\hat{B}_{1k},SDL[\hat{B}_{2k}]]=0$.141414See proof OSM-1 in
the supplemental online materials [18]. We are left to estimate
$V_{2}[\hat{B}_{2k}-\hat{B}_{1k}]$=$V_{SDL}[\hat{B}_{2k}]$, where the
subscript 2 on the variance indicates that it is the second component of
variance for $\hat{B}_{k}$, _i.e._ , the component due to independent noise
infusion.

Decompose $V_{SDL}[\hat{B}_{2k}]$ as

$V_{SDL}[\hat{B}_{2k}]=E\left[V_{SDL}[\hat{B}_{2k}]|I[j\in
S]]\right]+V\left[E[SDL[\hat{B}_{2k}]|I[j\in S]]\right].$ (22)

Note that $E[SDL[\hat{B}_{2k}]|I[j\in S]]=0$, so we only need to evaluate
$E[V_{SDL}[\hat{B}_{2k}]|I[j\in S]]$. We do this by simulating
$V_{SDL}[\hat{B}_{2k}|I[j\in S]]$ and noting that, when job records are
missing completely at random, our simulation estimator is unbiased if we use
the same weights as are used for $\hat{B}_{1k}$. Let $G$ be the number of
simulations and $g$ be the simulation index. For each simulation, and for each
employer $j$, draw $\delta_{j}^{(g)}$ from the symmetric ramp distribution
used by the QWI SDL system.151515We used the actual ramp distribution approved
for the QWI data, but the parameters of that distribution have never been
published because they are Census confidential. Compute
$SDL[\hat{B}_{2k}]^{(g)}$ using $\delta_{j}^{(g)}$. Then, the unbiased
estimator of the component of variance due to SDL in $\hat{B}_{k}$ is

$\hat{V}_{2}[\hat{B}_{2k}-\hat{B}_{1k}]=\hat{V}_{SDL}[\hat{B}_{2k}]=\frac{1}{G-1}\sum_{g=1}^{G}\left(SDL[\hat{B}_{2k}]^{(g)}\right)^{2}.$
(23)

Finally, consider the contribution to total variation arising from multiple
imputation of the job features $x_{i,j}$. We begin by noting that the
proportions defined in equation (15) are the only place where the job features
affect any of the estimators for $\hat{B}_{k}$ or components of its error
$\hat{B}_{k}-B_{k}$. For each implicate $l$, define the proportions

$p_{i,j,k}^{(l)}=\frac{I[x_{i,j}^{(l)}\in\Omega_{k}]b_{i,j}}{N_{B}}.$ (24)

Now, substitute $p_{i,j,k}^{(l)}$ into equation (21) to get

$\hat{B}_{k}^{(l)}=N_{B}w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}p_{i,j,k}^{(l)}.$ (25)

Notice that equation (25) is identical to equation (9) before averaging over
the $L$ implicates of the multiple imputation. Therefore, we can apply the
multiple imputation formulas directly to $\hat{B}_{k}^{(l)}$ yielding the
estimator in equation (9) and

$\hat{V}[\hat{B}_{k}]=\frac{1}{L}\sum_{l=1}^{L}\hat{V}[\hat{B}_{k}^{(l)}]+\frac{(L+1)}{L}\frac{1}{L-1}\sum_{l=1}^{L}\left(\hat{B}_{k}^{(l)}-\hat{B}_{k}\right)^{2}.$
(26)

For completeness we define the average within-implicate variance as

$\hat{V}_{W}[\hat{B}_{k}]=\frac{1}{L}\sum_{l=1}^{L}\hat{V}[\hat{B}_{k}^{(l)}]$
(27)

and the between-implicate variance due to imputation as

$\hat{V}_{B}[\hat{B}_{k}]=\frac{1}{L-1}\sum_{l=1}^{L}\left(\hat{B}_{k}^{(l)}-\hat{B}_{k}\right)^{2}.$
(28)

In the absence of SDL, implementing equation (26) would be straightforward,
given the formulas already derived in this section. The first term, the
average within-implicate variance, would be estimated by evaluating the
variance component $\hat{V}[\hat{B}_{1k}]$ in equation (20) for each implicate
and averaging. The second component would be computed by evaluating the
estimator in equation (16) for each implicate $\hat{B}_{k}^{(l)}$, averaging
to obtain $\hat{B}_{k}$, and substituting directly into the formula in
equation (26).

In the presence of multiplicative noise infusion SDL, the answer is not so
straightforward. We should use equation (25) to evaluate $\hat{B}_{k}^{(l)}$.
Then, use equations (20) and (23) to estimate the within-component of
multiple-imputation variance. But, as we showed above, the SDL is applied at
the employer-level, and the data on $b_{i,j}$ are never missing. Hence,
whether $V_{SDL}[\hat{B}_{2k}]$ is computed for each implicate, or computed
once for an arbitrary implicate should not matter. Conceptually, the error in
$SDL[\hat{B}_{2k}]$ is a component of total error independent of both the
record-level and item-level missing data randomness. We could estimate its
contribution by computing the components of variance due to missing UI
earnings records and missing job features as described in the paragraph above
but with $\delta_{j}=1$ everywhere, then add our estimate of
$V_{SDL}[\hat{B}_{2k}]$ to that. This is the method used in the tables of the
paper.

Because $\delta_{j}$ multiplies the sample selection random variable $I[j\in
S]$ and, consequently, the employer-level estimate
$\sum_{i=1}^{M_{j}}p_{i,j,k}^{(l)}$, the multiple-imputation variance formulas
are affected by its presence in equation (21). The expected value of this
interaction is zero for $B$, $M$ and $F$, but positive for $Z\\_W3$ and $W1$.
Consequently, we present estimates in the supplemental online materials that
include $\delta_{j}$ in the computation of the multiple imputation variance
components [18].161616We verified by small-scale simulation that the
differences between the text and online supplemental tables were random
fluctuations around the expected value of zero for $B$, $M$ and $F$. We also
verified that the inflation of the variance components of $Z\\_W3$ and $W1$
shown in the online supplemental tables was consistent with the known, but
still confidential, parameters of the distribution of $\delta_{j}$.

To summarize, in the text tables, columns estimating the average within-
implicate component of total variance average the values of
$\hat{V}_{1}[\hat{B}_{1k}^{(l)}]$ over $l=1,...,L=10$ implicates. That is,
they substitute $\hat{B}_{1k}^{(l)}$, defined for each implicate using
equation (16), for $\hat{B}_{k}^{(l)}$ in equation (27). In the text tables,
columns estimating the between-implicate component of variance due to multiple
imputation, substitute $\hat{B}_{1k}^{(l)}$ for $\hat{B}_{k}^{(l)}$ and
compute $\hat{B}_{k}$ with all $\delta_{j}=1$ in equation (28). In the text
tables, columns estimating the between component of total variance due to
noise infusion compute $\hat{V}_{SDL}[\hat{B}_{2k}]$ using equation (23)
exactly as defined, holding $x_{i,j}^{(l)}=x_{i,j}^{(1)}$ with $G=10$.

In the online supplemental tables, the columns estimating the average within-
implicate component of total variance evaluate equation (27) by substituting
$\hat{V}[\hat{B}_{1k}]$ evaluated using $\hat{B}_{k}^{(l)}$ for each implicate
for $\hat{V}[\hat{B}_{k}^{(l)}]$. In the online supplemental tables, the
columns estimating the between-implicate component of variance due to multiple
imputation estimate equation (28) exactly as shown in the text. Finally, in
the online supplemental tables, columns estimating the between component of
total variance due to noise infusion compute $\hat{V}_{SDL}[\hat{B}_{2k}]$
using equation (23) exactly as defined, holding $x_{i,j}^{(l)}=x_{i,j}^{(1)}$
with $G=10$.

In all results, the total variability is estimated as

$\hat{V}_{T}[\hat{B}_{k}]=\hat{V}_{W}[.]+\frac{L+1}{L}\left(\hat{V}_{B}[.]+\hat{V}_{SDL}[\hat{B}_{2k}]\right),$
(29)

with the appropriate estimators, as described in the paragraphs above,
substituted for $\hat{V}_{W}[.]$ and $\hat{V}_{B}[.]$. The coefficient of
variation is estimated as

$\hat{CV}=\frac{\sqrt{\hat{V}_{T}[\hat{B}_{k}]}}{\hat{B}_{k}}.$ (30)

To permit the estimation of approximate confidence intervals, we estimate
approximate degrees of freedom using the moment-matching formula from [23]

$\hat{DF}[\hat{B}_{k}]=min\left[N_{k}-1,\left(L-1\right)\left(1+\frac{L}{L+1}\frac{\hat{V}_{W}[.]}{\left(\hat{V}_{B}[.]+\hat{V}_{SDL}[\hat{B}_{2k}]\right)}\right)^{2}\right],$
(31)

where $N_{k}=\frac{\hat{B}_{1k}}{w}$ is the observed job count in cell $k$.

###  5.2 Modifications of the Formulas for $M$ and $F$

The formulas for $\hat{V}[\hat{M}_{k}]$ and $\hat{V}[\hat{F}_{k}]$, and their
components, are comparable to equation (29); however, unlike for $B$-based
jobs, the total populations of $M$\- and $F$-based jobs are unknown, as
explained in Section 3.2. Let $N_{M}$ and $N_{F}$ be the unknown total
population of $M$\- and $F$-based jobs, respectively. We assume that the same
fraction $f$ of these jobs are observed as for $B$, and that the missing
fraction $(1-f)$ of UI earnings records required to compute $m_{i,j}$ and
$f_{i,j}$, respectively, are missing completely at random. In each quarter,
then, let

$\displaystyle N_{M}$ $\displaystyle=w\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}m_{i,j}$ $\displaystyle N_{F}$
$\displaystyle=w\sum_{j=1}^{J}I[j\in S]\sum_{i=1}^{M_{j}}f_{i,j}.$ (32)

We now enumerate the modifications to equations (14)-(31) required to estimate
components of variance for $\hat{M}_{k}$ (resp., $\hat{F}_{k}$). In all
equations, substitute $N_{M}$ (resp., $N_{F}$) for $N_{B}$, substitute
$m_{i,j}$ (resp., $f_{i,j}$) for $b_{i,j}$, and substitute the symbol $M$
(resp., $F$) for the symbol $B$.

###  5.3 Variance Components for $Z\\_{W3}$

We now explain the modifications to equations (14)-(31) required to estimate
components of variance for $\hat{Z}\\_W3_{k}$. The frame for $Z\\_W3$ is the
same as the frame for $F$; hence, the total population of $F$-based jobs,
$N_{F}$ substitutes for $N_{B}$. Notice that the estimand (6) is a sub-
population average for monthly earnings of full-quarter employees; hence, we
need to build the conventional finite-population variance estimand and
estimator. Equations (14) -(15) are not needed because the estimator is
already in the required ratio form. Replace equation (16) with

$\hat{Z}\\_W3_{1k}=\frac{1}{\hat{F}_{k,noSDL}}w\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]f_{i,j}\frac{w1_{i,j}}{3}.$ (33)

We can proceed directly to the replacements for the variance equations.
Because we assumed that unreported UI earnings records are missing completely
at random, the finite-population variance of a mean from a simple random
sample of the proportion $f$ of the $F$-based jobs is given by [8, p. 23]:

$V_{1}[\hat{Z}\\_W3_{1k}]=\sum_{j=1}^{J}\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]\frac{\left(f_{i,j}\frac{w1_{i,j}}{3}-Z\\_W3_{k}\right)^{2}}{\hat{F}_{k,noSDL}-1}\frac{(1-f)}{f\hat{F}_{k,noSDL}}$
(34)

Note that $\hat{F}_{k,noSDL}$ is the estimated number of $F_{k}$ jobs in cell
$k$, and, therefore, $f\hat{F}_{k,noSDL}$ is the number of $F_{k}$ jobs in the
sample. The conventional unbiased estimator for a sub-population mean replaces
equation (20) and is given by [8, p. 26]:

$\hat{V}_{1}[\hat{Z}\\_W3_{1k}]=\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]\frac{\left(f_{i,j}\frac{w1_{i,j}}{3}-\hat{Z}\\_W3_{k}\right)^{2}}{f\hat{F}_{k,noSDL}-1}\frac{(1-f)}{f\hat{F}_{k,noSDL}}.$
(35)

Equation (21) is replaced by its analogue for $\hat{Z}\\_W3_{2k}$:

$\hat{Z}\\_W3_{2k}=\frac{1}{\hat{F}_{k,noSDL}}w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]f_{i,j}\frac{w1_{i,j}}{3}.$
(36)

Equation (22) and the analysis producing equation (23) are unchanged after
substituting $\hat{Z}W\\_3_{1k}$ for $\hat{B}_{1k}$ and $\hat{Z}\\_W3_{2k}$
for $\hat{B}_{2k}$. Equation (24) is not needed. Equation (25) is replaced
with:

$\hat{Z}\\_W3_{k}^{(l)}=\frac{1}{\hat{F}_{k,noSDL}}w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}I[x_{i,j}^{(l)}\in\Omega_{k}]f_{i,j}\frac{w1_{i,j}}{3}.$
(37)

Note that the divisor $\hat{F}_{k,noSDL}$ has been averaged over the
implicates. Equations (26)-(31) require only the substitution of the $Z\\_W3$
analogue of the $B$ quantity.

###  5.4 Variance Components for $W1$

Finally, we explain the modifications to equations (14)-(31) required to
estimate components of variance for $\hat{W}1_{k}$. The frame for
$\hat{W}1_{k}$ is the same as the frame for $M$; hence, the total population
of $M$-based jobs, $N_{M}$ substitutes for $N_{B}$. Notice that the estimand
(7) is a sub-population total for quarterly earnings of all employees
regardless of when they were active during the quarter ($M$-based jobs);
hence, we need to build the conventional finite-population variance estimand
and estimator. Equations (14) -(15) are not needed because the estimator can
be derived directly. Replace equation (16) with

$\hat{W}1_{1k}=w\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]w1_{i,j}.$ (38)

We can proceed directly to the replacements for the variance equations. Again,
because we assumed that unreported UI earnings records are missing completely
at random, the finite-population variance of a total from a simple random
sample of the proportion $f$ of the $M$-based jobs is given by [8, p. 24]:

$V_{1}[\hat{W}_{1k}]=\sum_{j=1}^{J}\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]\frac{\left(w1_{i,j}-\frac{W1_{k}}{M_{k}}\right)^{2}}{\hat{M}_{k,noSDL}-1}\frac{\hat{M}_{k,noSDL}^{2}(1-f)}{f\hat{M}_{k,noSDL}}$
(39)

where the estimated population of $M$-based jobs without SDL is

$\hat{M}_{k,noSDL}=w\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}\frac{1}{L}\sum_{l=1}^{L}I[x_{i,j}^{(l)}\in\Omega_{k}]m_{i,j}.$
(40)

Note that $\hat{M}_{k,noSDL}$ is the estimated population of $M_{k}$ jobs in
cell $k$, and, therefore, $f\hat{M}_{k,noSDL}$ is the number of $M_{k}$ jobs
in the sample.

The conventional unbiased estimator for a sub-population total replaces
equation (20) and is given by [8, p. 26]:

$\hat{V}_{1}[\hat{W}1_{1k}]=\sum_{j=1}^{J}I[j\in
S]\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]\frac{\left(w1_{i,j}-\frac{\hat{W}1_{k}}{M_{k}}\right)^{2}}{f\hat{M}_{k,noSDL}-1}\frac{\hat{M}_{k,noSDL}^{2}(1-f)}{f\hat{M}_{k,noSDL}}.$
(41)

Equation (21) is replaced by its analogue for $\hat{W}1_{k}$:

$\hat{W}1_{2k}=w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}I[x_{i,j}\in\Omega_{k}]w1_{i,j}.$ (42)

Equation (22) and the analysis producing equation (23) are unchanged after
substituting $\hat{W}1_{1k}$ for $\hat{B}_{1k}$ and $\hat{W}1_{2k}$ for
$\hat{B}_{2k}$. Equation (24) is not needed. Equation (25) is replaced with:

$\hat{W}1_{k}^{(l)}=w\sum_{j=1}^{J}I[j\in
S]\delta_{j}\sum_{i=1}^{M_{j}}I[x_{i,j}^{(l)}\in\Omega_{k}]w1_{i,j}.$ (43)

Equations (26)-(31) require only the substitution of the $W1$ analogue of the
$B$ quantity.

##  6 Results

We summarize the results for all establishments in the universe in Table
LABEL:tab:emptotall for total employment, $M$; in Table LABEL:tab:emp for
beginning-of-quarter employment, $B$; in Table LABEL:tab:empsall for full-
quarter employment, $F$; in Table LABEL:tab:ametot for average monthly
earnings of full-quarter employees, $Z\\_{W3}$; and Table LABEL:tab:payroll
for total payroll, $W1$. In all cases, $L=G=10$.

See pages 1 of qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf

See pages 2 of qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf

See pages 3 of qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf

See pages 4 of qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf

See pages 5 of qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf

###  6.1 Interpretation of the Tables

Tables LABEL:tab:emptotall-LABEL:tab:payroll have the same structure. The
supplemental online files contain tables A1-A5, which have the same structure
as the analogous table in the main text. The supplemental online files also
contain additional columns for Tables 1-5. The major row label is the level of
QWI tabulation. For example, the row labeled “Age $\times$ Gender” refers to
the collection of tabulations stratified by year, quarter, state, age
category, and gender. The data conform to the published QWI schema in [25],
which contains the levels for each of the stratifying variables. The minor row
label characterizes the publication cell by its job count size. For Tables
LABEL:tab:emptotall and LABEL:tab:payroll the size classes are based on total
flow-employment $M$, for Table LABEL:tab:emp, on the values of beginning-of-
quarter employment $B$, and for Tables LABEL:tab:empsall and LABEL:tab:ametot,
the classes are based on full-quarter employment $F$. The complete set of size
classes we summarize are:

  * •

Zero measured value, after rounding, which means that the estimated value is
zero. (All structural zeros are outside the frame.)

  * •

1-2, 3-9, 10-99, 100-999, which in each case means that the estimated value is
in the closed interval after rounding.

  * •

+1000, which means that the estimated value is in the interval $[1000,max]$
after rounding.

We report medians rather than averages for most statistics, to avoid the
influence of outlier cells on the results. “Median Total Variation” reports
the the total variation $\hat{V}_{T}$ at the median values of each component.
This is the overall summary measure of data quality for the five statistics
from QWI and LODES studied in this paper. The remaining columns provide
additional information for interpreting total variation and attributing
components to the various sources. The next three columns report the percent
of total variation due to “Within” ($\hat{V}_{W}$), “Between (Imputation)”
($\hat{V}_{B}$), and “Between (Noise Infusion)” ($\hat{V}_{SDL}$), evaluated
at the median values of each component. The between components are both
multiplied by $(L+1)/L$ so that the components sum to one when divided by the
median total variation.

We also report three quantiles of the coefficient of variation, defined as a
function of the total variance and the QWI estimator in (30). The quantile
statistics on the coefficient of variation can be used to assess the
proportionate total variation around the published value arising from all
sources of error. They can also be used as a portmanteau “fitness for use”
assessment of the cells in the associated row, considered as a single table.

We also produced statistics for the margin of error at the median total
variation, calculated at the 90% confidence level. We used the approximate
degrees of freedom as defined in equation (31) to compute the margin of error.
These statistics can be found in the supplemental online files with the same
table numbers as in the text. The last column in each table is one-half of the
90% confidence interval width using a $t$-distribution with the indicated
degrees of freedom.

We interpret the approximate margins of error as providing evidence about the
overall reliability of each statistic for cells that lie in the indicated
count range. For example, the median value of $B$ associated with the Age
$\times$ Gender cell in Table LABEL:tab:emp, in the +1000 row, is 22,385. The
approximate margin of error in that row is 289. Hence, the approximate 90%
confidence interval is 22,385 +/- 289. The median coefficient of variation is
0.0074 or 0.74%.

###  6.2 Discussion of Data Quality or Fitness-for-use

The total variation or the coefficient of variation can be used to summarize
the fitness-for-use of the published indicators for total flow-employment,
beginning-of-quarter employment, full-quarter employment, average monthly
earnings of full-quarter employees, and total quarterly payroll. It is clear
from Tables LABEL:tab:emptotall-LABEL:tab:payroll that the coefficient of
variation declines monotonically as the number of jobs used in the tabulation
value increases for each of the displayed quantiles. Careful attention to the
magnitudes of these coefficients of variation reveals that for even the most
detailed tables and for the stratifiers associated with the largest noise
infusion and between-implicate variance contributions to total variability,
the tabulations are very reliable when based on job counts of at least 10, and
moderately reliable for job counts of three to nine. This conclusion remains
valid even using the very conservative $95^{th}$ percentile of the
distribution of the coefficient of variation.171717The American Community
Survey uses a median coefficient of variation no greater than 61% as one of
its portmanteau fitness-for-use measures for the 1-year tabular summaries [24,
Section 13.7].

By contrast, the percent of total variation due to noise infusion is not a
traditional measure of fitness for use. It captures the contribution of the
error deliberately introduced to protect the confidentiality of the micro-
data. It is a necessary “cost of doing business” in the production of
statistics with granularity at the level published in the QWIs and
LODES/OnTheMap.

In all of the tables, the total variability declines as the number of jobs in
a cell increases, however the components of variation decline at different
rates. The between-implicate variation declines at a faster rate than the
noise infusion component as the number of jobs in a cell increases. As a
result, the noise infusion component accounts for a relatively larger share of
the total variability in cells that contain more jobs, but the total
variability of those cells is smaller. In cells with many jobs ($100-999$ and
$+1000$) the majority of the total variability is due to noise infusion. For
example, the median coefficient of variation is 0.0074 or 0.74% in Table
LABEL:tab:emp in the row Age $\times$ Gender $+1000$, which is almost entirely
due to noise infusion (95.1%). This implies that the 289 margin of error at
the median associated with a 90% confidence level is due almost entirely to
the confidentiality protection system, and not to any of the other sources of
error.

Education is imputed for the vast majority (about 87%) of individuals in the
LEHD data, based on a multistage ignorable missing data model that relies
heavily on the sampling properties of the Census 2000 long form and the
ACS—specifically that education is missing because the individual was not
sampled [17]. By contrast, worker age and gender are imputed for less than
seven percent of the individuals, race and ethnicity are imputed for about 18%
of the individuals. Looking closely at the median coefficients of variation
for the Age $\times$ Gender $\times$ Industry $\times$ County table in
comparison with the Gender $\times$ Education $\times$ Industry $\times$
County table, we see that for every count range, the Age $\times$ Gender table
has less total variation than the Gender $\times$ Education table and the
percentage due to between-implicate variation is less, except for cells with
measured zeros. Notice, in particular, that the percent of total variation due
to between-implicate variance is substantially similar in all rows labeled
$+1000$ when the table includes education, indicating that the high imputation
rate for education is a limiting factor. This is not the case for tables
including only age and gender or race and ethnicity. One conclusion is, thus,
that the total quality of the QWI data could be substantially improved by
investing in better education data. However, by conventional measures of
general fitness-for-use, the tabulations involving education are of high
quality but not as high as the quality of statistics that do not use
education.

##  7 Conclusion

We have conducted the first comprehensive total error and variability analysis
of five major publication variables in the Quarterly Workforce Indicators,
including two key employment indicators and the most widely used earnings
indicator. The beginning-of-quarter employment variable from QWI is also the
primary tabulation variable in the LEHD Origin-Destination Employment
Statistics; hence, our analysis is also applicable to workplace tabulations
directly from LODES or as displayed in OnTheMap, including OnTheMap for
Emergency Management. Tabulations involving 10 or more jobs are very reliable
having median coefficients of variation that decline from a worst case of 16%
(count range 10-99, detailed tables involving education) to a best case of
less than one percent (count range +1000, simple tables involving education).
Tabulations based on three to nine jobs represent a transition zone in the
sense that the 90% confidence bound does not generally include zero, however a
substantial number of cells are not statistically different from zero,
especially for tabulations by education. Our analysis further reveals that the
very smallest tabulations (estimated zeros and counts of one or two) are not
particularly reliable in the sense that they could easily range from zero to
three or more; however, the QWI tabulations already suppress estimates of one
or two with a flag that warns users of their unreliability and LODES/OnTheMap
primarily uses them to build larger aggregates that should be reliable by our
measures. Finally, our analysis includes the deliberate error introduced by
noise infusion for the purpose of statistical disclosure limitation; hence,
our total variation measures promote correct inferences from the published
data even in the presence of uncertainty due to SDL.

To the best of our knowledge, no other widely used statistical system based on
administrative records has produced a comprehensive total error analysis to
which the results in this paper can be compared. As compared to survey-based
estimates like those derived from the American Community Survey, for example,
the QWI employment and earnings tabulations have accuracy comparable to that
of the ACS [24], even when comparing state and PUMA-level estimates in the ACS
to county and core-based statistical areas in the QWI. The LODES/OTM estimates
for sub-county geographies and small sub-populations have much lower total
error than estimates from the ACS for comparably-sized sub-populations. The
ACS margins of error do not account for the uncertainty introduced by the
edit, imputation and statistical disclosure limitation systems, whereas ours
do. Designed surveys like the ACS deliver statistics on a much broader set of
variables and can be used for analyses that are far outside the scope of the
QWIs or LODES/OTM. But our analyses demonstrate that the total error of an
administrative-records based publishing system that combines data from many
sources can compare very favorably with much more expensive survey-based
systems for their common domains.

## References

  * [1] John M. Abowd, Bruno Crépon and Francis Kramarz  “Moment Estimation with Attrition: An Application to Economic Models”  In _Journal of the American Statistical Association_ 96.456, 2001, pp. 1223–1231
  * [2] John M. Abowd and Ian M. Schmutte  “Economic analysis and statistical disclosure limitation” Spring  In _Brookings Papers on Economic Activity_ , 2015, pp. 221–267  URL: https://www.brookings.edu/wp-content/uploads/2015/03/AbowdText.pdf
  * [3] John M. Abowd, Bryce E. Stephens, Lars Vilhuber, Fredrik Andersson, Kevin L. McKinney, Marc Roemer and Simon Woodcock  “The LEHD Infrastructure Files and the Creation of the Quarterly Workforce Indicators”  In _Producer Dyanamics: New Evidence from Micro Data_ University of Chicago Press, 2009, pp. 149–230  DOI: 10.1257/0002828041301812
  * [4] John M Abowd, Kaj Gittings, Kevin L McKinney, Bryce E Stephens, Lars Vilhuber and Simon Woodcock  “Dynamically consistent noise infusion and partially synthetic data as confidentiality protection measures for related time-series”  In _Federal Committee on Statistical Methodology, 2012 Research Conference Papers_ , 2012  URL: http://fcsm.sites.usa.gov/files/2014/05/Vilhuber_2012FCSM_VIII-C.pdf
  * [5] Gary Benedetto, John Haltiwanger, Julia Lane and Kevin Mckinney  “Using Worker Flows to Measure Firm Dynamics”  In _Journal of Business & Economic Statistics_ 25.3, 2007, pp. 299–313  DOI: 10.1198/073500106000000620
  * [6] Paul P. Biemer  “Total Survey Error: Design, Implementation, and Evaluation”  In _Public Opinion Quarterly_ 74.5, 2010, pp. 817  DOI: 10.1093/poq/nfq058
  * [7] Irene Brown, Marilyn Balogh, Anthony Caruso, Beth Schlein and Katherine J. Thompson  “Investigation of Variance Properties of Noise-Infused Estimates for the Survey of Business Owners (SBO)”, Proceedings of the Survey Research Methods Section of the American Statistical Association, 2009, pp. 1708–1720  URL: http://www.asasrms.org/Proceedings/y2009/Files/303766.pdf
  * [8] William G. Cochran  “Sampling Techniques”  Wiley, 1977
  * [9] Michael E Davern, Bruce D Meyer and Nikolas K Mittag  “Creating Improved Survey Data Products Using Linked Administrative-Survey Data”  In _Journal of Survey Statistics and Methodology_ , 2018, pp. smy017  DOI: 10.1093/jssam/smy017
  * [10] Irit Dinur and Kobbi Nissim  “Revealing information while preserving privacy”  In _Proceedings of the 22nd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems_ , 2003, pp. 202–210  DOI: 10.1145/773153.773173
  * [11] Cynthia Dwork, Frank McSherry, Kobbi Nissim and Adam Smith  “Calibrating Noise to Sensitivity in Private Data Analysis”  In _Theory of Cryptography. TCC 2006_ , Lecture Notes in Computer Science  Springer, 2006, pp. 265–284
  * [12] Eurostat  “Quality report of the European Union Labour Force Survey 2013”, eurostat Statistical working papers, 2014
  * [13] Timothy Evans, Laura Zayatz and John Slanta  “Using Noise for Disclosure Limitation of Establishment Tabular Data”  In _Journal of Official Statistics_ 14.4, 1998, pp. 537–551  URL: http://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/using-noise-for-disclosure-limitation-of-establishment-tabular-data.pdf
  * [14] Samuel Haney, Ashwin Machanavajjhala, John M. Abowd, Matthew Graham, Mark Kutzbach and Lars Vilhuber  “Utility Cost of Formal Privacy for Releasing National Employer-Employee Statistics”  In _SIGMOD ’17 Proceedings of the 2017 International Conference on Management of Data_ New York, NY, USA: ACM, 2017  DOI: 10.1145/3035918.3035940
  * [15] Michael Horrigan, Polly Phipps and Scott Fricker  “Development of a Quality Framework and Quality Indicators at the Bureau of Labor Statistics”  In _Joint Statistical Meetings 2014 -Government Statistics Section_ , 2014, pp. 325–338
  * [16] Roderick J.A. Little and Donald B. Rubin  “Statistical Analysis with Missing Data”  Hoboken, NJ: Wiley, 2002
  * [17] Kevin L. McKinney, Andrew Green, Lars Vilhuber and John M. Abowd  “Total Error and Variability Measures with Integrated Disclosure Limitation for Quarterly Workforce Indicators and LEHD Origin Destination Employment Statistics in On The Map”, 2017  URL: https://ideas.repec.org/p/cen/wpaper/17-71.html
  * [18] Kevin M. McKinney, Andrew Green, Lars Vilhuber and John M. Abowd  “Online Supplemental Materials for: ”Total Error and Variability Measures for the Quarterly Workforce Indicators and LEHD Origin Destination Employment Statistics in OnTheMap””, 2020  DOI: 10.5281/zenodo.3951670
  * [19] Mary H. Mulry and Andrew D. Keller  “Comparison of 2010 Census Nonresponse Follow-Up Proxy Responses with Administrative Records Using Census Coverage Measurement Results”  In _Journal of Official Statistics_ 33.2, 2017, pp. 455–475  DOI: 10.1515/jos-2017-0022
  * [20] Giles Reid, Felipa Zabala and Anders Holmberg  “Extending TSE to Administrative Data: A Quality Framework and Case Studies from Stats NZ”  In _Journal of Official Statistics_ 33.2, 2017, pp. 477–511  DOI: https://doi.org/10.1515/jos-2017-0023
  * [21] Donald B. Rubin  “Multiple Imputations in Sample Surveys—A Phenomenological Bayesian Approach to Nonresponse”, Proceedings of the Survey Research Methods Section of the American Statistical Association, 1978, pp. 20–34
  * [22] Donald B. Rubin  “Multiple Imputation for Nonresponse in Surveys”  New York: Wiley, 1987
  * [23] Donald B Rubin and Nathaniel Schenker  “Multiple Imputation for Interval Estimation From Simple Random Samples With Ignorable Nonresponse”  In _Journal of the American Statistical Association_ 81.394, 1986, pp. 366–374
  * [24] U.S. Census Bureau  “American Community Survey Multiyear Accuracy of the Data”, 2015, pp. 1–29  URL: http://www2.census.gov/programs-surveys/acs/tech_docs/accuracy/MultiyearACSAccuracyofData2014.pdf
  * [25] Lars Vilhuber and Heath Hayward  “labordynamicsinstitute/qwi_schemas: LEHD Schema V4.1.0”, 2017  DOI: 10.5281/zenodo.346006