# Algorithm-Assisted Decision Making and Racial Disparities in Housing: A
Study of the Allegheny Housing Assessment Tool

Lingwei Cheng1, Cameron Drayton1, Alexandra Chouldechova1, Rhema
Vaithianathan2 We thank the Allegheny County Department of Human Services for
their invaluable support and feedback.

###### Abstract

The demand for housing assistance across the United States far exceeds the
supply, leaving housing providers the task of prioritizing clients for receipt
of this limited resource. To be eligible for federal funding, local
homelessness systems are required to implement assessment tools as part of
their prioritization processes. The Vulnerability Index Service Prioritization
Decision Assistance Tool (VI-SPDAT) is the most commonly used assessment tool
nationwide. Recent studies have criticized the VI-SPDAT as exhibiting racial
bias (Cronley 2022; Petry et al. 2021), which may lead to unwarranted racial
disparities in housing provision. Such criticisms have led certain
jurisdictions to develop alternative tools. Using data from one such
prioritization tool, called the Allegheny Housing Assessment (AHA), we use
descriptive and quantitative analysis to assess whether the replacement of the
VI-SPDAT with the AHA impacts racial disparities in housing allocation. We
find that the VI-SPDAT tended to assign higher risk scores to white clients
and lower risk scores to Black clients, and that white clients were served at
a higher rates pre-AHA deployment. While post-deployment service decisions
became better aligned with the AHA score, and the distribution of AHA scores
is similar across racial groups, we do not find evidence of a corresponding
decrease in disparities in service rates. We attribute the persistent
disparity to the use of Alt-AHA, a survey-based tool that is used in cases of
low data quality, as well as group differences in eligibility-related factors,
such as chronic homelessness and veteran status. We discuss the implications
for housing service systems seeking to reduce racial disparities in their
service delivery.

##  1 Introduction

On any given night, roughly half a million people in the United States are
experiencing homelessness (U.S. Department of Housing and Urban Development
2022). Black and African American persons are disproportionately represented
among the homeless population, with lifetime rates of homelessness estimated
to be more than three times higher for non-Hispanic Blacks compared with non-
Hispanic whites (Fusaro, Levy, and Shaefer 2018). Some but not all of this
disparity is explained by racial differences in income, incarceration rates,
and exposure to traumatic events (Day 2015). Providing long-term supportive
housing is the single most effective intervention for reducing homelessness
(Evans, Philips, and Ruffini 2019).

As a requirement for participating in programs funded by the United States
Department of Housing and Urban Development (HUD), local homelessness systems
must implement assessment tools that prioritize the most vulnerable homeless
clients for supportive services. The most common housing assessment tool is an
interview-based instrument called the Vulnerability Index Service
Prioritization Decision Assistance Tool (VI-SPDAT) (Community Solutions 2016).
Problematically, recent studies suggest that the VI-SPDAT is poorly validated
(Brown et al. 2018) and racially biased (Cronley 2022; Petry et al. 2021).
Partly in response to such criticism, homelessness systems across the country
have been exploring alternative assessment tools, including ones based on
predictive risk models.

We study one such model developed and deployed by the Allegheny County
Department of Human Services (ACDHS). Allegheny County is a medium-sized US
county in Pennsylvania that contains the city of Pittsburgh. Allegheny
County’s Continuum of Care (CoC) receives HUD funding to coordinate and
provide homelessness services such as shelters, supportive housing, outreach,
engagement and assessment, and prevention strategies. ACDHS is responsible for
managing HUD funding and operating the Coordinated Entry system that assesses
and refers clients to CoC providers (Allegheny County Execuitve Office 2023).

In 2020 the county deployed a tool called the Allegheny Housing Assessment
(AHA) to “standardize the assessment process and move away from the interview-
based VI-SPDAT approach to an automated assessment based on administrative
data that can better predict the likelihood of future adverse outcomes”
(Vaithianathan and Kithulgoda 2020a). In this paper, we study the transition
from VI-SPDAT to the predictive risk model-based risk assessment, AHA, and
evaluate the AHA tool’s impact on racial disparities in access to supportive
housing services. Specifically, we want to understand:

  1. 1.

How did the prioritization for housing change when the VI-SPDAT was replaced
with the AHA?

  2. 2.

How does AHA impact the racial disparities in housing service enrollment
rates?

  3. 3.

How does the use of the alternative survey assessment tool (Alt-AHA) impact
racial disparities?

While there have been extensive evaluations of the performance, ethics, and
algorithmic fairness of the AHA models (Vaithianathan and Kithulgoda 2020a;
Eticas Research and Consulting 2020; Allegheny County Analytics 2020), we are
among the first to study the impact of its deployment111We are aware of other
ongoing work, but none that has been published..

Using extensive descriptive and quantitative analysis, we find that prior to
the deployment of AHA, VI-SPDAT tended to assign higher risk scores to white
clients and lower risk scores to Black clients. While there was no disparity
in service rates conditioned on VI-SPDAT scores, the net effect was that white
clients were served at a higher rate. Following the adoption of AHA, there was
an increase in service to Black clients, especially those at medium- and high-
risks. However, these increases paralleled those for white clients, and thus
did not result in a reduction in racial disparities. While there is no
evidence of racial disparity among clients assessed with AHA only, there is a
significant racial disparity in service rates among clients assessed with Alt-
AHA. White clients were about twice as likely as Black clients to be assessed
with Alt-AHA, and conditional on being assessed were much more likely to be
served. The differences in Alt-AHA assessment rates is due in large part to
differences in data quality, which is significantly lower for white clients.
Disparities in service rates across groups following an Alt-AHA assessment are
at least partly attributable to other explanatory factors that affect
eligibility, such as chronic homeless, disability, and veteran status.

Our study contributes to the literature on algorithmic fairness and human-AI
interaction in several important ways. First, to our knowledge, it is the
first study to investigate how algorithm-assisted decision-making affects
housing disparities using real data from a deployed tool. Second, we directly
study algorithm-assisted decision outcomes of an automated risk assessment
tool in the presence of a manual assessment option (the Alt-AHA). Lastly, by
identifying important differences between the widely used VI-SPDAT and a
machine learning–based alternative (AHA), our study has important policy
implications for housing providers.

##  2 Background

In response to longstanding housing shortfalls, in 2008 federal rules began
incentivizing housing providers to use risk assessment tools to ensure that
the most vulnerable clients are prioritized for supportive housing. To receive
federal funding, HUD requires housing providers to use tools that assess a
client’s housing and service needs and eligibility, and score their risk
vulnerability (U.S. Department of Housing and Urban Development 2017).

Figure 1: Mapping of Predictive Risk Model Scores to AHA Scores(Vaithianathan
and Kithulgoda 2020a)

In this section we describe the most widely used tool, the VI-SPDAT, and then
detail the development and deployment of the AHA tool that has since replaced
the VI-SPDAT in ACDHS operations.

###  2.1 VI-SPDAT

The VI-SPDAT was developed by merging OrgCode’s SPDAT (Service Prioritization
Decision Assistance Tool) Prescreen tool with Community Solutions’ VI
(Vulnerability Index) tool. It consists of a series of survey questions that
Coordinated Entry (CE) workers administer at the time of intake to understand
the client’s level of risk and need. The tool has been tremendously popular
and, by 2015, was being used in at least 39 states and countries (National
Alliance to End Homelessness 2022). Until 2020, it was also the tool used in
Allegheny County.

Recent research has concluded that the VI-SPDAT is poorly validated (Brown et
al. 2018) and racially biased (Cronley 2022; Petry et al. 2021; Institute for
Research on Poverty 2021). Moreover, the assessment questions can be invasive,
asking about drug use, risk-taking behavior, and mental health diagnoses.
Focus groups conducted in Allegheny County showed that former clients had
concerns about the long and detailed 45 minute interview the tool entailed,
and specifically its potential to re-traumatize. Despite the invasive
questions, a 2018 analysis by ACDHS found that the VI-SPDAT was not effective
at identifying clients who are at greatest risk of adverse outcomes associated
with housing instability and the scores did not significantly predict risk of
return to homeless services. In part in response to criticism, by December
2020 the maintainers of VI-SPDAT had announced that they would phase out the
tool (National Alliance to End Homelessness 2022).

###  2.2 The Allegheny Housing Assessment

In 2014, ACDHS issued a request for proposals (RFP) to explore the use of
predictive decision support tools in their operations. The RFP resulted in a
research partnership between ACDHS and the Center for Social Data Analytics at
the Auckland University of Technology (CSDA). By 2018, CSDA began working on a
machine learning–based risk assessment model that would eventually become the
Allegheny Housing Assessment (AHA).

The AHA tool is based on three separate machine learning models, each trained
to predict the likelihood of a given adverse event occurring in the 12 months
following a housing services assessment: (i) a jail stay; (ii) a Medicaid-
funded inpatient hospital stay for a behavioral health event; and (iii) 4 or
more Medicaid-funded Emergency Department events. Unlike the interview-based
VI-SPDAT, the AHA relies not on self-reported answers to interview questions,
but rather on demographic and county service utilization data stored in the
Allegheny County Data Warehouse (The Allegheny County Department of Human
Services 2018). Features considered by the models are drawn from child welfare
services, county jail, courts, juvenile probation, behavioral and physical
health, previous interactions with homeless services, previous interactions
with assisted housing, demographics (other than race), neighborhood data from
the American Community Survey, and household characteristics.

Each of the three models produces a risk score between 1-20, and the three
scores are then combined into a single AHA score. Figure 1 below summarizes
the combination step. Scores from the three separate models are binned, summed
and then converted to deciles to produce a final AHA score between 1-10, with
10 being the highest risk.

###  2.3 The Alt-AHA Tool

In certain cases, new clients may lack sufficient data to run the AHA tool.
For this reason, ACDHS complements the AHA with Alt-AHA, an alternative risk
assessment based on a subset of VI-SPDAT questions identified by tool
developers as being most predictive of risk. Alt-AHA was validated on the same
outcomes as AHA, and calibrated to ensure that clients scoring 10 on Alt-AHA
have a similar risk of adverse outcomes as those scoring 10 on AHA.

To help guide the use of Alt-AHA, each client on a referral receives a “data
quality indicator.” The indicator is 1 if the Data Warehouse has a unique ID
for the individual that has existed for at least 90 days and 0 otherwise.

CE workers are required to use the Alt-AHA when: (i) the quality indicator is
less than 1; or (ii) someone moved to Allegheny County within the last 90
days; or (iii) someone has not consistently been living in Allegheny County;
or (iv) someone is fleeing domestic violence and thus may have been unable to
utilize services. Alt-AHA is sometimes used outside of these four cases. When
CE workers use the Alt-AHA, they must record their reason for doing so.

An AHA score is always produced, irrespective of whether an Alt-AHA is
conducted. The final score used in prioritizing clients, hereafter referred to
as “posted risk score”, is the greater of the AHA and Alt-AHA scores. The next
section describes how these scores are used as part of the broader intake
process.

###  2.4 Housing Intake Process

Figure 2: DHS Housing Services Intake Process

The full client intake process is illustrated in Figure 2. Clients can access
the Coordinated Entry system for homeless services by contacting Allegheny
Link or through referrals by the Office of Community Services (OCS) Field
Units. For clients identified as homeless via the Current Living Situation
Screening, CE workers then conduct an assessment using AHA or Alt-AHA (prior
to August 2020, VI-SPDAT).

After screening and scoring, CE workers assess clients’ eligibility for
specific housing programs. This entails screening for chronic homeless status,
which can prioritize them for housing (HUD 2015). The combination of (Alt-)AHA
scores and special circumstances (e.g. veteran status, fleeing from domestic
abuse, or disability status) dictates program eligibility. The CE coordinator
then places eligible clients on waitlists for the appropriate housing types.
If the CE cannot reach a client for updates every 45 days from their last
contact, and the client is not enrolled in a street outreach program or
shelter, they are removed from the waitlist.

Once a provider has an available slot and the capacity to serve and find a
client housing, they request a referral from ACDHS off the waitlist. The
housing provider then enrolls clients into their programs, offering case
management services and housing assistance. Importantly, this does not
guarantee housing; clients might not accept the offer due to personal
preferences or circumstances such as hospitalization of incarceration. Despite
the AHA tool’s key role in housing allocation, human factors like maintaining
the wait list and locating clients still play a crucial part in the process,
influencing when clients are referred. However, CE workers’ ability to alter a
client’s waitlist position is limited by strict eligibility rules. This is in
contrast to other algorithm-assisted decision-making settings such as bail or
child maltreatment screening, where human decision-makers have considerably
more discretion.

##  3 Related Literature

Algorithm-assisted decision-making is increasingly used in the public sector,
including in homelessness prevention. Predictive models utilize proxy
outcomes, administrative data, and target specific sub-populations to
efficiently allocate limited resources in crucial settings. Various studies
have used these models to predict outcomes such as high-cost users (Toros and
Flaming 2018), first-time homelessness (Von Wachter et al. 2019), returns to
homelessness (Von Wachter et al. 2019; Kube, Das, and Fowler 2019), shelter
entries (Shinn et al. 2013), and chronic homelessness (Messier, John, and
Malik 2022; VanBerlo et al. 2021). Additional research focuses on specific
groups, predicting risks such as daily sheltering arrangements for homeless
youth (Suchting et al. 2020) and their susceptibility to substance abuse
(Tabar et al. 2020; Dou et al. 2021), as well as assessing homelessness risks
among veterans (Koh et al. 2022; Byrne, Montgomery, and Fargo 2019). The AHA
tool differs from these other approaches by focusing on long-term risks
associated with homelessness, providing algorithmic scores to prioritize
clients on housing waitlists controlled by ACDHS, rather than binary risk
assessments.

Recent studies on predictive tools in housing service allocation reveals
several significant challenges. The first concern is the accuracy of
administrative data in reflecting clients’ past circumstances and urgent needs
(Kuo et al. 2023; Karusala et al. 2019). Research by Karusala et al. (2019)
involving interviews with policymakers and caseworkers shows mixed feelings
about the objectivity and accuracy of the VI-SPDAT. Factors like clients’
trust in caseworkers and their privacy preferences impact what they disclose,
which in turn affects scoring. The scoring also depends on the coordinators’
ability to prompt questions and gauge the effects of a client’s mood and
mental status. Thus, the fairness of these tools may suffer from discrepancies
in what is intended to be measured versus what is actually measured and
assessed in the AI model (Jacobs 2021). Karusala et al. (2019) note that tools
like AHA, which rely on administrative rather than self-reported data, address
mismeasurement attributable to faulty self-report, but remain susceptible to
other biases in data collection.

The second concern focuses is the use of proxy outcomes in prediction, which
may not fully capture the subjective experiences of unhoused individuals Kuo
et al. (2023). Research shows that ML systems might de-prioritize crucial
values like efficiency and privacy that are essential for protecting homeless
individuals (Showkat et al. 2023). Building trust in these systems at provider
and bureaucratic levels is crucial for their support, as noted by Saxena et
al. (2021).

Debate around fairness in housing services also persists. Kube, Das, and
Fowler (2019) highlight a trade-off between fairness and efficiency, where
imposing fairness constraints can increase re-entry rates. Karusala et al.
(2019) note a division between service providers, who prioritize vulnerability
over race, and policymakers, who seek to address racial disparities in housing
access. Some researchers advocate using reparative algorithms to rectify
historical inequities (So et al. 2022). Our study specifically evaluates
racial disparities in the allocation of housing.

ACDHS commissioned an independent equity evaluation (Eticas Research and
Consulting 2020) of the models prior to deployment, which found that the
models operated consistently across racial sub-groups.

##  4 Methodology

###  4.1 Research Questions

In the study, we seek to understand how CE workers interact with the AHA tool
and how the AHA tool impacts racial disparity in housing allocation. More
specifically, we ask three Research Questions (RQ’s):

  1. 1.

How did the prioritization for housing change when the VI-SPDAT was replaced
with the AHA?

  2. 2.

How does AHA impact the racial disparities in housing service enrollment
rates?

  3. 3.

How does the use of the alternative survey assessment tool (Alt-AHA) impact
racial disparities?

We study changes in service rates. An assessment is “served” if it results in
an enrollment with a Permanent Supportive Housing (PSH), Rapid Rehousing (RR),
or Transitional Housing (TH) provider. This means the client can start
receiving comprehensive case management services, such as mental health and
substance use treatment, and housing assistance. As discussed in §2.4,
enrollment does not mean they are assigned to or have moved into housing yet.
We do not distinguish between housing types, as all three provide services
that address housing instability barriers. This study is not a comprehensive
evaluation of the impact of the tool on overall welfare—only on racial
disparities.

###  4.2 Descriptive Analysis and Empirical Model

We conduct descriptive analyses to examine high-level trends, and then more
rigorously study the research questions using logistic regressions to control
for client characteristics. Table 5 of the Appendix provides a summary of key
variables and how they are denoted throughout the paper. We also discuss the
distinction between demographics and housing eligibility characteristics in
Appendix E.

To answer RQ1, we use retrospectively generated AHA scores for the pre- and
post-deployment periods to investigate changes in service rates. We conduct
the following regression on clients who were assessed only via AHA, focusing
on $\beta_{3}$.

$\displaystyle\begin{aligned}
Served_{i,t}&=\beta_{0}+\beta_{1}Post_{i,t}+\beta_{2}Score+\beta_{3}Score*Post_{i,t}\\\
&+\beta_{4}Black_{i,t}+\beta_{5}^{\prime}X+\gamma_{t}+\delta_{t}+\epsilon_{i,t}\end{aligned}$
(1)

$\beta_{3}$ captures the change in association between service decisions and
AHA scores from pre to post. We control for client race ($Black$ indicator
variable), as well as other controls, $X$, which include age, gender, veteran
status, chronic homelessness status, disability status, etc., as shown in
Table 5. Time-fixed effects $\gamma_{t}$ and $\delta_{t}$ represent assessment
year and month, respectively, capture yearly and seasonal variations in
service provision. We use robust standard errors clustered by client ID to
account for multiple assessments per client and for among-client variation.

We study RQ2 with the following specification, using data only on clients
where AHA (not Alt-AHA) was used.

$\displaystyle\begin{aligned}
Served_{i,t}&=\beta_{0}+\beta_{1}Post_{i,t}+\beta_{2}Black_{i,t}+\beta_{3}{Score}_{i,t}\\\
&+\beta_{4}{RiskGroup}_{i,t}+\beta_{5}Black*Post_{i,t}\\\
&+\beta_{6}{RiskGroup}*Post_{i,t}+\beta_{7}{RiskGroup}*Black_{i,t}\\\
&+\beta_{8}{RiskGroup}*Black*Post_{i,t}\\\
&+\beta_{9}^{\prime}X+\gamma_{t}+\delta_{t}+\epsilon_{i,t}\end{aligned}$ (2)

This model allows for heterogeneity in the association between service
decisions, race, and AHA deployment across risk group strata. The risk strata
are defined in Table 5 as low-risk (AHA scores 1-4), medium-risk (scores 5-8),
and high-risk (scores 9-10). We present a simplified short-hand equation here
and include an expanded equation in Appendix D for clarification.

Two coefficients of interest, $\beta_{5}$ and $\beta_{8}$, measure changes in
service rates. $\beta_{5}$ reflects how service rates change for the low-risk
group post-AHA. $\beta_{8}$ shows the additional service rate change for Black
clients in the high-risk group, compared to white clients, pre- and post-AHA
deployment, relative to the low-risk group. Essentially, it shows the impact
of high risk status on racial disparities post-AHA. As a robustness check, we
test alternative specifications in Appendix F, treating AHA scores as
continuous or binary for high-risk. The results are consistent with the main
findings.

For RQ3, we use logistic regression on data from clients assessed with Alt-
AHA, using the following specification where the Alt-AHA score, if higher, is
used as the posted score for housing prioritization.

$\displaystyle\begin{aligned}
Served_{i,t}&=\beta_{0}+\beta_{1}Black_{i,t}+\beta_{2}{PostedScore}_{i,t}\\\
&+\beta_{4}{UseAltScore}+\beta_{5}{QualityIndicator}\\\
&+\beta_{6}^{\prime}X+\gamma_{t}+\delta_{t}+\epsilon_{i,t}\end{aligned}$ (3)

`QualityIndicator` assesses data quality on a 0 to 1 scale. All other
covariates $X$ and time fixed effects remain consistent. The coefficient of
interest, $\beta_{1}$, evaluates potential racial disparities in service rates
for clients prioritized based on Alt-AHA after controlling for other factors.

##  5 Data

We extract housing assessment data from the County’s Homelessness Management
Information System (HMIS) for assessments conducted between January 1, 2018
and March 29, 2022. This data is linked with enrollment data for clients in
PSH, RR, or TH services during the study period.

We retrospectively (re-)run the AHA tool to produce scores for pre- and post-
AHA deployment periods. We also obtain VI-SPDAT scores from the pre-deployment
period.

We exclude assessments with a race other than white or Black/African American,
as they represent only 3.2% of the data. We also exclude 36.1% of the
assessments associated with families or youth household types, focusing
instead on single adult household types. Prioritization of families or youth
households is more complicated and depends on a range of additional criteria
not captured by our data. Lastly, we exclude clients who are enrolled with
programs that do not require AHA scores. The final dataset contains 6,542
assessments, in which 66.1% were from the pre-AHA period. 399 Alt-AHA
assessments were conducted, with 370 using the Alt-AHA scores as their posted
scores.

Table 1 summarizes pre- and post-deployment client characteristics. We see
that the majority of single homeless clients are male, and over 90% report a
disability. Post deployment, the prevalence of chronically homeless clients
rises. When comparing Black and white clients, their assessment scores, ages,
and likelihood of being domestic violence survivors remain similar pre- and
post-deployment. However, they diverge in key eligibility criteria: white
clients exhibit higher rates of chronic homelessness and disability, whereas
Black clients are more likely to be veterans.

Before deployment, Black clients have a lower average VI-SPDAT score (7.5)
than white clients (8.4) on a 1-16 scale. Post-deployment, Black clients
average a slightly higher AHA score (7.1) compared to white clients (6.6) on a
1-10 scale. White clients are twice (24.4%) as likely to be assessed with Alt-
AHA compared to Black clients (11.8%), and to have those scores used for
prioritization (20.7% vs. 8.0% for Black clients). This significant difference
may be linked to consistently higher data quality indicators for Black
clients, with scores of 0.96 vs. 0.87 pre-deployment and 0.91 vs. 0.82 post-
deployment.

Finally, white clients were served at a higher rate than Black clients before
AHA deployment, 17.6% compared to 14.5%. The disparity persists post-
deployment.

Table 1: Summary Statistics by Race and Deployment Status

| Pre-deployment | Post-deployment
---|---|---
Statistic | Black/AA | White | Black/AA | White
% Female | 33.13 | 32.14 | 35.06 | 37.29
| (47.08) | (46.71) | (47.74) | (48.38)
% Male | 59.88 | 62.09 | 64.86 | 62.62
| (49.03) | (48.53) | (47.76) | (48.40)
% Gender Unknown | 6.99 | 5.78 | 0.09 | 0.09
| (25.50) | (23.33) | (2.96) | (3.05)
Age | 46.30 | 45.72 | 46.07 | 43.94
| (12.29) | (11.95) | (12.19) | (11.67)
% Chronic Homeless | 25.23 | 29.81 | 35.41 | 38.59
| (43.44) | (45.75) | (47.84) | (48.70)
% Disability | 91.18 | 95.16 | 90.97 | 93.88
| (28.37) | (21.47) | (28.67) | (23.99)
% Veteran | 9.10 | 7.78 | 7.80 | 6.40
| (28.77) | (26.79) | (26.83) | (24.49)
% Domestic Violence Survivor | 11.53 | 11.36 | 13.50 | 13.64
| (31.95) | (31.75) | (34.18) | (34.33)
AHA Score | 6.50 | 6.29 | 7.11 | 6.60
| (2.67) | (2.74) | (2.46) | (2.77)
VI-SPDAT Score | 7.46 | 8.41 | - | -
| (2.50) | (3.14) | - | -
Alt-AHA Score | - | - | 5.07 | 6.14
| - | - | (2.72) | (2.53)
% Conducted Alt-AHA | - | - | 11.83 | 24.49
| - | - | (32.31) | (43.02)
% Use Alt Score | - | - | 7.98 | 20.69
| - | - | (27.10) | (40.52)
Quality Indicator | 0.96 | 0.87 | 0.91 | 0.82
| (0.20) | (0.32) | (0.28) | (0.38)
Served | 14.52 | 17.56 | 19.54 | 23.28
| (35.24) | (38.06) | (39.67) | (42.28)
N | 2,176 | 2,147 | 1,141 | 1,078
Standard deviation in parentheses

##  6 Descriptive Analysis

We now compare risk scores and service rates from VI-SPDAT, AHA, and Alt-AHA
to gain high-level insight relating to our research questions. In Appendix A
we provide a brief analysis of concurrent events, particularly the impact of
COVID-19 and lockdown orders, surrounding the AHA deployment. While we note
some variation in service rates by race, these changes are not statistically
significant.

###  6.1 Score comparisons

#### AHA vs. VI-SPDAT Scores.

We begin by comparing the AHA and VI-SPDAT scores. Because the VI-SPDAT was no
longer conducted after the AHA was deployed, this comparison is based on data
during the pre-deployment period. VI-SPDAT scores range from 1-16 where 1 is
the lowest and 16 is the highest risk, and AHA scores range from 1-10. The
left panel on Figure 3 shows that there is a weak positive correlation between
the retrospectively generated AHA scores and the unconverted VI-SPDAT scores
(Spearmen rank order coefficient = 0.218, $P<0.0001$).

To facilitate a more direct comparison of the scores for the remainder of the
analysis, we bin the VI-SPDAT scores into a 1-10 range. Scores 1-4 are mapped
to 1, 13-16 are mapped to 10, and values in-between are mapped in a one-to-one
correspondence to 2:9. This mapping is monotonic, and is chosen to ensure that
there are comparable numbers of clients in the lowest and highest score bins
for both the AHA and (post-mapping) VI-SPDAT. Figure 3 (right) shows a heatmap
of scores after mapping. After conversion, the AHA and VI-SPDAT Scores remain
significantly weakly positively correlated (Spearman rank order coefficient =
0.217, $P<0.000$)

Figure 3: AHA vs. VI-SPDAT Scores Prior to AHA Implementation. Original VI-
SPDAT scores (left), converted VI-SPDAT scores (right).

#### AHA vs. Alt-AHA Scores.

Figure 4 compares AHA scores to Alt-AHA scores for clients who were assessed
with Alt-AHA in the post-deployment period. We see that clients whose initial
AHA scores were low (1-3), typically received much higher scores when assessed
with Alt-AHA. As a business rule, the higher of the two scores will be used as
the score (“posted score”) for prioritization. Alt-AHA and AHA scores do not
appear correlated (Spearman rank order coefficient $=-0.02$, $P=0.64$).

Figure 4: AHA vs. Alt-AHA Scores for clients who were assessed with Alt AHA.
Clients with low AHA scores (1-3) typically received higher Alt-AHA scores.

Figure 5 shows a breakdown of Alt-AHA use by race. We observe that white
clients are more likely than Black clients to be assessed via Alt-AHA (left
panel), and also more likely to then have their Alt-AHA scores used (right
panel). These differences are at least partly attributed to lower quality
indicators for white clients, as shown in Table 1. CE workers provide reasons
for disagreeing with AHA scores, which we discuss in Appendix B. There we find
that white clients are more often not residing consistently in Allegheny
County, have lower data quality indicators, and have lower service utilization
than Black clients.

Figure 5: Percent of Clients Assessed by Alt-AHA by Race (left); and Percent
of Clients Assessed by Alt-AHA whose Alt-AHA Scores were used for
Prioritization by Race (right). White clients are twice as likely to be
assessed via Alt-AHA than Black clients. Of those assessed by Alt-AHA, white
clients’ Alt-AHA scores are also more likely to be used for prioritization.

###  6.2 Score Distribution by Race

Figure 6 shows the (frequency) distribution of pre-deployment VI-SPDAT and AHA
scores, and post-deployment AHA and Alt-AHA scores222A cumulative density
version of these plots is shown in Figure 11 of Appendix C.. We present
within-racial-group frequencies instead of counts so that pre-vs-post
comparisons are not confounded by variation in the number of clients being
assessed over time.

Figure 6: AHA and VI-SPDAT Score Distributions by Deployment Period and Race,
Normalized by Race. Normalization within each racial group shows the
proportion of individuals receiving each risk score. Pre-AHA deployment, VI-
SPDAT classified more Black clients as low-risk and more white clients as
high-risk, while AHA would have classified more white clients as low-risk.
Post-AHA deployment, the score distribution shifted towards higher risk
levels. Both AHA and Alt-AHA scored Black and white clients at comparable
rates.

Comparing the pre-deployment plots, we see that VI-SPDAT scores were lower for
Black clients than white clients, with white clients being more likely to
receive VI-SPDAT scores in the 8-10 range. Conditional on VI-SPDAT score,
Figure 8 shows that Black and white clients were equally likely to receive
housing services. However, at least in part because VI-SPDAT scores tended to
be lower for Black clients, overall service rates were significantly lower for
Black clients than white clients pre-AHA (see Figure 7).

Unlike the VI-SPDAT, in both the pre- and post-deployment period, Black
clients were approximately equally likely to receive AHA scores in the 8-10
range. The third panel of Figure 6 shows the subsample where AHA scores
determined prioritization—this combines clients assessed only via AHA and
those assessed via Alt-AHA but whose AHA score was higher. Here, Black and
white clients exhibit similar score distributions. To the extent that
differences in prioritization score distributions drive disparities in service
rates, this analysis indicates that disparities might decrease once AHA is
introduced.

Second, transitioning from VI-SPDAT to AHA, we observe a significant shift in
the score distribution, with a larger percentage of clients across both races
now classified as higher risk, whereas previously, most fell into the low to
medium risk category. Post-AHA deployment, there is an increased
representation of Black clients in the medium and high-risk categories,
compared to the VI-SPDAT.

Finally, in the case of clients assessed via Alt-AHA whose Alt-AHA scores were
used for prioritization (where the posted scores are identical to their Alt-
AHA scores), the distribution shows a slightly higher percentage of Black
clients in the lower risk range and a slightly lower percentage in the high
risk range compared to white clients. Given the small sample size, the
difference between the distributions is unlikely to be statistically
significant.

In summary, we observe a notable disparity under VI-SPDAT, with a higher
percentage of Black clients categorized as lower risk and white clients as
higher risk. Post-deployment, both Black and white clients receive similar
scores under AHA, with Black clients showing an increased presence in the
medium to high-risk categories compared to their earlier VI-SPDAT scores.
White clients tend to receive higher risk scores when assessed by Alt-AHA.

###  6.3 Service Rates by Race

Figure 7 shows that, pre-AHA, white clients were served at a significantly
higher rate than Black clients ($t\text{-statistic}=-2.72$, P-value$=0.007$).
Post-deployment, the service rates increased for both groups, but white
clients continued to be served at a significantly higher rate than Black
clients ($t$-statistic$=-2.15$, P-value$=0.032$). When we break down post-
deployment assessments by whether the Alt-AHA scores were used, we find that
the two groups are served at similar rate when AHA scores were used. The
racial disparity in service rates appears to be driven mainly by the sub-group
of assessments where an Alt-AHA was used. For clients assessed via Alt-AHA, on
aggregate, there is a statistically significantly difference ($t$-statistic:
-2.72, P-value: 0.007) between Black and white clients, with white clients
(30.0%) about twice more likely to be enrolled with a provider than Black
clients (15.4%). However, it is important to note that the sample size for
Alt-AHA is small. The aggregate comparison does not account for risk scores or
other factors affecting client eligibility, such as veteran status and chronic
homelessness. Additionally, white clients generally have lower data quality
when assessed using Alt-AHA. We will revisit this more rigorously in § 7.3.

Figure 7: Percent of Assessments Served by a Provider by Race Pre- and Post-
AHA deployment. Pre-deployment, providers served white clients at a higher
rate than Black clients. Post-deployment, service rates increased for both,
but white clients continued to be served at higher rates. AHA-assessed clients
are served at similar rates by race. White Alt-AHA assessed clients are served
at significantly higher rates.

Figure 8 shows service rates conditioned on AHA and VI-SPDAT scores. We note
that in the pre-deployment period, VI-SPDAT scores strongly correlated with
the likelihood of enrolling with a provider. In contrast, conditioned on the
retroactively calculated AHA risk scores, the service rates appear
uncorrelated with AHA scores. Post-AHA deployment, we observe service rates
aligned more closely with AHA scores, with higher scores more likely to be
served.

Figure 8: Percent of Assessments Served by Race for Pre- and Post-deployment
Periods, Conditioned on Scores.

Furthermore, conditioned on VI-SPDAT scores, we observe no racial disparities
in service rates during the pre-deployment period. However, as mentioned
previously, this lack of association does not imply an absence of racial
disparity, since VI-SPDAT tended to underestimate the risk of Black clients.
If we consider the proxy harms that AHA measures, when CE workers assessed
risks using VI-SPDAT without access to AHA scores in the pre-deployment
period, white clients where served at higher rates than Black clients among
those whose AHA risk score was 9 or 10, as shown in the first panel to the
left in Figure 8. Post-deployment, the introduction of AHA scores appears to
have closed this disparity for clients in the high-risk category. We will test
this hypothesis more rigorously in § 7.2.

When examining the third and fourth panels in Figure 8, which depict service
rates for AHA and Alt-AHA assessments, respectively, we observe no significant
gap in service for assessments conducted via AHA. While there is a significant
overall difference between Black and white clients assessed via Alt-AHA, it
disappears when conditioned on Alt-AHA scores—likely due to small sample
sizes.

###  6.4 Conclusions of Descriptive Analysis

Addressing RQ1, prior to AHA deployment, service rates are highly associated
with VI-SPDAT scores, and largely uncorrelated with AHA scores. However, post-
AHA deployment, service rates begin to correlate strongly with AHA scores,
especially for clients assessed exclusively with AHA.

Addressing RQ2, both pre- and post-deployment, white clients were served at a
significantly higher rate than Black clients. While there is no overall
disparity in service rates in clients assessed by AHA only, service rates for
clients assessed by Alt-AHA are higher for white clients than Black clients.
There is no racial disparity conditioned on VI-SPDAT scores pre-AHA. Yet,
retrospective AHA scoring identify two key service rate disparities. (1) VI-
SPDAT tended to assign higher risk scores to white clients compared to Black
clients. By contrast, AHA scores are more comparable across groups. (2) Under
VI-SPDAT, white clients were served at significantly higher rates than Black
clients at the same retrospective AHA risk scores, particularly in the high-
risk category. The deployment of AHA reduced the service rate disparity among
high-risk clients.

Finally, for RQ3, we find that post-AHA deployment, white clients were more
likely to be assessed via Alt-AHA, their Alt-AHA scores were more likely to be
higher than their AHA scores compared to Black clients, and they were more
likely to be served. However, this may be due to differences in data quality,
demographics, and eligibility criteria.

##  7 Quantitative Results

In this section, we further interrogate the findings from the descriptive
analysis using logistic regressions to control for additional explanatory
variables.

###  7.1 RQ1

Because we retrospectively (re-)calculated both pre- and post-deployment AHA
scores, we can examine whether the switch from VI-SPDAT to AHA resulted in
service decisions becoming more closely aligned with AHA scores.

Table 2 shows the results of testing this hypothesis using regression
specification (1). In the table columns we progressively control for
demographics and eligibility criteria, and then also time fixed effects.

Table 2: Service Rate by AHA Implementation

| (1) | (2) | (3)
---|---|---|---
Post | -2.0128∗∗∗ | -2.0106∗∗∗ | -2.2432∗∗∗
| (0.3298) | (0.4166) | (0.4601)
AHA Score | 0.0494∗∗ | -0.0167 | -0.0142
| (0.0152) | (0.0188) | (0.0189)
AHA Score * Post | 0.2845∗∗∗ | 0.3061∗∗∗ | 0.3052∗∗∗
| (0.0394) | (0.0444) | (0.0449)
Black | -0.1875∗∗ | -0.1527 | -0.1586∗
| (0.0659) | (0.0791) | (0.0797)
Demographics & Elig. Crit. |  | $\checkmark$ | $\checkmark$
Quarter FE |  |  | $\checkmark$
Yearly FE |  |  | $\checkmark$
N | 6,143 | 6,143 | 6,143
Pseudo R-squared | 0.0270 | 0.2792 | 0.2818
Standard errors in parentheses
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

Comparing the `AHA Score` coefficient across columns (1) and (2), we see that
once eligibility criteria are controlled for, there’s no statistically
significant association between decisions and the AHA score pre-deployment. On
the other hand, the coefficient of `AHA Score * Post` is consistently
statistically significant across columns, indicating that service rates became
significantly positively associated AHA scores post-deployment. Across all
columns, the coefficients on `Black` consistently show that Black clients are
statistically significantly less likely to be served compared to white
clients.

###  7.2 RQ2

As discussed in §6.2, when CE workers relied solely on VI-SPDAT without access
to AHA scores, they tended to serve white clients at a higher rate within the
AHA score range 9 and above. We further examine the change in service rate by
risk groups using the Equation 2 allowing for heterogeneous treatment effect
proposed in § 4.

Table 3: Service Rate by Race Risk Group, and AHA Implementation

| (1) | (2) | (3)
---|---|---|---
Black | 0.0868 | 0.1461 | 0.1569
| (0.1568) | (0.1910) | (0.1914)
Post | -0.6282 | -0.6564 | -0.8331
| (0.3572) | (0.4607) | (0.4894)
AHA Score | 0.1220∗∗ | 0.0572 | 0.0612
| (0.0383) | (0.0456) | (0.0458)
High Risk | -0.2178 | -0.2574 | -0.2478
| (0.2976) | (0.3498) | (0.3510)
Medium Risk | -0.1598 | -0.0639 | -0.0524
| (0.2150) | (0.2468) | (0.2479)
High Risk * Black | -0.5237∗ | -0.4661 | -0.5095
| (0.2149) | (0.2612) | (0.2627)
Medium Risk * Black | -0.3707 | -0.4163 | -0.4240
| (0.2030) | (0.2443) | (0.2444)
High Risk * Post | 1.1161∗∗ | 1.4088∗∗ | 1.3372∗∗
| (0.3877) | (0.4557) | (0.4617)
Medium Risk * Post | 0.5399 | 0.7230 | 0.6542
| (0.3930) | (0.4570) | (0.4635)
Black * Post | -0.4639 | -0.5409 | -0.5895
| (0.5401) | (0.6375) | (0.6426)
High Risk * Black * Post | 0.8694 | 0.8474 | 0.9304
| (0.5815) | (0.6876) | (0.6929)
Medium Risk * Black * Post | 0.5988 | 0.6240 | 0.6547
| (0.5867) | (0.6934) | (0.6982)
Demographics & Elig. Crit. |  | $\checkmark$ | $\checkmark$
Quarter FE |  |  | $\checkmark$
Yearly FE |  |  | $\checkmark$
N | 6,143 | 6,143 | 6,143
Pseudo R-squared | 0.0262 | 0.2781 | 0.2808
Standard errors in parentheses
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

We find that that the coefficients of `High Risk * Post` are consistently
statistically significant across the specifications, indicating that high risk
white clients are more likely to be served post-AHA deployment compared to
pre.

The coefficients of `High Risk * Black * Post` and `Medium Risk * Black *
Post` are not statistically significant for either specification that controls
for demographics and eligibility criteria. We therefore do not have sufficient
evidence to conclude that the change in service rates post-deployment is
different for Black clients than for white clients for any of the risk strata.

###  7.3 RQ3

As for the use of Alt-AHA, across all models, higher posted scores and quality
indicator scores are consistently associated with a higher likelihood of
receiving services through Alt-AHA usage. Conversely, the use of Alt-AHA alone
does not significantly impact service likelihood.

When considering the effect of race and posted score (the higher of Alt-AHA or
AHA scores that is used for prioritization for this sub-population) in column
(1) and (2) and (3) on service rates, Black clients are significantly less
likely to receive services through Alt-AHA compared to white clients, even
after adjusting for differences in data quality (in (3)). Once demographic,
eligibility and time fixed effects are added to the regression, the race
coefficient is no longer statistically significant. However, this is more due
to an increase in the standard error of the coefficient estimate than a clear
decrease in the magnitude of the coefficient. The evidence of higher service
rates for white clients under Alt-AHA is therefore borderline.

Table 4: Alt-AHA Service Rate

| (1) | (2) | (3) | (4) | (5)
---|---|---|---|---|---
Black | -0.787∗∗ | -0.613∗ | -0.688∗ | -0.572 | -0.658
| (0.280) | (0.312) | (0.321) | (0.370) | (0.384)
Posted Score |  | 0.484∗∗∗ | 0.504∗∗∗ | 0.500∗∗∗ | 0.538∗∗∗
|  | (0.0715) | (0.079) | (0.099) | (0.117)
Use Alt AHA |  |  | -0.078 | -0.300 | -0.447
|  |  | (0.399) | (0.519) | (0.534)
Quality Indicator |  |  | 1.039∗∗∗ | 0.844∗ | 0.771∗
|  |  | (0.284) | (0.356) | (0.361)
Demographics & Elig Crit |  |  |  | $\checkmark$ | $\checkmark$
Quarter FE |  |  |  |  | $\checkmark$
Yearly FE |  |  |  |  | $\checkmark$
N | 399 | 399 | 399 | 371 | 371
Pseudo R-squared | 0.0201 | 0.1799 | 0.2119 | 0.4092 | 0.4244
Standard errors in parentheses |
disability != 1 predicts failure perfectly; disability omitted and 28 obs not used |
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$ |

###  7.4 Conclusion of Quantitative Analysis

In summary, our quantitative analysis confirms, as we would anticipate, that
post-AHA deployment, service decisions become more closely aligned with AHA
scores. Overall, we do not find evidence that the AHA deployment significantly
bridges the disparity in service rates between Black and white clients. The
post-deployment change in service rates for medium and high risk Black clients
is similar to that observed for white clients. Lastly, while there is
borderline evidence that white clients are more likely to be served when Alt-
AHA is used, ultimately the sample size underpinning that analysis is too
small to draw strong conclusions.

##  8 Limitations

Our study, based on administrative data, has some limitations. We have tried
to model the use of AHA risk scores as accurately as possible within real
decision-making contexts. However, the complexity of the process from initial
assessment to the final service enrollment means we must make some assumptions
when interpreting our results.

First, it is essential to acknowledge the possibility that the effects
identified in our study are in part attributable to changes during the COVID
period related to housing policy and client composition that influenced
service allocations. We further discuss the observed effects of the stay-at-
home order and other COVID-era impacts in Appendix §A.

In order to conclude that the effects we identified are attributable to the
deployment of AHA rather than other unmeasured factors, we need to assume that
there are no “unobservables” that both (i) changed at the same time as AHA was
deployed, and (ii) were relied upon by CE workers in their service decisions.
It could be problematic for our study design if, for instance, some
information gathered during the assessment process but not reflected in the
administrative data influenced CE worker decisions. We were not able to
identify alternative observational study designs that would not be susceptible
to potential unobserved confounding.

##  9 Discussion

We present the first evaluation of the impact of replacing the VI-SPDAT with
the Allegheny Housing Assessment on housing service outcomes. Specifically, we
examine how CE decisions change post-deployment, and the effects of those
changes on racial disparities in service rates. While earlier studies of AHA
considered the predictive performance and group-fairness properties of the
tool’s constituent models, they relied on pre-deployment data and did not
study the tool’s influence on decisions.

We find that white clients were significantly more likely than Black clients
to receive high VI-SPDAT scores, but were approximately equally likely to
receive high AHA scores. Later analyses do not, however, enable us to conclude
that the shift to AHA-based prioritization significantly reduced the racial
disparities in service decisions present under the VI-SPDAT policies. Pre-AHA,
white clients were served at higher rates than Black clients, both overall and
conditional on AHA score. We find that racial disparities largely persisted
post-AHA, particularly in the subset of clients assessed with Alt-AHA. Our
analysis indicates that the persistent disparities are at least in part
attributable to differences in data quality, and in differences in eligibility
criteria across groups.

This study has several policy implications for CoC providers considering a
switch from the VI-SPDAT to predictive models like the AHA tool or other
survey-based assessment such as the Alt-AHA. Our results show that predictive
models like AHA can reduce or altogether eliminate differences in the
prioritization score distributions. This, however, does not guarantee that
overall disparities in service decisions will be reduced as a result. Whereas
we found no significant post-deployment disparity in service rates among
clients assessed with AHA, disparities among clients assessed with Alt-AHA
were at least as large as those existing pre-deployment. This finding
underscores difficulties with anticipating the impact of a predictive risk
model deployment in settings where the model cannot be run due to poor data
quality in a substantial number of cases.

As discussed extensively in the literature review, there is an important
concern about whether administrative data can accurately capture clients’ past
and current needs (Kuo et al. 2023). Efforts to improve data accuracy and
enhance cultural competence among service providers are critical for
addressing disparities effectively. Policymakers must consider the strengths
and weaknesses of risk assessments derived from administrative data and
predictive modeling. Effectively integrating AI with CE workers’ expertise
could be pivotal in reducing racial disparities in housing service provision.

Future studies should further explore how CE workers use Alt-AHA or deviate
from the standard AHA procedure to advocate for their clients. These findings
would be invaluable in offering a clearer understanding of the additional risk
factors captured during the interview process, and shaping the future
iteration of the algorithm. This expanded investigation would significantly
contribute to refining and improving the decision-making processes in social
services.

## Researcher Positionality Statement

All authors have previously conducted evaluations of algorithmic tools in
either child welfare or housing services deployed by the ACDHS for academic
research. The second author has interned part-time at the ACDHS, gaining
firsthand experience with their operational processes and tools. The last
author led the development of the AHA tool, and the third author provided
feedback on the methodology used in model development and validation. As a
result, the research team collectively has a deep understanding of the design,
development, validation and deployment process surrounding the AHA tool.

We recognize that this experience and familiarity also present the potential
for conflicts of interest. In conducting the research described in this paper
we maintained a firm commitment to academic integrity and transparency. As per
the terms of our data use agreement, ACDHS was provided with the submission
version of this manuscript and given the opportunity to review and provide
feedback. The data use agreement does not enable ACDHS to block the
publication. Through this review process we received helpful feedback that
helped improve the clarity of the manuscript, led to a better-motivated model
specification for RQ1, and also resulted in several robustness checks now
presented in the appendix.

Our collective expertise includes backgrounds in public policy, statistics,
and health economics, enabling a multidisciplinary approach to this
evaluation. We are motivated by a shared goal of improving the efficacy and
fairness of algorithmic tools in social services. Our work is informed not
only through collaborations with government agencies, but also through
engagement with stakeholders most likely to be impacted by the technologies.

## References

  * Allegheny County Analytics (2020) Allegheny County Analytics. 2020.  AHA DHS Response to Eticas.  Allegheny County Analytics Website.
  * Allegheny County Execuitve Office (2023) Allegheny County Execuitve Office. 2023.  Allegheny County Continuum of Care.  _DHS Advisory Boards_.
  * Brown et al. (2018) Brown, M.; Cummings, C.; Lyons, J.; Carrión, A.; and Watson, D. P. 2018.  Reliability and validity of the Vulnerability Index-Service Prioritization Decision Assistance Tool (VI-SPDAT) in real-world implementation.  _Journal of Social Distress and the Homeless_ , 27(2): 110–117.
  * Byrne, Montgomery, and Fargo (2019) Byrne, T.; Montgomery, A. E.; and Fargo, J. D. 2019.  Predictive modeling of housing instability and homelessness in the Veterans Health Administration.  _Health Services Research_ , 54(1): 75–85.
  * CBS Pittsburgh (2020) CBS Pittsburgh. 2020.  Allegheny County Health Department Lifts Coronavirus Order; State Orders Still Stand.
  * Community Solutions (2016) Community Solutions. 2016.  Vulnerability Index-Service Prioritization Decision Assistance Tool (VI-SPDAT).  https://pehgc.org/wp-content/uploads/2016/09/VI-SPDAT-v2.01-Single-US-Fillable.pdf.
  * Cronley (2022) Cronley, C. 2022.  Invisible intersectionality in measuring vulnerability among individuals experiencing homelessness – critically appraising the VI-SPDAT.  _Journal of Social Distress and Homelessness_ , 31(1): 23–33.
  * Day (2015) Day, J. E. 2015.  _Racial disparities and place effects for housing instability and homelessness in the US_.  The University of Utah.
  * Deto (2020) Deto, R. 2020.  Stay-at-Home Order Issued for Allegheny County Due to Coronavirus Spread.  Accessed: 11-28-2023.
  * Dou et al. (2021) Dou, Z.-Y.; Barman-Adhikari, A.; Fang, F.; and Yadav, A. 2021.  Harnessing Social Media to Identify Homeless Youth At-Risk of Substance Use.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(17): 14748–14756.
  * Eticas Research and Consulting (2020) Eticas Research and Consulting. 2020.  Eticas Assessment.  Website.
  * Evans, Philips, and Ruffini (2019) Evans, W. N.; Philips, D. C.; and Ruffini, K. J. 2019.  Reducing and preventing homelessness: A review of the evidence and charting a research agenda.
  * Fusaro, Levy, and Shaefer (2018) Fusaro, V. A.; Levy, H. G.; and Shaefer, H. L. 2018.  Racial and ethnic disparities in the lifetime prevalence of homelessness in the United States.  _Demography_ , 55(6): 2119–2128.
  * HUD (2015) HUD. 2015.  Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH): Defining Chronically Homeless Final Rule.  _HUD Exchange_.
  * Institute for Research on Poverty (2021) Institute for Research on Poverty. 2021.  Understanding the Role of Caseworker Bias in Racial and Ethnic Inequities in Human Services.  https://www.irp.wisc.edu/wp/wp-content/uploads/2021/09/Memo-2-Caseworker-Bias-09272021pdf.pdf.  Accessed: December 7, 2023.
  * Jacobs (2021) Jacobs, A. Z. 2021.  Measurement as governance in and for responsible AI.  _arXiv preprint arXiv:2109.05658_.
  * Karusala et al. (2019) Karusala, N.; Wilson, J.; Vayanos, P.; and Rice, E. 2019.  Street-Level Realities of Data Practices in Homeless Services Provision.  _Proc. ACM Hum.-Comput. Interact._ , 3(CSCW).
  * Koh et al. (2022) Koh, K. A.; Montgomery, A. E.; O’Brien, R. W.; Kennedy, C. J.; Luedtke, A.; Sampson, N. A.; Gildea, S. M.; Hwang, I.; King, A. J.; Petriceks, A. H.; Petukhova, M. V.; Stein, M. B.; Ursano, R. J.; and Kessler, R. C. 2022.  Predicting Homelessness Among U.S. Army Soldiers No Longer on Active Duty.  _American Journal of Preventive Medicine_ , 63(1): 13–23.
  * Kube, Das, and Fowler (2019) Kube, A.; Das, S.; and Fowler, P. J. 2019.  Allocating Interventions Based on Predicted Outcomes: A Case Study on Homelessness Services.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01): 622–629.
  * Kuo et al. (2023) Kuo, T.-S.; Shen, H.; Geum, J.; Jones, N.; Hong, J. I.; Zhu, H.; and Holstein, K. 2023.  Understanding Frontline Workers’ and Unhoused Individuals’ Perspectives on AI Used in Homeless Services.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI ’23. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450394215.
  * Messier, John, and Malik (2022) Messier, G.; John, C.; and Malik, A. 2022.  Predicting chronic homelessness: The importance of comparing algorithms using client histories.  _Journal of Technology in Human Services_ , 40(2): 122–133.
  * National Alliance to End Homelessness (2022) National Alliance to End Homelessness, H. R. I. 2022.  NextGenTools VISPDAT Brief.  https://endhomelessness.org/wp-content/uploads/2022/08/NextGenTools˙VISPDATBrief˙08-30-22.pdf.  Accessed: December 7, 2023.
  * Petry et al. (2021) Petry, L.; Hill, C.; Vayanos, P.; Rice, E.; Hsu, H.-T.; and Morton, M. 2021.  Associations Between the Vulnerability Index-Service Prioritization Decision Assistance Tool and Returns to Homelessness Among Single Adults in the United States.  _Cityscape_ , 23(2): pp. 293–324.
  * Saxena et al. (2021) Saxena, D.; Badillo-Urquiola, K.; Wisniewski, P. J.; and Guha, S. 2021.  A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare.
  * Shinn et al. (2013) Shinn, M.; Greer, A. L.; Bainbridge, J.; Kwon, J.; and Zuiderveen, S. 2013.  Efficient targeting of homelessness prevention services for families.  _American journal of public health_ , 103(S2): S324–S330.
  * Showkat et al. (2023) Showkat, D.; Smith, A. D. R.; Lingqing, W.; and To, A. 2023.  “Who is the Right Homeless Client?”: Values in Algorithmic Homelessness Service Provision and Machine Learning Research.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI ’23. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450394215.
  * So et al. (2022) So, W.; Lohia, P.; Pimplikar, R.; Hosoi, A.; and D’Ignazio, C. 2022.  Beyond Fairness: Reparative Algorithms to Address Historical Injustices of Housing Discrimination in the US.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, 988–1004. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450393522.
  * Suchting et al. (2020) Suchting, R.; Businelle, M. S.; Hwang, S. W.; Padhye, N. S.; Yang, Y.; and Santa Maria, D. M. 2020.  Predicting Daily Sheltering Arrangements among Youth Experiencing Homelessness Using Diary Measurements Collected by Ecological Momentary Assessment.  _International Journal of Environmental Research and Public Health_ , 17(18).
  * Tabar et al. (2020) Tabar, M.; Park, H.; Winkler, S.; Lee, D.; Barman-Adhikari, A.; and Yadav, A. 2020.  Identifying Homeless Youth At-Risk of Substance Use Disorder: Data-Driven Insights for Policymakers.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, 3092–3100. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450379984.
  * The Allegheny County Department of Human Services (2018) The Allegheny County Department of Human Services. 2018.  Allegheny County Data Warehouse.  Published on July 2018.
  * Toros and Flaming (2018) Toros, H.; and Flaming, D. 2018.  Prioritizing Homeless Assistance Using Predictive Algorithms: An Evidence-Based Approach.  _Cityscape_ , 20(1): 117–146.
  * U.S. Department of Housing and Urban Development (2017) U.S. Department of Housing and Urban Development. 2017.  Notice CPD-17-01: Notice Establishing Additional Requirements for a Continuum of Care Centralized or Coordinated Assessment System.  https://www.hud.gov/sites/documents/17-01CPDN.PDF.
  * U.S. Department of Housing and Urban Development (2022) U.S. Department of Housing and Urban Development. 2022.  The 2022 Annual Homeless Assessment Report (AHAR) to Congress: Part 1.  https://www.huduser.gov/portal/sites/default/files/pdf/2022-AHAR-Part-1.pdf.
  * Vaithianathan and Kithulgoda (2020a) Vaithianathan, R.; and Kithulgoda, C. I. 2020a.  Using Predictive Risk Modeling to Prioritize Services for People Experiencing Homelessness in Allegheny County.
  * Vaithianathan and Kithulgoda (2020b) Vaithianathan, R.; and Kithulgoda, C. I. 2020b.  Using Predictive Risk Modeling to Prioritize Services for People Experiencing Homelessness in Allegheny County: Methodology Paper for the Allegheny Housing Assessment.  Technical report, Allegheny County Analytics.
  * VanBerlo et al. (2021) VanBerlo, B.; Ross, M. A.; Rivard, J.; and Booker, R. 2021.  Interpretable machine learning approaches to prediction of chronic homelessness.  _Engineering Applications of Artificial Intelligence_ , 102: 104243.
  * Von Wachter et al. (2019) Von Wachter, T.; Bertrand, M.; Pollack, H.; Rountree, J.; and Blackwell, B. 2019.  Predicting and preventing homelessness in Los Angeles.  _California Policy Lab and University of Chicago Poverty Lab_.

##  Appendix A Impact of Concurrent Events During Tool Launch on Racial
Disparity

Figure 9 shows the service rate for each quarter by race over our analysis
period. We highlight three events within our analysis period that could affect
service rates and racial disparity: the issuance of the ”Stay at Home Order”
by Pennsylvania and the state’s Department of Health on March 23rd, 2020 (Deto
2020), the lift of the order in Allegheny County on September 24th, 2020 (CBS
Pittsburgh 2020), and one year post the start of the order. These dates are
denoted by the purple, green, and yellow dashed lines, respectively, on Figure
9. In response to the order and HUD COVID-requirements, ACDHS had to swiftly
modify their programs, specifically reducing the number of unhoused clients.
During the implementation of the order, we observe increasing service rates
for both Black and white clients, aligned with ACDHS’s change in business
practice. After the AHA deployment, followed shortly by the lift of the order,
we observe a short period of fall in the white-Black disparity - but the
disparity grew again in favor of white clients one year post after the stay-
at-home order. However, it’s important to note that these differences are
unconditioned on risk levels and also lack statistical significance, a point
we assess more rigorously in § 7.2.

Figure 9: Service Rate by Race Overtime by Quarter. We mark three important
events in our analysis period that prompted ACDHS to adjust their programs and
could potentially affect racial disparity in service allocations:
Pennsylvania’s ”Stay at Home Order” issued by the state’s Department of Health
on March 23rd, 2020, the lift of the order in Allegheny County on September
24th, 2020, and one year post the start of the order. Despite the order’s
implementation, we observe no significant statistical difference in service
rates between Black and white clients over time.

##  Appendix B Reasons for Using Alt-AHA

In Figure 10 we show the reasons for why CE workers disagreed with the AHA
scores and decided to conduct Alt-AHA instead. We show the counts of the
reasons by race and the percentages of the reasons within each race
respectively in the top and bottom graphs. We see that CE workers disagreed
with AHA scores for white clients mostly because they do not live consistently
in Allegheny county, have poor quality indicators and have low service
utilization. CE workers disagreed with AHA scores for Black clients mostly
because they do not live consistently in Allegheny county, have poor quality
indicators and did not have client ID (MCI) in the system at the time of
contact.

Figure 10: Scores Disagree Reasons by Race (Counts and Percentages)

##  Appendix C Score Distribution

In Figure 11, we show the percentage of assessments above each scores by race.
All assessments has a score one or above, therefore we start with 100% for
both white and Black clients. The size of the bars diminishes as there are
smaller percentages of higher risk scores. We can interpret the graphs as
showing the percentage of clients above a score threshold, where the higher
the score the more vulnerable and therefore more eligible for services. We see
that both pre- and post-AHA deployment, AHA generates more comparable scores
for both racial groups. In contrast, at every scores, there are consistently
higher percentages of white clients than Black clients according to VI-SPDAT.
With similar service rates under VI-SPDAT, this highlights the scoring
mechanism as a potential source of disparity as there are always a higher
percentage of white clients above every score thresholds.

Figure 11: Cumulative Distribution of AHA and VI-SPDAT Scores by Deployment
Phase and Race (Above Threshold). This figure shows the cumulative percentage
of assessments above each score by race. AHA scores are more comparable
between Black and white clients pre- and post-deployment, while VI-SPDAT
consistently scores a higher percentage of white clients above each threshold.

##  Appendix D Research Methods

Here we expand the equation in described in § 4. low-risk is 1 if a client’s
AHA score is between 1-4 and 0 otherwise. medium-risk is 1 if a client’s AHA
score is between 5-8 and 0 otherwise. high-risk is 1 if a client’s AHA score
is between 9-10 and 0 otherwise. We chose low-risk group as our base group for
comparison. Thus, the coefficient of interest here is $7H$ which shows the
additional difference in change in service rate for Black clients in the high-
risk group pre and post-AHA deployment compared to white clients compared to
such change in the low-risk group.

$\displaystyle Served_{i,t}$
$\displaystyle=\beta_{0}+\beta_{1}Post_{i,t}+\beta_{2}Black_{i,t}$
$\displaystyle\quad+\beta_{3}Score_{i,t}+\beta_{4L}LowRisk_{i,t}$
$\displaystyle\quad+\beta_{4M}MediumRisk_{i,t}+\beta_{4H}HighRisk_{i,t}$
$\displaystyle\quad+\beta_{5}(Black\times Post_{i,t})$
$\displaystyle\quad+\beta_{6L}(LowRisk_{i,t}\times Post_{i,t})$
$\displaystyle\quad+\beta_{6M}(MediumRisk_{i,t}\times Post_{i,t})$
$\displaystyle\quad+\beta_{6H}(HighRisk_{i,t}\times Post_{i,t})$
$\displaystyle\quad+\beta_{7L}(LowRisk_{i,t}\times Black_{i,t})$
$\displaystyle\quad+\beta_{7M}(MediumRisk_{i,t}\times Black_{i,t})$
$\displaystyle\quad+\beta_{7H}(HighRisk_{i,t}\times Black_{i,t})$
$\displaystyle\quad+\beta_{8L}(LowRisk_{i,t}\times Black\times Post_{i,t})$
$\displaystyle\quad+\beta_{8M}(MediumRisk_{i,t}\times Black\times Post_{i,t})$
$\displaystyle\quad+\beta_{8H}(HighRisk_{i,t}\times Black\times Post_{i,t})$
$\displaystyle\quad+\beta_{9}X+\beta_{10}AltAHA+\gamma_{t}+\delta_{t}+\epsilon_{i,t}$
(4)

Table 5: Terminology and Notation

Term |  Notation |  Description
---|---|---
Dependent Variable |  |
Served |  $Served\in\\{0,1\\}$ |  1 if a client is enrolled by a housing service provider, 0 otherwise
Independent Variables |  |
AHA Score |  $AHAScore\in\\{1,\ldots,10\\}$ |  An integer AHA risk score between 1 and 10 assessed by the AHA tool.
Posted Score |  $PostedScore\in\\{1,\ldots,10\\}$ |  An integer risk score between 1 and 10 that is the maximum (AHA Score, Alt-AHA Score) for assessments where Alt-AHA were conducted
Use Alt Score |  $UseAltScore\in\\{0,1\\}$ |  1 if the Alt-AHA scores were used for prioritization, 0 otherwise (only applicable to assessments where Alt-AHA were conducted)
Quality Indicator |  $QualityIndicator\in\\{0,\ldots,1\\}$ |  a continuous variable showing the quality of the client’s data with 0 being the lowest and 1 being the highest.
Black |  $Black\in\\{0,1\\}$ |  1 if the client is Black, 0 otherwise
Post |  $Post\in\\{0,1\\}$ |  1 if the assessment was taken post AHA implementation, 0 otherwise
Risk Group |  $RiskGroup\in\\{\text{high, medium, low}\\}$ |  A categorical variable indicating if a client’s AHA score is high risk (9-10), medium (5-8), or low (1-4)
High Risk |  $HighRisk\in\\{0,1\\}$ |  1 if the client’s AHA score is 9 or 10, 0 otherwise
Medium Risk |  $MediumRisk\in\\{0,1\\}$ |  1 if the client’s AHA score is between 5 and 8 (inclusive), 0 otherwise
Low Risk |  $LowRisk\in\\{0,1\\}$ |  1 if the client’s AHA score is between 1 and 4 (inclusive), 0 otherwise
Assessment Characteristics |  $X$ |  A vector of variables on the characteristics of an assessment including the client’s age, gender, veteran status, chronic homeless status, disability status, whether a client is a domestic violence survivor, whether the client is the head of the household, the size of the household, the type of household (Single or Households without Children).
Yearly Fixed Effects |  $\gamma$ |  Yearly fixed effects controlling for the yearly variation in service rate
Quarterly Fixed Effects |  $\delta$ |  Quarterly fixed effects controlling for seasonal variation in service rate

##  Appendix E The Effects of Demographics and Eligibility Criteria

According to the AHA Methodology Update report (Vaithianathan and Kithulgoda
2020b), demographics include only age and gender—race is not considered when
generating AHA scores. Therefore, we classify age and gender as demographics,
while categorizing all other features, such as chronic homelessness status,
disability, veteran status, and domestic violence survivor status, as
eligibility criteria based on discussions with ACDHS (see section 2.4 for
details about the business process).

In Table 6, we regress demographics and a combination of demographics and
eligibility criteria on AHA scores using the sub-sample for which Alt-AHA was
never conducted, indicating sufficient data quality for AHA scores. This
allows us to disentangle the effects of demographics alone and demographics
combined with eligibility criteria on AHA scores.

Unsurprisingly, demographics alone explain only 1.92% of the variation in
scores, as indicated by the adjusted R-squared value, because demographic
features are just a small subset of all features used for prediction. However,
after including eligibility criteria, we find that race, demographics, and
eligibility criteria together explain 10.70% of the variation in scores,
suggesting that eligibility criteria are significantly associated with the
scores. Despite this, the explanatory power remains relatively low. Thus,
while adding eligibility criteria is expected to reduce the size of the
coefficients associated with AHA scores due to collinearity, they will also
directly impact service rates.

Table 6: Association Between Demographics and Eligibility Criteria and AHA
Scores

| (1) | (2)
---|---|---
Black | 0.2148∗ | 0.2624∗∗
| (0.0867) | (0.0831)
Gender Female | -0.2710 | -0.4576∗∗∗
| (0.2140) | (0.0962)
Gender Unknown | 0.0000 | -0.0132
| (.) | (0.2074)
Gender Male | 0.2195 | 0.0000
| (0.2047) | (.)
Age | -0.0246∗∗∗ | -0.0236∗∗∗
| (0.0035) | (0.0034)
Chronic Homeless |  | 0.6375∗∗∗
|  | (0.0762)
Disability |  | 1.8576∗∗∗
|  | (0.1186)
Is Veteran |  | -0.5812∗∗∗
|  | (0.1615)
Is Domestic Violence Survivor |  | -0.1046
|  | (0.1180)
Household Size |  | -0.3052
|  | (0.2347)
Single Household |  | -0.9896∗∗∗
|  | (0.0735)
Head of Household |  | 0.9292∗
|  | (0.3720)
N | 6,143 | 6,143
Adjusted R-squared | 0.0192 | 0.1070
Standard errors in parentheses
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

##  Appendix F Alternative Specifications for RQ2

In this section, we examine two alternative specification for RQ2 to
illustrate the robustness of the results on the heterogeneous effects of AHA
scores on service rate.

The first alternative specification treats AHA Score as a continuous variable
without segmenting assessments into different risk categories. In the second
alternative specification, we create a binary variable `high score` which
equals to 1 if the AHA Score associated with the assessment is higher or equal
to 9, and 0 otherwise. This allows us to more directly distinguish assessments
with high risk scores from the rest. The results are shown in Table 7 and
Table 8 respectively.

Aligned with the result presented in the main text, we find the coefficients
associated with the triple interaction terms of `Black * Post * AHA Score` and
`Black * Post * High Score` to be not statistically significantly. This shows
that the relationship between high AHA scores and service rates does not
differ significantly by race in the post-intervention period, supporting the
robustness of our findings.

Table 7: Service Rate by Race Risk Group, and AHA Implementation (Alternative
Specification 1)

| (1) | (2) | (3)
---|---|---|---
Black | 0.2369 | 0.3413 | 0.3717
| (0.2165) | (0.2628) | (0.2640)
Post | -1.6446∗∗∗ | -1.5083∗∗ | -1.6980∗∗
| (0.4610) | (0.5282) | (0.5580)
AHA Score | 0.0824∗∗∗ | 0.0191 | 0.0243
| (0.0203) | (0.0250) | (0.0250)
Black * AHA Score | -0.0714∗ | -0.0776∗ | -0.0830∗
| (0.0306) | (0.0370) | (0.0372)
Post * AHA Score | 0.2246∗∗∗ | 0.2324∗∗∗ | 0.2244∗∗∗
| (0.0551) | (0.0595) | (0.0602)
Black * Post | -0.7790 | -1.1041 | -1.2179
| (0.6583) | (0.7398) | (0.7500)
Black * Post * AHA Score | 0.1256 | 0.1570 | 0.1717
| (0.0787) | (0.0890) | (0.0901)
N | 6,143 | 6,143 | 6,143
Pseudo R-squared | 0.0282 | 0.2803 | 0.2830
Standard errors in parentheses
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

Table 8: Service Rate by Race Risk Group, and AHA Implementation (Alternative
Specification 2)

| (1) | (2) | (3)
---|---|---|---
Black | -0.1350 | -0.1100 | -0.1031
| (0.0979) | (0.1164) | (0.1173)
Post | -0.0922 | 0.0033 | -0.2449
| (0.1424) | (0.2570) | (0.3025)
High Score | 0.3959∗∗∗ | 0.0155 | 0.0344
| (0.1161) | (0.1420) | (0.1424)
Black * High Score | -0.3033 | -0.2035 | -0.2425
| (0.1757) | (0.2123) | (0.2140)
Post * High Score | 0.5890∗∗ | 0.7660∗∗ | 0.7443∗∗
| (0.2064) | (0.2437) | (0.2442)
Black * Post | -0.0158 | -0.0683 | -0.0980
| (0.2011) | (0.2376) | (0.2394)
Black * Post * High Score | 0.4139 | 0.3610 | 0.4240
| (0.2932) | (0.3481) | (0.3501)
N | 6,143 | 6,143 | 6,143
Pseudo R-squared | 0.0210 | 0.2754 | 0.2782
Standard errors in parentheses
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$