1.5

Short title for running head Month Year Vol Issue

#  Algorithmic Fairness and Social Welfare

Annie Liang and Jay Lu Liang: Northwestern, 2211 Campus Drive, Evanston IL
60208<EMAIL_ADDRESS>Lu: UCLA, 315 Portola Plaza Los Angeles,
CA 90095<EMAIL_ADDRESS>We thank Andrei Iakovlev for helpful comments.

Algorithms are increasingly used to guide high-stakes decisions about
individuals (for example, which patients to treat, or what kind of loan to
offer a prospective borrower). Consequently, substantial interest has
developed around defining and measuring the “fairness” of these algorithms
(Roth and Kearns, 2019; Barocas, Hardt and Narayanan, 2023).

These definitions of fair algorithms share two features: First, they
prioritize the role of a pre-defined group identity (e.g., race or gender) by
focusing on how the algorithm’s impact differs systematically across
groups.111Exceptions include, among others, Dwork et al. (2012)’s comparison
of outcomes across similar individuals. Second, they are statistical in
nature; for example, comparing false positive rates, or assessing whether
group identity is independent of the decision (both viewed as random
variables).

These notions are facially distinct from a social welfare approach to
fairness, in particular one based on “veil of ignorance” thought experiments
in which individuals choose how to structure society prior to the realization
of their social identity. In this paper, we seek to understand and organize
the relationship between these different approaches to fairness. Can the
optimization criteria proposed in the algorithmic fairness literature also be
motivated as the choices of someone from behind the veil of ignorance? If not,
what properties distinguish either approach to fairness?

We review the algorithmic fairness approach in Section 2, and the social
welfare approach in Section 3. We show in Section 4 that these approaches are
fundamentally different. We conclude by proposing a framework that nests both
approaches in Section 5.

##  1 Setup

There is a population of subjects each described by three variables: a
covariate vector $X\in\mathcal{X}$, a type $Y\in\mathcal{Y}$ (i.e., what the
algorithm would like to predict), and a group identity $G\in\mathcal{G}$. For
simplicity of exposition, we assume the sets $\mathcal{X}$, $\mathcal{Y}$, and
$\mathcal{G}$ are finite. The population distribution relating covariate
vectors, types, and group identities is denoted by
$\mu\in\Delta(\mathcal{X}\times\mathcal{Y}\times\mathcal{G})$.

Each subject receives a decision from the finite set $\mathcal{D}$ as the
(randomized) output of an algorithm
$a:\mathcal{X}\rightarrow\Delta(\mathcal{D}).$ Together with $\mu$, each
algorithm $a$ pins down a full joint distribution $P\in\Delta(\Omega)$ over
the entire state space
$\Omega=\mathcal{X}\times\mathcal{Y}\times\mathcal{G}\times\mathcal{D}$. We
define $\mathcal{P}_{\mu}\subseteq\Delta(\Omega)$ to be the set of feasible
distributions that can be implemented by some algorithm $a$ given the
population distribution $\mu$. A Designer chooses from the set of feasible
distributions $\mathcal{P}_{\mu}$.222This is equivalent to choosing among
algorithms but more convenient for the subsequent exposition.

##  2 Fair Algorithms

In a popular approach in the algorithmic fairness literature, the Designer
maximizes a measure of the accuracy of the algorithm subject to a fairness
constraint. The primitives of this approach are an accuracy measure
$v:\mathcal{D}\times\mathcal{Y}\rightarrow\mathbb{R}$, and a fairness
constraint (which we elaborate on below). The Designer solves the optimization
problem

(1) $\displaystyle\sup_{P\in\mathcal{P}_{\mu}}$
$\displaystyle\mathbb{E}_{P}[v(D,Y)]\quad\mbox{ s.t. }(*)$

where leading choices of the fairness constraint $(*)$ include the following
(see e.g., Mitchell et al. (2021) for a reference):

######  Definition 1 (Equalized Odds).

$D\perp\\!\\!\\!\perp G\mid Y$.

######  Definition 2 (Equality of False Negatives).

$D\perp\\!\\!\\!\perp G\mid Y=1$.

######  Definition 3 (Equality of False Positives).

$D\perp\\!\\!\\!\perp G\mid Y=0$.

######  Definition 4 (Statistical Parity).

$D\perp\\!\\!\\!\perp G$.

Algorithms that satisfy these fairness constraints are sometimes viewed as
“non-discriminatory.” Under this interpretation, the Designer first commits to
choosing a non-discriminatory algorithm and subsequently maximizes accuracy
among them. The prioritization of a non-discriminatory algorithm may reflect
the designer’s intrinsic preferences or external constraints (e.g., imposed by
policy, law, or public opinion); the optimization problem in (1) is consistent
with either interpretation.

##  3 Social Welfare Approach

The approach discussed above is qualitatively different from a long tradition
in moral philosophy and economics, which instead measures social welfare by
aggregating utilities across individuals in society. Here, fairness
considerations stem from contemplating how an individual would choose to
structure society prior to the realization of the individual’s own identity
(Harsanyi, 1953, 1955; Rawls, 1971). From behind this “veil of ignorance,” a
preference for fairness can be micro-founded as risk aversion over the
realization of identity. We review a formalization of this approach.

The Designer forms preferences by considering a hypothetical individual whose
identity has yet to be realized. The possible identities are indexed by
$i\in\mathcal{I}$ (each realized with probability $p_{i}$), with $u_{i}$
denoting the eventual utility under identity $i$. The individual’s ex-ante
payoffs, aggregating over the possible identities, are given by
$\sum_{i\in\mathcal{I}}p_{i}\phi(u_{i})$ for some concave and strictly
increasing function $\phi:\mathbb{R}\rightarrow\mathbb{R}$.333For an
axiomatization and further discussion of this representation, see Grant et al.
(2010).

When $\phi$ is linear (corresponding to risk-neutrality), this representation
returns utilitarianism, à la Harsanyi. Strict concavity of $\phi$ corresponds
to risk aversion over which identity one assumes following the lifting of the
veil of ignorance, and characterizes a preference for fairness. For example,
if $\phi(u)=\sqrt{u}$ and there are two equally-likely individuals, then the
“more fair” utility vector $(u_{1},u_{2})=(2,2)$ is preferred over the “less
fair” utility vector $(u_{1},u_{2})=(1,3)$. An infinite risk-aversion limit
returns Rawls’ preferred perspective, in which social outcomes are evaluated
based on the most disadvantaged individual, i.e.
$\min_{i\in\mathcal{I}}u_{i}$.444Let $\phi(u)=-u^{-\gamma}$ and consider
preferences in the limit as $\gamma\rightarrow\infty$.

This framework can be adapted to accommodate a group-based notion of fairness.
To do this, we extend the thought experiment and suppose our Designer adopts
the perspective of an individual behind the veil who anticipates a _compound_
lottery: First, group identity is realized, and then a second lottery
determines the remaining characteristics. If the individual has different risk
preferences over the two-stages of the compound lottery, the Designer will
solve the following maximization problem

(2)
$\displaystyle\sup_{P\in\mathcal{P}_{\mu}}\sum_{g}p_{g}\phi\left(U\left(P_{g}\right)\right)$

where $P_{g}$ denotes the conditional distribution of $P$ given $G=g$ and
$U(P_{g})=\mathbb{E}_{P_{g}}[u(D,Y)]$ is the expected utility of group
$g$.555This setup and representation resemble models of second-order expected
utility; see Kreps and Porteus (1978). We could also allow $U(\cdot)$ to
capture different risk preferences over the remaining covariates, and do not
expect our results to change. Concavity of $\phi$ now reflects aversion to
group-identity risk.

##  4 Comparing the Two Approaches

We first demonstrate in a simple example that the two approaches can be
incompatible.

######  Example 1.

The population is divided equally into two groups $G\in\\{0,1\\}$. The type
$Y\in\\{0,1\\}$ is an indicator for whether a patient needs treatment, and the
decision $D\in\\{0,1\\}$ is an indicator for whether the patient is treated.
The only observable covariate is $X=G$, so the decision $D$ is the (random)
outcome of an algorithm $a:\mathcal{G}\rightarrow\Delta(\\{0,1\\})$.

There is a $\delta\in(1/2,1)$ such that the population distribution $\mu$
satisfies

(3) $\mu(Y=j\mid G=j)=\delta$

for both $j\in\\{0,1\\}$. That is, most members of group $G=1$ need treatment,
while most members of group $G=0$ do not need treatment.

Consider two Designers. The first solves the constrained optimization problem
in (1) given the Equalized Odds constraint in Definition 1
($D\perp\\!\\!\\!\perp G\mid Y$). We leave the accuracy measure $v$
unspecified.

The second is our social welfare Designer who solves the maximization problem
in (2), endowed with utility function $u(d,y)=\mathbbm{1}(d=y)$ and any
concave and strictly increasing $\phi$.

######  Claim 1.

In Example 1, any solution to the constrained optimization problem (1) is not
a solution to the social welfare problem (2), and vice versa.

###### Proof.

Since the only covariate on which the algorithm conditions is group identity,
each algorithm can be identified with a pair $(q_{0},q_{1})\in[0,1]^{2}$ where
$q_{g}=P_{g}(D=1)$. Since every pair is feasible, it is possible for the
social welfare Designer to optimize for the expected utilities of the two
groups separately. His payoff is maximized by setting $q_{1}=1$ and $q_{0}=0$,
yielding a payoff of
$\frac{1}{2}\phi(\delta)+\frac{1}{2}\phi(\delta)=\phi(\delta)$. This algorithm
clearly fails Equalized Odds. In contrast, an algorithm satisfies Equalized
Odds if and only if $q_{0}=q_{1}:=q$.

In this example, the two Designers are in conflict. The social welfare
Designer prefers to treat everyone in group $G=1$ and no one in group $G=0$,
but this algorithm fails Equalized Odds and so is unacceptable to the
constrained optimization Designer.

In the other direction, any algorithm that satisfies Equalized Odds leads to a
lower payoff for the social welfare Designer. How large is the loss? We showed
in the proof of Claim 1 that the social welfare Designer’s preferred algorithm
gives him a payoff of $\phi(\delta)$. In contrast, any algorithm that
satisfies Equalized Odds gives the social welfare Designer a payoff of

(4) $\displaystyle\frac{1}{2}$
$\displaystyle\phi\left(q\delta+(1-q)(1-\delta)\right)$
$\displaystyle\quad+\frac{1}{2}\phi\left((1-q)\delta+q(1-\delta)\right).$

Since $\phi$ is concave, Jensen’s inequality implies that (4) is less than

$\phi\left(\frac{1}{2}\delta+\frac{1}{2}(1-\delta)\right)=\phi\left(\frac{1}{2}\right)<\phi(\delta)$

where the inequality follows from our assumptions that $\delta>1/2$ and that
$\phi$ is strictly increasing.

Thus, not only is the social welfare Designer’s payoff strictly lower under
the constrained optimization algorithm, but interestingly the disparity in
payoffs becomes larger the more the social welfare Designer cares about
fairness (i.e., the more concave $\phi$ is). So the normative content of the
two approaches about fairness are qualitatively different: Not only do the
Designers have different optimal algorithms, but their fairness objectives
push them in contrasting directions.

We observe finally that the disagreement between designers cannot be resolved
by relaxing the fairness constraint in the constrained optimization problem.
To see this, observe that the social welfare Designer’s preferred algorithm
perfectly correlates $D$ and $G$. Thus it not only fails Equalized Odds, but
would fail any moderate relaxation of Equalized Odds as well. For example, if
Equalized Odds is relaxed to the condition that

(5) $\displaystyle|\mathbb{E}[D$ $\displaystyle\mid Y=y,G=1]$
$\displaystyle-\mathbb{E}[D\mid Y=y,G=0]|\leq\varepsilon.$

for each $y$, then the statement of the claim holds for all $\varepsilon<1$.

We now show that these findings are not special to our particular
specifications of the two Designers (e.g., the choice of utility in the social
welfare approach or the consideration of Equalized Odds as the fairness
constraint) but reflect a more general tension between the approaches. Towards
this, note that the Designer’s choices can be modeled as a _choice rule_
$c:\Delta(\mathcal{X},\mathcal{Y},\mathcal{G})\rightarrow\Delta(\Omega)$
specifying a feasible distribution $c(\mu)\in\mathcal{P}_{\mu}$ for every
population distribution $\mu$. Let $\mathcal{C}_{co}$ be the set of choice
rules for all constrained optimization Designers, as we vary over the fairness
constraints $(*)$ from Definitions 1-4 and all accuracy measures $v$ (with an
arbitrary tie-breaking rule).666If no feasible distribution satisfies the
fairness constraint, then pick any feasible distribution. Let
$\mathcal{C}_{sw}$ be the set of choice rules for all social welfare
Designers, as we vary over all concave and strictly increasing functions
$\phi$ and all _nontrivial_ utility functions $u$ (defined below), again with
any tie-breaking rule.777The subsequent Proposition 1 holds without the
concavity restriction but we maintain it for ease of comparability with the
algorithmic fairness approach.

######  Definition 5.

A utility function $u:\mathcal{D}\times\mathcal{Y}\rightarrow\mathbb{R}$ is
_nontrivial_ if not all types rank decisions the same, i.e.
$u(d,y)>u(d^{\prime},y)$ and $u(d,y^{\prime})<u(d^{\prime},y^{\prime})$ for
some $d,d^{\prime},y,y^{\prime}$.

The following result says that the constrained optimization and the social
welfare approaches yield distinct choice rules.

######  Proposition 1.

Any constrained optimization choice rule is not a social welfare choice rule,
and vice versa, i.e. $\mathcal{C}_{co}\cap\mathcal{C}_{sw}=\emptyset$

###### Proof.

It is sufficient to show that for every $v$, $(*)$, nontrivial $u$, and
strictly increasing and concave $\phi$, there exists a $\mu$ such that
$c(\mu)$ is different under the two approaches. We will show this by
construction, adapting Example 1. We prove the result assuming binary $Y$,
$G$, and $D$, although the argument can be extended otherwise. Suppose that
$X=G$ and define $q_{g}=P_{g}(D=1)$, observing that each algorithm is again
identified with the pair $(q_{0},q_{1})$.

By assumption that $u$ is nontrivial, there exist $y_{0},y_{1}\in\mathcal{Y}$
such that $u(1,y_{1})>u(0,y_{1})$ but $u(1,y_{0})<u(0,y_{0})$. Consider a
population distribution $\mu$ where there is a $\delta$ such that

$\displaystyle\mu(Y=y_{j}$ $\displaystyle\mid G=j)=\delta$

for each $j\in\\{0,1\\}$. Then (2) reduces to choosing
$(q_{0},q_{1})\in[0,1]^{2}$ to maximize

$\displaystyle\frac{1}{2}\phi((1-q_{1})\delta
u(0,y_{1})+q_{1}(1-\delta)u(1,y_{0})$ $\displaystyle\,\,\,\,\,+q_{1}\delta
u(1,y_{1})+(1-q_{1})(1-\delta)u(0,y_{0}))+$
$\displaystyle\frac{1}{2}\phi((1-q_{0})(1-\delta)u(0,y_{1})+q_{0}\delta
u(1,y_{0})$ $\displaystyle\,\,\,\,\,+q_{0}(1-\delta)u(1,y_{1})+(1-q_{0})\delta
u(0,y_{0}))$

We can choose $\delta$ large enough such that the unique solution is
$(q_{0},q_{1})=(0,1)$. But each of the fairness constraints in Definitions 1-4
is satisfied if and only if $q_{0}=q_{1}$, so we have the desired result.

The result does not say that social welfare Designers and constrained
optimization Designers never agree.888For example, if (3) in Example 1 were
replaced by $\mu(Y=1\mid G=g)=\delta$, then the algorithm that treated
everyone in the population would maximize the social welfare Designer’s
payoff, and also satisfy Equalized Odds. Instead, for any social welfare
Designer and any constrained optimization Designer, there is _some_ population
(i.e. $\mu$) for which the Designers’ preferred distributions diverge. Since
to our knowledge neither view of fairness is predicated on specific
assumptions on the population, we interpret the approaches as intended
“universally,” and Proposition 1 as thus establishing that the two approaches
are distinct.

Intuitively, the result follows because the two Designers view fairness
differently. The constrained optimization Designer requires conditional
independence of certain variables given others, which is a statistical
property across the two groups. The social welfare Designer is instead
concerned with what the joint distribution of all the variables implies for
the _utilities_ across the two groups. When not all distributions on the state
space are feasible, these two approaches can be in conflict with one another.

The challenge of micro-founding the constrained optimization approach using a
social welfare narrative does not necessarily imply that those approaches are
misguided, but does imply that they require novel justifications. We leave
exploration of such questions for future work.

##  5 A More General Framework

In this final section, we propose a general framework that nests the two
approaches. First note the constrained optimization problem specified in (1)
can be reformulated as

$\displaystyle\sup_{P\in\mathcal{P}_{\mu}}$
$\displaystyle\mathbb{E}_{P}[v(D,Y)]-h(P_{g_{1}},\dots,P_{g_{n}})$

where $h$ is an indicator function that returns infinite cost if the fairness
constraint $(*)$ is violated. For example, the Equalized Odds constraint
(Definition 1) can be expressed as

$\displaystyle{P_{g_{1}}(D|Y=y)}=\dots={P_{g_{n}}(D|Y=y)}$

for all $y$. Similar expressions can be written for the other constraints in
Definitions 1-4.999The same holds for any relaxed versions of these
constraints, e.g. condition (5).

Extending this representation, we consider the following maximization problem

$\displaystyle\sup_{P\in\mathcal{P}_{\mu}}$
$\displaystyle\phi(\mathbb{E}_{P}[v(D,Y)])-h(P)$

where $\phi(\cdot)$ is a strictly increasing function and
$h\left(\cdot\right)$ is a measure of unfairness that satisfies
$h\left(P\right)\geq 0$ with $h\left(P\right)=0$ if $P_{g}$ is the same for
all $g$. In addition to nesting the constrained optimization problem, this
representation returns the social welfare formulation when $v=u$, $\phi$ is
concave and

$h\left(P\right)=\phi\left(\sum_{g}p_{g}U\left(P_{g}\right)\right)-\sum_{g}p_{g}\phi\left(U\left(P_{g}\right)\right)$

This unfairness measure $h$ can be interpreted as comparing the utility of an
individual whose identity is known to be the expected group identity, versus
the expected utility of an individual with a random group identity. By
Jensen’s inequality, $h(P)$ is always positive, and its value is larger when
the distribution over utilities (induced by different realizations of the
group identity) is more dispersed. Alternative choices for the unfairness
measure $h$ may generate novel preferences for fair algorithms, and would be
interesting avenues for future research.

## References

  * (1)
  * Barocas, Hardt and Narayanan (2023) Barocas, Solon, Moritz Hardt, and Arvind Narayanan. 2023\. Fairness and Machine Learning: Limitations and Opportunities. MIT Press.
  * Dwork et al. (2012) Dwork, Cynthia, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012\. “Fairness through awareness.” 214–226.
  * Grant et al. (2010) Grant, Simon, Atsushi Kajii, Ben Polak, and Zvi Safra. 2010\. “Generalized Utilitarianism and Harsanyi’s Impartial Observer Theorem.” Econometrica, 79: 1939–1971.
  * Harsanyi (1953) Harsanyi, John. 1953\. “Cardinal Utility in Welfare Economics and in the Theory of Risk-Taking.” Journal of Political Economy, 61: 434–435.
  * Harsanyi (1955) Harsanyi, John. 1955\. “Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparison of Utility: Comment.” Journal of Political Economy, 63: 309–321.
  * Kreps and Porteus (1978) Kreps, D., and E. Porteus. 1978\. “Temporal Resolution of Uncertainty and Dynamic Choice Theory.” Econometrica, 46(1): 185–200.
  * Mitchell et al. (2021) Mitchell, Shira, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2021\. “Algorithmic Fairness: Choices, Assumptions, and Definitions.” Annual Review of Statistics and Its Application, 8: 141–163.
  * Rawls (1971) Rawls, John. 1971\. A Theory of Justice. Harvard University Press.
  * Roth and Kearns (2019) Roth, Aaron, and Michael Kearns. 2019\. The Ethical Algorithm: The Science of Socially Aware Algorithm Design. Oxford University Press.


# A Standardized Framework for Collecting Graduate Student Input in Faculty
Searches

Yasmeen Asali Department of Astronomy, Yale University, 52 Hillhouse Ave, New
Haven, CT 06511, USA  Astronomy Student Council (ASC) at Yale, 2021/22
Konstantin Gerbig Department of Astronomy, Yale University, 52 Hillhouse Ave,
New Haven, CT 06511, USA  Astronomy Student Council (ASC) at Yale, 2021/22
Aritra Ghosh Department of Astronomy, Yale University, 52 Hillhouse Ave, New
Haven, CT 06511, USA  Astronomy Student Council (ASC) at Yale, 2021/22
Christopher Lindsay Department of Astronomy, Yale University, 52 Hillhouse
Ave, New Haven, CT 06511, USA  Astronomy Student Council (ASC) at Yale,
2021/22  Zili Shen Department of Astronomy, Yale University, 52 Hillhouse Ave,
New Haven, CT 06511, USA  Astronomy Student Council (ASC) at Yale, 2021/22
Marla Geha Department of Astronomy, Yale University, 52 Hillhouse Ave, New
Haven, CT 06511, USA

###### Abstract

We present a procedure designed to standardize input received during faculty
searches with the goal of amplifying student voices. The framework was
originally used to collect feedback from graduate students, but it can be
adapted easily to collect feedback from undergraduate students, faculty, staff
or other stakeholders. Implementing this framework requires agreement across
participating parties and minimal organization prior to the start of faculty
candidate visits.

††software: Python (Van Rossum & Drake, 2009)

##  1 Introduction

Faculty hiring at major research universities includes input from many
different groups. Graduate students are highly impacted by available faculty
in a department and can provide critical and unique feedback during the hiring
process. In a living document written by members of Colorado State
University’s Graduate Center for Inclusive Mentoring and Graduate Student
Council titled Graduate Student Input in the Faculty Search Process (2021),
the authors also argued that high levels of graduate student involvement can
reflect well on the department itself, thus potentially attracting higher
quality candidates. Further, graduate students themselves benefit from the
professional development experience that is active participation in a hiring
process. For these reasons, graduate student input is considered best practice
in faculty hiring processes (see e.g., Columbia University, Office of the
Provost, 2016; Cornell University, Office of Faculty Development and
Diversity, 2021).

However, the degree of graduate student involvement varies significantly
between university departments, and can encompass for example student
representation on the hiring committee (Rutgers University, Department of
Sociology, 2022) or the collection of graduate input from web-based feedback
forms (University of Washington, Paul G. Allen School of Computer Science &
Engineering, 2021).

Inspired by a survey designed during a 2022 faculty hiring process at Yale
University, we present a procedure to amplify graduate student voices in the
hiring process. This write-up and associated survey documentation is intended
to be easily adopted by other departments to standardize and improve the
faculty hiring process.

In 2022, the Department of Astronomy at Yale University initiated a search for
a new faculty member. Current graduate students were given the opportunity to
interact with all the shortlisted candidates via an hour-long meeting, as well
as the candidate’s colloquium and other informal departmental interactions. To
reduce subjectivity, the Yale Astronomy Student Council (ASC) designed and
implemented a procedure to gather and present data collected from all graduate
students during these interactions. The ASC is an elected body of 3 to 5
graduate student members that work to address a wide variety of issues
concerning graduate students and the graudate program within the department.
These data and an associated write-up were presented at a department faculty
meeting at the end of the candidate interview process.

This paper was inspired by the positive response to the faculty hiring survey
from both students and faculty. Although we cannot quantify its impact, the
survey provided a more standardized method to provide student feedback into
the decision making process compared to previous faculty hires at Yale
Astronomy. It provided a clear picture of graduate student opinions,
highlighting both points of consensus and areas of disagreement.

With the goal of reducing the barrier for others to adopt these more objective
evaluation methods, we provide an overview of the survey design,
implementation and data presentation. Example survey forms and python code to
analyze results are provided via a public GitHub repository (see Appendix A).

We note that this entire work was carried out at an American R1 research
university with its own specific history, bias, and socio-economic existence.
It might be important to take this into consideration while
adapting/implementing this procedure at a different institution.

##  2 Survey design and implementation

Our framework for graduate student input in faculty hiring was developed and
is intended for a procedure that includes multi-day visits for each short-
listed candidate. The process must be set in motion prior to the first
candidate’s arrival. The first step must be to establish a consensus about the
structure of the visits, such that they are run homogeneously across faculty
candidates.

In our case, the graduate student body was invited to department-wide science
colloquia presented by the faculty candidates. In addition, there were also
scheduled, one hour long zoom meetings open exclusively to the candidate and
graduate students, which were moderated by ASC members. These two events
constituted the basis for graduate students to form an opinion and provide
feedback on the candidates. Two weeks after the last candidate’s interview,
there was a meeting with the graduate student body and current faculty, where
the results of the survey were presented. The entire process was conducted
fully virtually in light of the Covid19 pandemic. A summary flowchart of our
method for collecting graduate student feedback is shown in Fig. 1.

Figure 1: Flowchart showing our procedure for collecting feedback on the
faculty candidates from the graduate students. Blue boxes indicate ASC actions
and red boxes indicate input from the full graduate student body.

###  2.1 Selecting interview questions

During each meeting, the ASC asked the candidate a number of set questions,
with time allocated for both candidate response and potential spontaneous
follow-up questions from graduate students. The set list of questions was
determined prior to the first candidate’s interview and remained unchanged for
the duration of the hiring process. Graduate students were first asked to
contribute and then to cross-vote for questions in order to establish a
ranking. From the list of questions, the ASC identified four general areas of
interest: advising students, teaching (specifically graduate classes),
departmental community, and research. Appendix B contains the developed list
of questions.

Note that, in order to avoid overlap with the candidates’ colloquia, where
they discussed their respective research programs and scientific goals, the
graduate student questions were intended to focus more on the candidates’
thoughts on advising, teaching, developing departmental community, and their
specific impact on the student body.

###  2.2 Structure of the graduate student - faculty candidate meeting

The graduate student - faculty candidate meetings started with a brief
introduction by the faculty candidate. The remaining time was divided into
four sections corresponding to the previously identified areas of interests.
The ASC always asked the faculty candidate the question with highest number of
votes first before moving on to the questions with the next highest numbers of
votes until time ran out on that area of interest. This ensured that
candidates, while being confronted with varying numbers of questions, all
answered the same 2-3 top voted questions within each area of interest.

Finally, the last 10 minutes were specifically allocated for the candidate to
ask the graduate student body questions, which we found to be particularly
useful for graduate students to gauge the faculty candidate’s priorities.

###  2.3 Data collection

The ASC sent out a link to a Google form survey after each candidate had given
their colloquium. Graduate students were required to log in with their Yale
email addresses to verify their identities. However, their emails were not
recorded with their responses in order to preserve anonymity. Graduate
students could indicate whether they were above or below Year 2 at the time of
the survey, corresponding to whether they were taking classes or had advanced
to candidacy, although we did not end up using this information in the final
analysis.

Students graded each candidate on a 0-5 scale (with 0 being “absolutely don’t
hire” and 5 being “strong hire”) for each of the four identified areas of
interest. Finally, each graduate student was asked to provide an overall
assessment, again from 0-5. Students also had the opportunity to write long-
form responses if they had additional comments about the candidate for each
section.

We encouraged the graduate students to fill out the survey for each candidate
before the subsequent week’s faculty candidate - graduate student meeting but
kept each survey form open in case students would like to edit their
responses, and to increase participation. We note that keeping the survey form
open after the week of each candidate’s visit introduces the problem of
recency bias and leads to graduate students comparing two faculty candidates
directly if filling out two survey forms at the same time.

All feedback forms were closed one week after the last candidate’s interview.
Leading up to this deadline, the ASC sent out reminder emails to the graduate
student body to fill out the survey forms. To encourage graduate student
participation and mitigate low number statistics, these emails included the
current number of responses for each candidate.

##  3 Survey analysis and mock results

Figure 2: The mean values of the grades assigned by graduate students on
different parameters for all the candidates. The dashed lines represent the
average values of scores for individual candidates. The shown results are
based on randomly generated mock data.  Figure 3: Same as Figure 2, but now
showing the median values of the scores instead of the mean. The shown results
are based on randomly generated mock data. Figure 4: Histograms of grades for
Grace H.,Vera R., and Carl S. The numbers in the legend refer to the number of
respondents for each candidate. The mean($\mu$), median($\tilde{\mu}$), and
mode (Mo) of each distribution are shown in the top-left of each panel. The
number of comments for each respondent, separated into positive, neutral, and
negative, is shown on the right. The shown results are based on randomly
generated mock data. Figure 5: Histograms of grades for Albert E. and
Subrahmanyan C. The shown results are based on randomly generated mock data.
Figure 6: Histograms of grades for Nancy R. and Edwin H. The shown results are
based on randomly generated mock data.

In this section, we present the analysis of our survey as well as show results
obtained by running our analysis pipeline on generated mock data. The real
survey conducted in Yale Astronomy in spring 2022 is not included in this
paper to keep its contents confidential.

The aim of this section is to demonstrate our analysis and visualization
pipeline, for which we believe the mock data to suffice. We stress that
results based on our mock data differ from any real survey, and dissuade from
their interpretation. The generation of our mock data and associated caveats
are described in Appendix C.

###  3.1 Numerical grades

In Fig. 2, we have summarized the average values obtained for each of the
broad sections on which graduate students graded the candidates. We want to
note that “Overall Score” is not the average score of the previous broad
sections but was an additional question in the survey, asking students to
summarize their overall opinion on whether the candidate should be hired as a
faculty member (see Sect. 2.3). The dashed lines in Fig. 2 show the averages
over all five categories for each candidate. Figure 3 depicts the same data
but using the median as the measure of average instead of the arithmetic mean.
We leave it to the reader to judge which measure they think is better suited
for the data at hand. In order to aid in this process, we show the summarized
histograms of all the candidates (split into 3-2-2) in Figs. 4, 5, and 6. We
did not plot more than three candidates in each panel to avoid overcrowding.

While the mock data in Figs. 4, 5, and 6 more or less retrace the underlying
probability distributions, real survey results can follow any distribution. In
our survey, we explicitly did not curate the data from numerical evaluations
(such as removing outliers) to match perceived probability distributions.

###  3.2 Individual comments

The ASC also classified the comments received for each candidate as negative,
neutral, or positive, with the goal of providing a quantitative measure
intended as a quick-to-read stand-in for the general sentiment of the
comments. The number of comments for each candidate in the mock data, broken
down into these categories, is shown in the top right of Figs. 4, 5, 6. Since
this classification is inherently highly subjective, we strongly discouraged
readers from relying on this subjective metric alone, and instead encouraged
basing opinions on the actual comments that were included in our original
report. We also considered using different metrics of concisely conveying the
overall graduate student sentiment from the comments. One such idea was
curating the comments for each candidate in brief summaries. However, this
arguably would have increased subjectivity, and likely magnified the
summaries’ authors’ opinions. Thus, we did not include this in our report.

Another point of consideration is that the number of comments in general does
not correspond to the number of numerical grades given. In our case, roughly
half of the graduate students who responded to the survey via numerical grades
also added specific comments. In addition, each feedback form had a maximum of
five possible text-form comment submissions. Because of this, the graduate
student population that filled out the comments may significantly differ from
the population that evaluated on the provided numerical scale. For example in
the mock data, Grace H.’s 34 comments (see Fig. 4), may be authored by
anywhere between 7 to 34 graduate students (although in this example the
latter would require some graduate students to only submit comments and no
numerical rating, which was not possible the way our Google form was set up;
see Sect. 5). For these reasons, while the comments certainly help convey a
more detailed assessment of a candidate, they express the views of a subset of
individual graduate students and thus are not necessarily representative.

##  4 Survey presentation

Once the data was analysed, the ASC compiled the results into a draft write-
up. The graduate students were given the opportunity to provide feedback on
the write-up and take out any comments they did not want to be included in the
final report. The final report was made available to all the faculty members a
few days before a meeting of all graduate students and faculty members where
the ASC presented the results of the survey.

The report contained a brief description of our survey design, but focused
mostly on the survey results. As such, we included the comparative plots (as
presented for mock data in Sect. 3), as well as individual results for each
candidates. The latter included all unedited long form comments submitted
through the Google form, if not previously retracted by the respective
graduate student author. Our write-up did not include any additional
discussion or evaluation of the faculty candidates beyond the survey results
and associated plots.

The meeting between graduate students and faculty was centered around a
presentation of the survey, its design and results, by members of the ASC. Our
aim was to represent the graduate student body’s assessment of the candidates
as a whole. As such, we focused on the numerical evaluation visualized in the
comparative plots as shown in Sect. 3 for mock data. Since the individual
comments were too plentiful to present in the hour-long meeting, and we wanted
avoid cherry picking, we pointed the faculty to the write-up for anonymous
individual opinions on the candidates. The presentation was followed by a
discussion between faculty and graduate students.

##  5 Discussion and recommendations for future implementations

The ASC led and conducted this survey with the primary objective of collecting
student opinion on prospective faculty members in a method that maximizes
fairness, transparency, and objectivity to all prospective faculty candidates
as well as participating graduate students. In this section, we include our
reflections on having gone through this process and our recommendations for
institutions who might want to adopt this approach.

The predetermined set of questions for each candidate interview as well as the
standardized survey provided graduate students the tools necessary to evaluate
the different candidates consistently and objectively. Asking students to
submit all questions beforehand ensured that students who might not have been
comfortable speaking up at the interaction still got an opportunity to ask
their questions. It also ensured that the interviews did not change
drastically based on whether certain graduate students were present in a
specific interview or not. Another important aspect of our discussion
moderation was having rough time limits for each broad section. This ensured
that we were able to cover all topics for all candidates, while still giving
graduate students reasonable opportunity to ask follow-up questions. Graduate
students found the time allocated for the candidate to ask questions
themselves of strong use in forming an opinion.

The feedback forms were designed such that it was mandatory for respondents to
assign grades for all sub-sections of the form. We feel that in future
surveys, we would prefer making all grading questions optional in order to
accommodate situations where respondents do not want to grade a candidate on a
specific section.111In the current survey, ASC members manually deleted grades
when students specifically wrote in comments that their grades for a specific
section should be ignored. In addition, the setup of the feedback form
implicitly introduced equal weights to the four probed categories ‘Advising’,
‘Teaching’ ‘Community’, and ‘Research’. Moreover, it was difficult for the
evaluating students to quantify the candidate on other metrics. While the
fifth question, ‘general assessment’ was intended to provide some room for
evaluation outside these four rigid categories, it was arguably limited in
effect it was scored on a 0-5 scale as well, thus making this undefined grade
exactly a fifth of the total score. For example, a student may strongly object
a candidate’s hiring for reasons that are not captured in the fixed four
categories. In the most extreme case (5 points in the four categories, 1 point
in ‘general assessment’), this would yield and average score of 4.2, hardly
representative of the student’s actual opinion. Possible pathways to mitigate
this issue include having more categories, ideally voted on before the start
of the hiring process; giving the ‘general assessment’ score a greater weight
compared to the four rigid categories; including the option to introduce
custom weights to the five questions that account for the perceived importance
of the corresponding categories.

While compiling the final report, the ASC focused on making its contents as
data-centric as possible. We did not include any detailed analysis of the
results or the comments beyond making summary plots, as shown in this paper.
This was primarily because we felt including any detailed analysis or doing
comment moderation/curation would include potential bias of ASC members into
the process.

Since the ASC was involved in a faculty hire for the first time, we did not
have an established framework of how the survey would be done, how the results
would be analyzed and presented. The ASC devised different appropriate
strategies throughout the process and informed different departmental
stakeholders along the way. We feel that it would have been better if we would
have been able to outline the entire process in detail to the students,
faculty, and candidates before the process started.

We should also note that while graduate students presented this report to all
faculty members, no students were involved in the final faculty meeting which
took place to finalize the hire. Though the department had specifically
reached out to students asking for feedback on the candidates, graduate
students did not know beforehand how their feedback would be utilized. Having
this knowledge beforehand might be beneficial in terms of both survey design
and execution.

Although this survey was designed and used to collect graduate student
feedback, it could also be adapted easily to collect feedback from
undergraduate students, faculty and staff members. It is also important to
note that this process was carried out entirely digitally. All meetings
mentioned in this document happened over Zoom. Thus, there might be additional
considerations that need to be made while repeating this in a non-digital
setting.

, Matplotlib (Hunter, 2007), Numpy (Harris et al., 2020)

## Acknowledgments

The authors thank Michelle Nearon, Larry Gladney, Alison Coil for valuable
input. The authors also emphasize the importance of all Yale Astronomy
graduate students for providing feedback, as well as the Yale Astronomy
faculty for inviting graduate student input.

##  Appendix A Accessing The Survey Form & Analysis Pipeline

The form used to collect survey responses as well as our analysis pipelines
can be accessed at the following GitHub repository –
https://github.com/aritraghsh09/asc-faculty-hire-paper-assets

##  Appendix B Interview questions

This appendix contains a list of questions that were asked to each faculty
candidate during their meeting with the graduate students (see Sect. 2.1). For
brevity, we only show the most upvoted questions, i.e. those that were always
asked to each candidate. In addition, some questions were edited in order to
retain confidentiality. These questions are indicated by purple coloring and
italic font.

###  B.1 Advising and supporting students

  1. 1.

What do you think you would be able to bring to the table in terms of
supporting graduate students facing non-academic challenges? (for eg.
financial hardships, challenges related to family/partners, challenges related
to mental health)

  2. 2.

How many graduate students (at a time) do you see yourself advising? Do you
see yourself running a “group,” or treating each student and their projects
independently?

  3. 3.

What specific needs do you think international students have? How would you be
able to support them?

  4. 4.

In what capacity have you worked with undergraduate or graduate students?

  5. 5.

What is your advising style?

###  B.2 Teaching

  1. 1.

If you could invent a new graduate-level class, what would the syllabus look
like?

  2. 2.

What kind of teaching pedagogy do you believe in? Are there any specific
resources regarding teaching at our institution that you would be excited to
use?

###  B.3 Departmental community

  1. 1.

Besides research and teaching, what aspect of the departmental community would
you be interested in engaging with? Would you like to start something new?
Why? What?

  2. 2.

Do you have any experience in building community? What do you think are the
most crucial steps/actions needed in order to build a wholesome community?

###  B.4 Research

  1. 1.

Tell us your vision for a summer project, a one year project, and a PhD thesis
project?

  2. 2.

Do you have plans to collaborate with faculty, postdocs and students from
other departments at our institution? If so, how?

  3. 3.

Given how the graduate curriculum is structured at our institution, how would
you expect your students to balance their research and class/teaching
responsibilities?

##  Appendix C Generating mock data

Mock data was generated to approximate the results of the survey in the
interest of preserving anonymity and confidentiality. We generated mock data
for 7 candidates in the five categories of our original survey:
Advising/Support Students, Teaching, Departmental Community, Research, and
Overall Score. For each mock candidate, we selected a random number of survey
responses between 11 and 15, which were the original lower and upper bounds of
our responses. We generated a random average score for each candidate selected
from a normal distribution with a mean of 4.0 and and standard deviation of
1.0, with a forced upper bound of 5.0. We would like to emphasize that this
distribution was not selected to match the distribution of responses received
during the survey. Rather, these average scores are just selected to generate
mock data with a large spread for the purpose of visualization.

Once each mock candidate is assigned an average score, we use this average to
generate random data in each of the five categories for that candidate for a
selected number of survey responses. For example, data for the candidate Nancy
R. was generated using a normal distribution centered at a mock average of
1.69 and a standard deviation of 1.0 for 14 responses in each of the five
categories. Each category has the same number of responses because each
category was a required response on our survey.

For each candidate we also generate a random number of comments classified as
positive, neutral, and negative. The comments themselves are not generated,
rather we simply generate a random number of comments of each type. We used
the same upper and lower bounds for each type of comment as we received in the
original survey. The number of positive comments ranges from 7 to 29, the
number of neutral comments ranges from 0 to 12, and the number of negative
comments ranges from 1 to 7. We generate 7 sets of random comments (one set
per candidate), then compute an average comment score for each set. These sets
are sorted by average score, and matched based on the average score of each of
the seven candidates so that the higher ranked candidates based on the
categorical data are allotted the higher ranked comments. This step was taken
to better approximate our original data, where highly rated candidates based
on the categorical scores were also highly rated based on the comments.

## References

  * Columbia University, Office of the Provost (2016) Columbia University, Office of the Provost. 2016, Guide to Best Practices in Faculty Search and Hiring, https://provost.columbia.edu/sites/default/files/content/BestPracticesFacultySearchHiring.pdf
  * Cornell University, Office of Faculty Development and Diversity (2021) Cornell University, Office of Faculty Development and Diversity. 2021, Best Practices in Faculty Recruitment, https://facultydevelopment.cornell.edu/best-practices-in-faculty-recruitment/
  * Graduate Student Input in the Faculty Search Process (2021) Graduate Student Input in the Faculty Search Process. 2021, living document written by members of CSUs’ Graduate Center for Inclusive Mentoring aned Graduate Student Council and reviewed by the Office of Equal Opportunity, https://graduateschool.colostate.edu/wp-content/uploads/2021/10/Graduate-Student-Input-in-the-Faculty-Search-Process.pdf
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Rutgers University, Department of Sociology (2022) Rutgers University, Department of Sociology. 2022, Student Handbook Appendix B: Guidelines for Graduate Participation in Faculty Recruitment, https://sociology.rutgers.edu/academics/graduate/graduate-student-handbook/
  * University of Washington, Paul G. Allen School of Computer Science & Engineering (2021) University of Washington, Paul G. Allen School of Computer Science & Engineering. 2021, The Faculty Recruiting Process - Grad Student Prospective, https://www.cs.washington.edu/students/grad/gsc/facultyrecruiting/process#grads
  * Van Rossum & Drake (2009) Van Rossum, G., & Drake, F. L. 2009, Python 3 Reference Manual (Scotts Valley, CA: CreateSpace)