will be sent verbatim to the PIs, and they will also be shared with other
reviewers during Stage 2.

  * •

Do not spend time trying to guess who is the proposal team behind the proposal
you are reviewing. Your review should be based solely on the scientific merit
of the proposal. The identity of the team behind it has no relevance for your
review.

7\. Other best practices.

  * •

Do not summarize the proposal: The purpose of the review is to evaluate the
scientific merits of the proposal, not to summarize it. While you may provide
a concise overview of the proposal, it should not constitute the bulk of the
reviews.

  * •

Do not include statements about scheduling feasibility. If there are any
scheduling feasibility issues with the proposal, the JAO will address them
directly with the PI.

  * •

Do not include explicit references to other proposals that you are reviewing,
such as project codes.

  * •

Do not ask questions. A question is usually an indirect way to indicate there
is a weakness in the proposal, but the weakness should be stated explicitly.
For example, instead of “Why was a sample size of 10 chosen?” write “The
proposal did not provide a strong justification for the sample size of 10.”

  * •

Do not use sarcasm or any insulting language.

8\. Re-read your reviews and scientific rankings.

  * •

Once you have completed your assessments, re-read your reviews and ask how you
would react if you received them. If you feel that the reviews would upset
you, revise them.

  * •

Check to see if the strengths and weaknesses in the reviews are consistent
with the scientific rankings. If not, consider revising the reviews or the
rankings.

##  Appendix C Scientific Category/Keyword Pairs

The following list presents the scientific categories, keywords, and their
shorthands which were available to PIs in Cycle 8.

Category 1 – Cosmology and the high redshift universe
a. Lyman Alpha Emitters/Blobs (LAE/LAB)
b. Lyman Break Galaxies (LBG)
c. Starburst galaxies
d. Sub-mm Galaxies (SMG)
e. High-z Active Galactic Nuclei (AGN)
f. Gravitational lenses
g. Damped Lyman Alpha (DLA) systems
h. Cosmic Microwave Background (CMB)/Sunyaev-Zel’dovich Effect (SZE)
i. Galaxy structure & evolution
j. Gamma Ray Bursts (GRB)
k. Galaxy Clusters

Category 2 – Galaxies and galactic nuclei
a. Starbursts, star formation
b. Active Galactic Nuclei (AGN)/Quasars (QSO)
c. Spiral galaxies
d. Merging and interacting galaxies
e. Surveys of galaxies
f. Outflows, jets, feedback
g. Early-type galaxies
h. Galaxy groups and clusters
i. Galaxy chemistry
j. Galactic Centers/nuclei
k. Dwarf/metal-poor galaxies
l. Luminous and Ultra-Luminous Infra-Red Galaxies (LIRG & ULIRG)
m. Giant Molecular Clouds (GMC) properties

Category 3 – ISM, star formation and astrochemistry
a. Outflows, jets and ionized winds
b. High-mass star formation
c. Intermediate-mass star formation
d. Low-mass star formation
e. Pre-stellar cores, Infra-Red Dark Clouds (IRDC)
f. Astrochemistry
g. Inter-Stellar Medium (ISM)/Molecular clouds
h. Photon-Dominated Regions (PDR)/X-Ray Dominated Regions (XDR)
i. HII regions
j. Magellanic Clouds

Category 4 – Circumstellar disks, exoplanets and the solar system
a. Debris disks
b. Disks around low-mass stars
c. Disks around high-mass stars
d. Exoplanets
e. Solar system: Comets
f. Solar system: Planetary atmospheres
g. Solar system: Planetary surfaces
h. Solar system: Trans-Neptunian Objects (TNOs)
i. Solar system: Asteroids

Category 5 – Stellar evolution and the Sun
a. The Sun
b. Main sequence stars
c. Asymptotic Giant Branch (AGB) stars
d. Post-AGB stars
e. Hypergiants
f. Evolved stars: Shaping/physical structure
g. Evolved stars: Chemistry
h. Cataclysmic stars
i. Luminous Blue Variables (LBV)
j. White dwarfs
k. Brown dwarfs
l. Supernovae (SN) ejecta
m. Pulsars and neutron stars
n. Black holes
o. Transients

##  Appendix D PI survey results

This appendix lists the questions and responses for the Cycle 8 PI survey.
Many of the survey questions were adopted from a similar ESO survey in their
pilot review using distributed peer review (Patat et al., 2019). ALMA PIs were
also asked similar questions in the surveys conducted after the Cycle 7 Main
Call which used panel reviews, and the Cycle 7 Supplemental Call with
distributed peer review (see Carpenter et al., 2020). The response rate for
the Cycle 7 surveys were 23% and 70% for the Main and Supplemental calls,
respectively. Where appropriate, we show the survey results from Cycle 8 with
these previous ALMA survey results.

###  D.1 Are the individual comments on your proposal clear and
understandable?

  * •

Fully: The comments are clear, whether I agree with the comments
scientifically or not. (96 responses)

  * •

Mostly: Most of the comments are clear. (281 responses)

  * •

Somewhat: Some of the comments are clear. (87 responses)

  * •

No: Few, if any, of the comments are clear. (9 responses)

Figure 39: Results of the PI survey question asking if the overall comments
are clear and understandable.

###  D.2 Are the comments scientifically accurate?

  * •

Fully: The comments are scientifically accurate. (39 responses)

  * •

Mostly: Most of the comments are scientifically accurate. (239 responses)

  * •

Somewhat: Some of the comments are scientifically accurate. (169 responses)

  * •

No: Few, if any, of the comments are scientifically accurate. (16 responses)

Figure 40: Results of the PI survey question asking if the overall comments
are scientifically accurate.

###  D.3 Will the comments help you to improve future ALMA proposals?

  * •

Fully: The comments will allow me to improve future proposals. (48 responses)

  * •

Mostly: Most of the comments will allow me to improve future proposals. (143
responses)

  * •

Somewhat: Some of the comments might help me strengthen future proposals. (237
responses)

  * •

No: The comments will not help me improve future proposals. (34 responses)

Figure 41: Results of the PI survey question asking if the overall comments
will help improve future proposals.

###  D.4 Were the comments written in a respectful and professional manner?

  * •

Yes: All of the comments were respectful even if I do not agree with the
comments scientifically. (354 responses)

  * •

Somewhat: Most of the comments were respectful. (103 responses)

  * •

No: Many of the comments contain unprofessional remarks. (3 responses)

Figure 42: Results of the PI survey question asking if overall comments were
written in a respectful and professional manner. In the PI survey for the
Cycle 7 main call, the only options were “yes” and “no”.

###  D.5  Are you concerned about confidentiality in ALMA review processes?

  * •

I am neither more nor less concerned about confidentiality issues in the
Distributed Peer Review process compared to the Panel Review process. (158
responses)

  * •

I am more concerned about confidentiality in the Distributed Peer Review
process. (108 responses)

  * •

I am less concerned about confidentiality in the Distributed Peer Review
process. (65 responses)

  * •

I have no strong opinion on this point. (131 responses)

Figure 43: Results of the PI survey question asking if they are more concerned
or less concerned about confidentiality in distributed peer review versus
panel reviews.

###  D.6 Are you concerned about the robustness of ALMA review processes
against any biases?

  * •

I think that the Distributed Peer Review process is as robust against biases
as the Panel Review process. (90 responses)

  * •

I think that the Distributed Peer Review process is more robust against biases
compared to the Panel Review process. (113 responses)

  * •

I think that the Distributed Peer Review process is less robust against biases
compared to the Panel Review process. (161 responses)

  * •

I have no strong opinion on this point. (98 responses)

Figure 44: Results of the PI survey question asking if they are more concerned
or less concerned about biases in distributed peer review versus panel
reviews.

###  D.7 How many years has it been since you obtained your PhD?

  * •

I do not have a PhD yet. (80 responses)

  * •

3 years or fewer. (101 responses)

  * •

Between 4 and 12 years. (147 responses)

  * •

More than 12 years. (133 responses)

Figure 45: Results of the PI survey asking the number of years since they
obtained their PhD.

###  D.8 Please take a few minutes to rate the helpfulness of each review that
you received, indicating the extent to which this comment will help to improve
your proposal in the future. Positive comments like “best proposal I ever
read” can be ranked as not helpful as it does not improve the proposal
further.

  * •

This review is very helpful. (1605)

  * •

This review is somewhat helpful. (3253 responses)

  * •

This review is inaccurate or otherwise not helpful. (1616 responses)

  * •

This review is inappropriate or unprofessional. (162)

Figure 46: Results of the PI survey question asking to rate the helpfulness of
the individual reviews of their proposal.

##  Appendix E Computation of the overall ranks

Each reviewer ranks the proposals in their set from 1 to 10. Nominally each
proposal then will receive 10 ranks. The individual ranks need to be combined
to form an overall ranked list. Two approaches have been investigated to form
this ranked list: (i) averaging the individual ranks and (ii) modeling the
ranks using the Plackett-Luce algorithm. The advantages and disadvantages of
these two approaches are discussed below. For the Cycle 8 main call, the
overall ranks were computed by averaging the individual ranks without outlier
rejection.

###  E.1 Averaging individual ranks

The overall rank of the proposal can be determined by averaging the ranks from
the individual reviewers,121212In practice, the ranks are normalized in the
handful of cases where reviewers rank less than 10 proposals. and then sorting
by the average rank. The main advantage of this approach is the simplicity and
robustness of the calculation. The main disadvantage is that an individual
rank from a reviewer reflects the relative quality of the proposals in their
proposal set. In the case where, for example, a reviewer receives 10 excellent
proposals, one of the proposals will be assigned rank 10 even if the reviewer
found all proposals to be very strong. Similarly, if the reviewer is assigned
10 relatively poor proposals, one will need to be assigned a rank of 1.
Therefore, from the PI perspective, interpreting a rank is ambiguous without
knowing the strength of the other proposals.

###  E.2 Plackett-Luce model

The Plackett-Luce model (Plackett, 1975) overcomes the main disadvantage of
straight averaging by considering the relative strength of the proposal within
a proposal set. Following Turner et al. (2019) and Maystre (2018), consider a
reviewer that has $n$ proposals to review in their proposal set:

$S=\\{i_{1},i_{2},...,i_{n}\\}.$ (E1)

Following Luce’s choice axiom (Luce, 1959), we assume that a reviewer’s
preference for proposal $i_{j}$ over proposal $i_{k}$ is independent of the
remaining proposals in the set $S$. Therefore, the probability of selecting
proposal $i_{j}$ as the top proposal from $S$ is given by

$P(i_{j}|S)={e^{\theta_{i_{j}}}\over\sum_{i\in S}\,\,e^{\theta_{i}}},$ (E2)

where $\theta_{i_{j}}$ represents the “strength” or “worth” of proposal
$i_{j}$. Larger differences in $\theta_{i_{j}}$ then correspond to larger
differences in the perceived scientific merit. The rankings then can be viewed
as a sequence of choices. The top proposal is selected from all proposals in
the set, the second ranked proposal is selected from the remaining proposals,
and so forth until all proposals are ranked. The joint probability of the
specific ranking $i_{1}\succ...\succ i_{n}$ is then

$\displaystyle\mathbb{P}(i_{1}\succ i_{2}\succ...\succ i_{n}|S)$
$\displaystyle\,\,=\,\,$
$\displaystyle{e^{\theta_{i_{1}}}\over{e^{\theta_{i_{1}}}+e^{\theta_{i_{2}}}+...+e^{\theta_{i_{n}}}}}\,\,{e^{\theta_{i_{2}}}\over{e^{\theta_{i_{2}}}+...+e^{\theta_{i_{n}}}}}\,\,...\,\,{e^{\theta_{i_{n-1}}}\over{e^{\theta_{i_{n-1}}}+e^{\theta_{i_{n}}}}}$
(E3) $\displaystyle\,\,=\,\,$
$\displaystyle\prod_{j=1}^{n}{e^{\theta_{i_{j}}}\over{\sum_{i\in
A_{j}}e^{\theta_{i}}}},$ (E4)

where $A_{j}$ is the set of proposals $\\{i_{j},i_{j+1},...,i_{n}\\}$ from
which $i_{j}$ is chosen. From the independent rankings from each of the
reviewers, maximum likelihood techniques can be used to solve for the values
of $\theta_{i}$, and therefore the rank-ordered list of proposals. Priors on
the parameters $\theta_{i}$ can be placed to regulate the solution. In
practice, each parameter was solved with a Gaussian prior with variance
$\alpha^{-1}$. Unless otherwise stated, we set $\alpha=10^{-6}$ to apply loose
constraints on the model parameters. The solutions were obtained with the
python code choix version 0.3.4 (Maystre, 2018).

###  E.3 Simulations

Monte Carlo simulations were used to determine the efficacy of using the
average ranks and the Plackett-Luce model. The basic assumption is that there
is an intrinsic relative scientific merit to the proposals. A “true rank” was
assigned randomly to each proposal. Ten proposals are then assigned to a
reviewer in the simulation following the same proposal assignments used in the
actual Cycle 8 distributed peer review process. The simulation encompasses the
1496 proposals that successfully completed Stage 1. The simulated overall
ranks were then computed by averaging the individual ranks or applying the
Plackett-Luce algorithm.

In conducting the simulations, a reviewer’s ability to order the proposals by
the true rank was set by $\sigma_{\mathrm{ranks}}$. For a given reviewer, the
ranks were randomized using a normal probability distribution with dispersion
$\sigma_{\mathrm{ranks}}$. The randomized ranks were then sorted, which
defined the rank ordered list of proposals for that reviewer. For
$\sigma_{\mathrm{ranks}}=0$, reviewers can perfectly recover the true proposal
rankings. The individual ranks from the reviewers were then averaged without
outlier rejection to derive the overall ranked list of proposals.

Figure 47 shows the results of the simulations. The abscissa in each plot
shows the true proposal rank and the ordinate shows the derived overall rank.
The top row shows the recovered ranks computed by averaging the individual
ranks, and the bottom row shows the ranks recovered from the Plackett-Luce
model. The panels from left to right show the results for
$\sigma_{\mathrm{ranks}}=0$, 100, and 500. The residuals after fitting a
linear relation between the recovered and true ranks are indicated by
$\sigma_{\mathrm{resid}}$.

As shown in the top left panel of Figure 47, when reviewers have the perfect
ability to recover the proposal ranks, the true ranks can be recovered with an
rms of $\sim$5% (=82/1496) when averaging the ranks. This is because a given
reviewer ranks a small subset of all proposals and the true ranks can only be
recovered in the limit of a very large number of reviewers or reviewing all
proposals. For the same simulation, the Plackett-Luce model can recover the
true rankings with an rms of 1.8%, which is three times smaller than straight
averaging, since it can take into account the competition within a proposal
set. However, as the randomness of the reviewer ranks increases (i.e., as
$\sigma_{\mathrm{ranks}}$ increases), the difference between averaging and
Plackett-Luce decreases. For $\sigma_{\mathrm{ranks}}=100$, Plackett-Luce can
recover the true ranks with 1.7 times smaller rms, while for
$\sigma_{\mathrm{ranks}}=500$, the rms of the residuals are essentially the
same.

Figure 47: Monte Carlo simulations that explore the efficiency of averaging
and the Plackett-Luce model to recover the “true rankings” (see text). The top
row shows the results for averaging individual ranks and the bottom row shows
the Plackett-Luce model. The panels from left to right illustrate different
effectiveness in recovering the true proposal ranks as parameterized by the
gaussian dispersion $\sigma_{\mathrm{ranks}}$.

###  E.4 Application to Cycle 8

Figure 48 compares the overall ranks derived from averaging the individual
ranks and the Plackett-Luce model. The Plackett-Luce ranks were derived by
applying weak constraints in the model parameters (left panel) and strong
constraints (right panel). A general correlation between the ranks is seen as
expected. However, if no constraints are applied to the model parameters,
individual proposals in the bottom quartile of the average ranks are found in
the second quartile by the Plackett-Luce model. These extreme outliers are
removed by applying model constraints on the parameters as shown in the right
panel. Identifying the specific reason for the differences in the ranks for
individual proposals is not straightforward since in the Plackett-Luce model,
the rankings depend on the how all of the proposals are rated relative to each
other across all of the proposal sets.

As shown in Section 6.2, the scatter in the individual ranks is dominated by
the individual opinions of the reviewers. In this regime, simply averaging the
ranks recovers the true ranks with similar precision as the Plackett-Luce
model when $\sigma_{\mathrm{ranks}}$ is large (see Section E.3). Given the
scatter in the ranks, and the sensitivity of the Plackett-Luce results in this
regime to the regularization parameter (see Figure 48), we used the simple
averaging of individual ranks to compute the overall ranks in Cycle 8. Further
investigation of the Plackett-Luce algorithm is needed to understand how best
to apply this model in practice.

Figure 48: Correlation in the overall ranks for Cycle 8 distributed peer
review computing using the average ranks without outlier rejection and the
Plackett-Luce model. The Plackett-Luce solutions were obtained with the
regularization parameter $\alpha=10^{-6}$ (left panel) and $\alpha=10$ (right
panel).