# Bridging Learnersourcing and AI: Exploring the Dynamics of Student-AI
Collaborative Feedback Generation

Anjali Singh  University of MichiganUSA<EMAIL_ADDRESS>,  Christopher
Brooks  University of MichiganUSA email ,  Xu Wang  University of MichiganUSA
email ,  Warren Li  University of MichiganUSA email ,  Juho Kim  KAISTSouth
Korea email and  Deepti Wilson  University of MichiganUSA email

###### Abstract.

###### Abstract.

This paper explores the space of optimizing feedback mechanisms in complex
domains such as data science, by combining two prevailing approaches:
Artificial Intelligence (AI) and learnersourcing. Towards addressing the
challenges posed by each approach, this work compares traditional
learnersourcing with an AI-supported approach. We report on the results of a
randomized controlled experiment conducted with 72 Master’s level students in
a data visualization course, comparing two conditions: students writing hints
independently versus revising hints generated by GPT-4. The study aimed to
evaluate the quality of learnersourced hints, examine the impact of student
performance on hint quality, gauge learner preference for writing hints with
or without AI support, and explore the potential of the student-AI
collaborative exercise in fostering critical thinking about LLMs. Based on our
findings, we provide insights for designing learnersourcing activities
leveraging AI support and optimizing students’ learning as they interact with
LLMs.

Learnersourcing, GPT-4, Feedback Generation, Data Visualization

##  1\. Introduction

Timely and effective feedback is essential for supporting learning (Shute,
2008). To overcome the challenges in providing personalized feedback at scale
many tools rely upon Artificial Intelligence (AI) (Balse et al., 2023;
Pankiewicz and Baker, 2023). The advent of conversational Large Language
Models (LLMs) such as GPT-4 (Brown et al., 2020) has fueled new interest in
using AI directly for feedback. However, recent work has shown that providing
LLM-generated feedback to learners can make them over-reliant on such support
(Pankiewicz and Baker, 2023). Moreover, LLMs suffer from the ‘black box’
phenomenon, leading to a lack of transparency and limited interpretability of
their outputs (Yan et al., 2023). Limitations of these models to answer
domain-specific prompts truthfully (Balse et al., 2023) have necessitated
expert oversight for incorporation into educational settings.

An alternative approach for scaling educational feedback is to leverage the
power of the learners themselves, via a method referred to as learnersourcing
(Kim et al., 2015). In this approach learners are engaged in the generation of
educational content or assessments, furthering their own understanding of the
course content while supporting the learning of their peers. Learnersourcing
can be impactful not only for generating learning resources at scale, but also
for the learning benefits it provides to the students who contribute learning
resources, as doing so requires activation of higher-order thinking skills
(Khosravi et al., 2021). However, complex learnersourcing tasks can be
challenging for students, especially when they lack task-specific knowledge
(Singh et al., 2022). Consider the learnersourcing task of writing a hint as a
form of formative feedback to a peers’ incorrect assignment submission: first
the learner must identify the set of mistakes in the incorrect solution and
map those mistakes to misconcenceptions. Then they must determine appropriate
domain-based remediation strategies. Since providing this strategy directly
undermines the hint receiver’s knowledge construction, the learner writing the
hint must provide feedback to address the mistakes without giving away the
full solution. Finally, considerations related to the delivery of the hint
(e.g., tone and specificity) must be considered. In short: writing good hints
is difficult, which can lead to the generation of poor-quality learnersourced
feedback.

In this work, we explore the dynamics of student-AI collaborative hint writing
for generating high quality hints while providing student hint-writers with a
meaningful learning experience. While hints generated by LLMs, when properly
prompted, can be useful for providing task-specific knowledge, it is unclear
how the accuracy and specificity of LLM-generated hints impact the quality of
learnersourced hints. For instance, LLMs often provide verbose rather than
specific responses, which can be distracting to learners. To this end, this
work compares the learnersourcing task of writing hints when learners are
given LLM-generated hints to improve versus when they write hints on their
own.

We situate our research in the domain of data science education with the goal
of generating high-quality formative feedback on students’ incorrect
programming assignment solutions. We conducted a randomized controlled
experiment with 72 students in a Master’s level data visualization course. The
core learning activity in this experiment engaged students in writing a hint
for an incorrect solution to a programming assignment that they had recently
worked on, by comparing it to a correct solution. We had two experimental
conditions: (1) Hint-writing, where students wrote a hint for the incorrect
submission by themselves, and (2) AI-hint-revision, where students wrote a
hint after being shown one generated by the LLM GPT-4 (Brown et al., 2020).
While we engaged in prompt engineering with GPT-4 for this task, we are not
aiming to answer the question of whether LLM-generated hints can be used
directly per se, as the value of learnersourcing goes beyond feedback
generation and provides metacognitive benefits to hint writers. Instead, we
are interested in the interaction between the learners engaged in
learnersourcing and the LLM-generated hint, and thus investigated the
following research questions:

  * •

RQ1. What is the difference between the quality of hints when learners write
them independently versus when they revise a hint written by GPT-4?

  * •

RQ2. How is the quality of learnersourced hints impacted by students’ course
performance?

  * •

RQ3. Which design (with or without AI support) do the students prefer?

  * •

RQ4. To what extent do the students find the AI-hint-revision variant of the
exercise helpful for thinking critically about the accuracy and
appropriateness of responses provided by GPT-4?

This work contributes to our understanding of how student interactions with
AI-based tools, specifically LLMs, impact the outputs of learnersourcing
tasks. As students, teachers, and technology platforms rapidly adopt LLM
technologies, it is important to understand how this adoption modifies the
artifacts of the learning environment (RQ1), whether student performance is a
factor in any changes observed (RQ2), whether student motivational
preferences, and thus their willingness to engage in future learnersourcing
tasks, are affected by such changes (RQ3), and how the AI supported task
itself modifies student perceptions of learning in learnersourcing contexts
(RQ4).

##  2\. Related Work

###  2.1. Reflecting on Mistakes as a Learning Activity

Reflecting on mistakes plays a pivotal role in enhancing memory, attention,
and active learning. (Metcalfe, 2017). Prior research has highlighted the
learning benefits from contrasting incorrect methods with correct ones, such
as focusing students’ attention on the distinguishing features of the correct
examples and the underlying concepts (Durkin and Rittle-Johnson, 2012), and
improving procedural flexibility (Rittle-Johnson and Star, 2011). In addition
to identifying mistakes, the task of hint-writing involves explaining the
mistakes and how to fix them, without giving away the full solution. Prompting
students to explain learning materials not only enhances their comprehension
by promoting deep-level cognitive processes (e.g., organization and
integration of information (Fiorella and Mayer, 2016)), but can also help
elicit metacognitive processes which are conducive to enacting effective
cognitive strategies (Fukaya, 2013). Elaborating on mistakes in domains
prioritizing problem-solving has been found to substantially augment students’
learning outcomes (Loibl and Leuders, 2019). Data visualization, and in
general data science, is one such domain where students concurrently learn
effective ways of problem-solving and communicating data-driven insights.
Actively engaging learners in writing constructive hints for incorrect
solutions to problems they recently solved can encourage them to think deeply
about data science code and compare different approaches to solving the same
problem.

###  2.2. Learnersourcing and AI for Generating Educational Content

Learnersourcing, as defined by Kim (Kim et al., 2015), refers to a
pedagogically supported form of crowdsourcing in which “learners collectively
generate useful content for future learners while engaging in a meaningful
learning experience themselves”. Learnersourcing has been shown to positively
impact learning for both the learners who learn from the learnersourced
artifacts, and the learners who create them (Singh et al., 2022). For
generating hints and explanations at scale, learnersourcing has been leveraged
in several ways. Guo et al. prompted learners write an explanation right after
they overcame a programming struggle (Guo et al., 2020). Glassman et al.
(Glassman et al., 2016) created workflows involving learners reflecting on an
obstacle that they recently overcame and comparing their solution to those of
other students (with both better and worse solutions) to generate a hint for
improving the relatively worse solution. Despite its many benefits,
learnersourcing is impeded by challenges such as lack of motivation among
contributors which can lead to the generation of low quality artifacts (Singh
et al., 2021).

Concurrently, researchers exploring the use of LLMs for generating feedback
have found that LLMs like GPT-3.5 are able to generate helpful personalized
hints for students solving programming assignments, but cautioned that
students may over-rely on such feedback (Pankiewicz and Baker, 2023). Others
have reported issues such as high variability in the accuracy of identified
mistakes and suggested fixes in the generated feedback (Balse et al., 2023).
One potential solution is to leverage human-AI partnerships involving humans-
in-the-loop to both provide inputs to and evaluate and improve the LLM
generated artefacts (Denny et al., 2022). We explore this opportunity by
engaging learners in a data visualization course in evaluating and revising
LLM generated hints for incorrect programming assignment solutions.

##  3\. Method

###  3.1. Course Context and Study Population

The study was conducted in an online Data Visualization course offered to
graduate students as part of an online Masters of Applied Data Science program
by the University of Michigan. This program consists of adult learners (mean
age = 32.50 years, SD = 7.99), many of whom are mid-career professionals
looking to acquire data science skills. Students in this program are required
to have introductory programming and statistics knowledge. The JupyterLab
computational notebook environment (Jupyter, 2022) and Python are used for the
programming component of this course. The course had four weekly programming
assignments that were manually graded by the instructional team.

###  3.2. Experiment Design

We conducted a mixed-model design experiment (consisting of both a within-
subjects and a between-subjects component) with a crossover design. All
students were required to engage in two ‘reflections’ that instructed them to
write a hint for an incorrect solution to a programming assignment that they
had recently completed. For reflection-1 (given in the first week of the
course), approximately half the students were instructed to revise a GPT-4
generated hint for the incorrect solution (AI-hint-revision condition), while
the others were instructed to write a hint independently (Hint-writing
condition). In reflection-2 (given in the second week), the roles reversed:
those who previously didn’t receive the GPT-4 hint now did, and vice versa.
This experiment design was chosen to gauge the difference between the quality
of hints generated by students in the AI-hint-revision versus Hint-writing
conditions, as well as understand each student’s preference of writing hints
with versus without AI support.

###  3.3. Reflection Assignments

Each reflection was graded (worth 2.5% of the total course grade), and was
based on the programming assignment for that week. Students received full
points for completing the reflection, and no points otherwise. In each
reflection, we first described the task to students: identifying the mistakes
in an incorrect solution to a question from the most recent programming
assignment by comparing it against a correct solution, and writing a hint such
that someone who wrote the incorrect solution could use it to identify their
mistakes and fix their code. For each reflection, the same incorrect and
correct solutions were shown to all students. Students in the AI-hint-revision
condition were informed that they would be shown a hint generated by GPT-4 to
help them in this task, and it would be their job to validate if it is correct
and appropriately phrased. Next, students in both conditions were informed of
the potential learning benefits of this exercise, i.e., encouraging them to
think critically, learning from mistakes, and improving their problem-solving
skills. Importantly, the incorrect solution for reflection-1 had one major
mistake (e.g., incorrectly defining the mean and standard deviation when
plotting the t-distribution) and one minor mistake (e.g., not labeling the
axes), while for reflection-2 the incorrect solution had one major and two
minor mistakes111The instructions for each reflection can be viewed here.

#### Design of Hint-Writing Task

As it is recommended to provide students with worked examples of complex
learnersourcing tasks involving multiple skills (Doroudi et al., 2016), we
provided a simple example based on labeling plot axes to demonstrate what they
were expected to do in the hint-writing exercise. After going through the
worked example, students proceeded to the main exercise. The programming
assignment instructions were reproduced and they were shown a correct and an
incorrect solution (which included both the code and the resulting
visualization plot) for that assignment side-by-side. After this, students in
the Hint-writing condition were asked to write a hint for the incorrect
solution, and students in the AI-hint-revision condition were shown the GPT-4
hint and asked to verify and rewrite the hint as required. Learners who were
shown the GPT-4 hint were also warned that it could be incorrect, incomplete,
or both.

###  3.4. Prompting GPT-4 for Hints

To generate the LLM hints for each reflection, we prompted GPT-4222The full
prompt text is available here with almost identical instructions as those
provided to the students to write their hint, including the worked example.
After some experimentation with prompting (for instance, asking GPT-4 to write
concise hints when it produced verbose hints and not referring to the correct
solution provided in the prompt in the hint text) we selected the first hint
that GPT-4 generated to our final prompt.

###  3.5. Data Collection and Outcome Measures

The reflections were created and distributed using the
Qualtrics333https://www.qualtrics.com/ platform. To evaluate the hints written
by learners we qualitatively coded each hint using the criteria described
below. These criteria are based on prior literature on desirable qualities of
formative feedback (Shute, 2008; Thurlings et al., 2013). For each criterion,
we assigned each hint a binary value 0 or 1 (with 1 being better than 0),
except for the metric ‘Utility’ for which we rated each hint on a 1–7 ordinal
scale. The incorrect solutions for each assignment consisted of one major
mistake, which we refer to as the ‘primary mistake’. Accordingly, the first
author defined the criteria for evaluating each hint with the help of another
author. Both authors, who had prior experience in assessing data science
learners, held multiple rounds of discussions to define each criterion and
finalize the codebook. Each author independently coded a random sample of 35%
of all generated hints using the codebook, discussed cases of disagreement,
and iterated thrice on the rubric until strong agreement was achieved (¿= 0.8
Cohen’s Kappa) for each criterion. Before the final round of coding, we
removed any extra text from hints that students had included, such as ”Hint:”
or references to the GPT-4 hint that could be easily separated from the main
hint text (e.g., “The ChatGPT hint is adequate”). We removed the data for
three students who did not write any meaningful hints and wrote text that was
irrelevant to the course material (e.g., ”..”). Finally, the first author
coded all the hints based on the following criteria: (1) Full Accuracy:
Whether all mistakes were identified or not, (2) Partial Accuracy: Whether at
least one mistake was identified or not, (3) Extraneous Information: Whether
the hint contained any information that was not necessary for fixing the
mistakes, (4) Specificity: Whether the hint for the primary mistake was
supported with details without giving away the full solution and explained
how/where the incorrect solution did not meet the assignment goals, (5)
Phrasing: Whether the hint is pedagogically appropriately phrased, using a
positive and encouraging tone rather than a harsh or critical tone, and (6)
Utility, which had the following levels:

  1. (1)

The hint does not cover the primary mistake and requires modification to make
it more specific and/or pedagogically appropriate by improving its phrasing
and/or removing any extraneous information.

  2. (2)

The hint does not cover the primary mistake. Otherwise it is specific and well
phrased.

  3. (3)

The hint covers at least the primary mistake. It requires modification to make
it both more specific, and more pedagogically appropriate by improving its
phrasing and/or removing any extraneous information.

  4. (4)

The hint covers at least the primary mistake. It requires modification to make
it either more specific or improve its phrasing or remove any extraneous
information.

  5. (5)

The hint covers at least the primary mistake. It is specific, well phrased and
has no extraneous information.

  6. (6)

The hint is fully accurate, but requires modification to make it either more
specific or improve its phrasing. Any extraneous information is not
detrimental to students’ learning.

  7. (7)

The hint is fully accurate, specific, appropriately phrased and does not have
any extraneous information

Additionally, we captured the time spent on writing each hint.

###  3.6. Understanding Students’ Learning Experience

We collected students’ responses to three survey questions to learn about
their experience of working on the reflections. After each reflection, we
asked students if they would be interested in engaging in similar exercises in
future courses, with ‘Yes/No’ as the response options. We asked students in
the AI-hint-revision condition to indicate the extent to which the exercise
helped them think critically about the accuracy and appropriateness of
responses generated by GPT-4 on a 5-point Likert scale. Finally, after
reflection-2, we asked them which design (with or with the GPT-4 scaffold)
they preferred. They were also given the option to explain the reasons behind
their preference.

###  3.7. Data Analysis

To understand the effect of the experimental condition and students’ course
performance on the hint quality and time spent, we used: (1) a Generalized
Linear Mixed Effects model for the outcome variables with binary values, (2) a
Linear Mixed Effects model for time spent, and (3) a Cumulative Link Mixed
Model for Utility which had ordinal values in the range 1–7. We measured
students’ course performance based on their scores in the manually graded
programming assignment-2, which reflected their performance in the course at
the time the study was being conducted. Specifically, we looked at the
quartiles in which each students’ assignment-2 scores belonged. In each of the
models, the experimental condition, course performance, and the reflection
number (1 or 2) were used as the fixed effects and the student IDs were used
as the random effects to take into account the variability amongst students.
We also included an interaction term for experimental condition and course
performance to investigate the differential effect of condition on students at
different levels of course performance. All of these analyses were conducted
in R using the lme4 (Bates et al., 2009) and ordinal packages (Christensen,
2018). To analyze the responses to the survey questions regarding students’
learning experience and preference for task design, we only included the data
of those students who completed both reflections. To understand the reasons
behind students’ preference for task design, the first author qualitatively
analyzed their open ended responses, categorized them into themes and refined
the themes after multiple readings of the data.

##  4\. Results

###  4.1. Data Overview

Out of a total of 78 students in the course, 63 and 60 students submitted a
meaningful response to reflection-1 and reflection-2 respectively. 51 students
completed both reflections and 72 students completed at least one reflection.
Table 1 shows the number of students per condition along with the means and
standard deviations for the outcome variables Full Accuracy, Extraneous
Information, Phrasing, Specificity, Utility, and time-spent for each
reflection.

Reflection | Condition | # Students |  | Full Accuracy
---
(0-1)
| Extraneous
---
Info (0-1)
| Phrasing
---
(0-1)
| Specificty
---
(0-1)
| Utility
---
(1-7)
| Time-spent
---
(seconds)
Reflection-1 | Hint-writing | 31 | 0.48 (0.5) | 0.77 (0.43) | 0.77 (0.43) | 0.52 (0.51) | 4.90 (1.68) | 287.82 (413.82)
AI-hint-revision | 32 | 0.66 (0.48) | 0.84 (0.37) | 0.75 (0.44) | 0.59 (0.50) | 5.41 (1.85) | 335.48 (319.37)
GPT-4 hint |  | 1 | 0 | 1 | 1 | 6 |
Reflection-2 | Hint-writing | 36 | 0.19 (0.40) | 0.67 (0.48) | 0.61 (0.49) | 0.33 (0.48) | 3.72 (1.50) | 407.43 (538.93)
AI-hint-revision | 24 | 0.25 (0.44) | 0.46 (0.51) | 0.75 (0.44) | 0.59 (0.50) | 4.00 (1.44) | 515.70 (591.88)
GPT-4 hint |  | 0 | 0 | 1 | 1 | 4 |
Table 1. Number of students per condition and mean (standard deviation) values
for Full Accuracy, Extraneous Information, Phrasing, Specificity (each coded
on ordinal 0-1 scale), Utility (coded on ordinal 1-7 scale) and Time-spent (in
seconds) for each reflection

There was no significant difference based on prior knowledge of students
between conditions as measured by their assignment-1 scores (p=0.30).
Additionally, for each reflection, we calculated the normalized counts (NC) of
students per each tercile444We used terciles as the Assignment-1 scores could
not be distributed into quartiles of Assignment-1 scores ($NC_{1}$, $NC_{2}$,
$NC_{3}$, with $NC_{3}$ denoting the NC of highest performing and $NC_{1}$
denoting the NC of lowest performing students). Based on Assignment-1 score
terciles, students were evenly distributed across each condition for
reflection-1 (AI-hint-revision: $NC_{1}$=0.22, $NC_{2}$=0.28, $NC_{3}$=0.50;
Hint-writing: $NC_{1}$=0.19, $NC_{2}$=0.29, $NC_{3}$=0.52 ). However, the
distribution for reflection-2 was relatively uneven (AI-hint-revision:
$NC_{1}$=0.17, $NC_{2}$=0.33, $NC_{3}$=0.50; Hint-writing: $NC_{1}$=0.25,
$NC_{2}$=0.25, $NC_{3}$=0.50). This was because some students completed only
one of the two reflections.

###  4.2. Study Results

We evaluated the GPT-4 hints based on the criteria described in Section 3.5.
The GPT-4 hint for reflection-1 was fully accurate, specific, well phrased,
but had some extraneous information. Therefore, it was rated 6 for Utility. In
contrast, while the GPT-4 hint for reflection-2 was well phrased and specific,
it was only partially accurate and consisted of extraneous information. Since
this hint identified the primary mistake but required modification to remove
the extraneous information, it was rated as 4 for Utility. Importantly, unlike
the extraneous information in reflection-1’s GPT-4 hint, the extraneous
information in reflection-2’s GPT-4 hint was potentially distracting for
students’ learning. For comparison with the learnersourced hints, Table 1
includes the applicable ratings for each GPT-4 hint.

We now discuss the study results based on the outcome variables described in
Section 3.5. Compared to the students in the Hint-writing condition, students
in the AI-hint-revision condition wrote hints with higher mean values of Full
Accuracy, Specificity, and Utility in each reflection, and higher mean value
of Phrasing in reflection-2 (see Table 1). Regarding Extraneous Information,
students in the AI-hint-revision condition had a higher mean in reflection-1,
but a lower mean compared to the students in Hint-writing condition in
reflection-2. Students in the AI-hint-revision condition also spent more time
writing hints in each reflection. Not all of these results were statistically
significant.

We now report the results for those outcome variables which were significantly
impacted by the factors condition, course performance, and/or the reflection
number555The results of all the statistical analyses are provided in detail
here. We use p¡0.05 as the criterion for assessing statistical significance.
The generalized linear mixed effects model analyses (using binomial family
distribution with a logit link) for the outcome variables with binary values
revealed the following: (i) The effect of reflection number on the log-odds of
Full Accuracy was statistically significant ($\hat{\beta}$ = -1.79, SE = 0.54,
p ¡ 0.01). Post-hoc pairwise comparisons with Tukey’s adjustment confirmed
that students wrote significantly more fully accurate hints in reflection-1
compared to reflection-2 for both the Hint-writing (p=0.001) and AI-hint-
revision (p=0.001) conditions. (ii) Students wrote significantly better
phrased hints in reflection-1 compared to reflection-2 ($\hat{\beta}$ = -6.40,
SE = 2.96, p ¡ 0.05). Additionally, the model suggested that condition had a
significant impact on the log-odds of phrasing ($\hat{\beta}$ = 8.09, SE =
3.39, p ¡ 0.05) and that the interaction between condition and students’
course performance was significant ($\hat{\beta}$ = -5.36, SE = 2.20, p ¡
0.05). However, post-hoc comparisons with Tukey’s adjustment did not reveal a
significant difference (at our threshold of p¡0.05) between conditions for
phrasing when evaluated at the mean value of course performance (p=0.1017).
The Cumulative Link Mixed Model revealed that the hints written for
reflection-1 had significantly higher Utility than those written for
reflection-2 ($\hat{\beta}$ = -1.85, SE = 0.40, p ¡ 0.001). Further, students’
course performance had a significant impact on hint utility ($\hat{\beta}$ =
0.58, SE = 0.24, p ¡ 0.05). Fig. 1 shows the normalized counts of students per
each level of Utility for each reflection. For comparison, the yellow vertical
lines indicate the Utility of the GPT-4 hint for each reflection.

Figure 1. Normalized counts of students per each level of Utility (with ‘7’
representing most usable and ‘1’ representing least usable hints) for each
reflection. The yellow vertical lines indicate the Utility of the GPT-4 hint
for each reflection.

In response to the question “Would you be interested in doing this exercise in
a future course?”, 86.96% students from the Hint-writing condition and 71.43%
students from the AI-hint-revision condition said “Yes” after reflection-1,
and 75% students from the Hint-writing condition and 91.3% students from the
AI-hint-revision condition said “Yes” after reflection-2. For the question,
“How helpful was this exercise in encouraging you to think critically about
the accuracy and appropriateness of responses provided by GPT-4?”, 71.9%
students in reflection-1 and 95.9% students in reflection-2 found the exercise
with GPT-4 hint at least somewhat helpful for thinking critically about LLM
responses. The median ratings were 3 and 4 on a 5 point Likert scale for
reflection-1 and reflection-2, respectively. Fig. 2 shows the distribution of
students’ responses to this question. In response to the question “Did you
like doing this exercise with or without the GPT-4 hint?”, 54.9% students
indicated “With GPT-4” and the remaining indicated “Without GPT-4”.

Based on their open-ended responses regarding the reasons behind their choice,
we found the following reasons in favor of engaging in the hint writing
activity supported by a GPT-4 hint: (1) Students were curious about the
capabilities of GPT-4 and felt that this exercise pushed them to think
critically about its abilities and limitations (“This [exercise] is a great
addition [to the course] because it takes into account the modern reality of
students today and how they can best use the tools available to them to learn
something new”). (2) Students found the GPT-4 hint helpful for writing their
own hint as it provided guidance with framing the hint and identifying the
mistakes in the incorrect code (“I think having a certain direction to go in
based on GPT-4 hint allows me to dive deeper into those concepts with more
specificity and also primes me to do my own work rather than taking GPT-4
hints at face value.”). In contrast, the students who indicated that they
would prefer to write hints without the GPT-4 hint mentioned the following
reasons: (1) Students did not trust GPT-4’s responses and felt that the GPT-4
hint could be misleading (“I feel like GPT-4 sometimes give a hallucinated
codes and answers”). (2) Students felt that the GPT-4 hint biased the way they
thought about writing their own hint (“GPT-4 pigeonholes me into a thought
process that involves everything that GPT-4 has included in their hint, be it
right or wrong”). (3) Students felt that including the GPT-4 hint hinders
learning and would have liked to write a hint without any external help (“The
GPT-4 response takes away from the initial investigation I do to understand
all mistakes in the code. I like doing the initial investigation by myself
first and then maybe seeing a GPT-4 response.”). (4) Students felt that the
GPT-4 hint increased the task complexity (“ChatGPT makes me confused.”).

Figure 2. Students’ responses regarding the extent to which the AI-hint-
revision variant of the exercise helped them think critically about the
accuracy and appropriateness of GPT-4 responses

##  5\. Discussion and Future Work

Regarding RQ1, on the difference between the quality of hints written by
learners with versus without the GPT-4 scaffold, we found that the mean
usability of hints written with the GPT-4 scaffold was higher than the hints
written without it. However, this result was not statistically significant.
This could be due to the current scaffold design or low statistical power
given our chosen threshold of significance (p¡0.05), which is a limitation of
our study. We also found that students who revised the GPT-4 hints wrote
relatively better phrased hints compared to those who wrote it independently.
This result was marginally significant and should be carefully considered in
follow-up investigations. An intuitive finding, Based on Fig. 1 and Table 1,
it is worth noting that most students in the AI-hint-revision condition wrote
hints with Utility values that were close to the GPT-4 hint’s Utility.

To interpret these results, we first need to understand the differences in the
quality of the GPT-4 hints for each reflection. There were fewer mistakes in
reflection-1’s incorrect solution compared to reflection-2. Moreover, the
problem statement for this week’s programming assignment reminded students to
label the axes and the plots appropriately, as this was the first assignment
in the course. In contrast, the problem statement for reflection-2 did not
remind students of these data visualization basics, which made it more
difficult. It is likely that these differences in the problem statements
(which were included as is in the GPT-4 prompts) and their difficulty levels
contributed to the varying levels of usability of GPT-4 hints. Additionally,
the extraneous information in the GPT-4 hint for reflection-2 could have been
distracting for the learners (as mentioned in Section 4.2). This could explain
why, in reflection-1 (where the GPT-4 hint was more usable) relatively more
students were able to improve upon the quality of the GPT-4 hint compared to
reflection-2 (where the GPT-4 hint was less usable), as shown in Table 1. This
suggests that poor quality LLM hints can negatively impact the quality of
learnersourced hints. This raises the need to explore techniques for selecting
LLM hints that pass a given quality threshold, before showing to learners.
Recent work (Phung et al., 2023) showing how educators can be given control
over the quality of LLM-generated feedback before providing to students can be
helpful in this regard.

Regarding RQ2, on the relationship between students’ course performance and
the quality of their hints, we found that higher performing students wrote
more usable hints. This finding is corroborated by prior research highlighting
that higher performing students write more comprehensive and accurate
explanations (Chi et al., 1989). Future research can investigate further how
student performance influences hint quality when written with different
scaffold designs.

Regarding RQ3, on students’ preference for writing hints with or without the
GPT-4 scaffold, we did not find a clear preference for either design. On
probing the reasons behind students’ preferences, students’ lack of trust in
LLMs and willingness to engage in this activity independently, without getting
support or getting biased, emerged as prominent reasons for preferring to
write hints without the GPT-4 scaffold. This could be explained by the
background of students in the data visualization course, who are all adult
learners. The students enrolled in the graduate degree program that this
course is part of typically demonstrate high levels of motivation for learning
data science, as many of them are interested in switching career paths. This
could explain their interest in engaging in this activity without any
additional scaffolding. Their lack of trust in GPT-4 highlights the need for
instructors to elaborate on the steps taken to mitigate any harmful effects of
LLM responses, such as being transparent about the prompting strategies used.

Regarding RQ4 and in terms of student learning, the hint writing activities
gave learners an opportunity to actively engage with LLMs and encouraged the
majority of them to think critically about GPT-4’s abilities. More research is
needed to understand how to design and present such scaffolds to students
based on their motivation levels and abilities, to concurrently help them
learn deeply and write high quality hints.

Taken together, these results suggest that LLM-generated hints can serve as
useful scaffolds for students to write high quality hints, particularly when
the LLM-generated hints are highly accurate. However, it is important to be
careful about the design of such scaffolds. For instance, students can be
given more agency in hint-writing by providing the AI-scaffold after they have
written a hint on their own to provide them with a ‘second opinion’. This
design has been found useful for improving the quality of generated artifacts
in prior research on learnersourcing (Choi et al., 2022) and can be especially
useful for highly motivated learners. Perhaps for learners with a broader
range of motivation levels, such as MOOC learners (Singh et al., 2021), the
AI-scaffold could be provided on demand.

##  6\. Limitations and Conclusion

This study contributes to research on understanding the impacts of LLMs in
education, and how we can optimize student learning as students interact with
LLMs. A limitation of the study is that for the second reflection activity,
students were not evenly distributed across experimental conditions based on
their prior knowledge. Additionally, the number of students needs to be
increased to enhance the statistical power of the study. Finally, we focused
specifically on a single course at a single institution, although, having two
reflections based on two different assignments helped address this limitation
to some extent. Overall, as LLMs continue to become more powerful and accurate
with time (Bubeck et al., 2023), this work has promising implications for
using them to scaffold learnersourcing tasks like feedback generation while
providing students with an opportunity to learn from interacting with them.

## References

  * (1)
  * Balse et al. (2023) Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil Warriem, and Prajish Prasad. 2023.  Investigating the potential of gpt-3 in providing feedback for programming assessments. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_. 292–298.
  * Bates et al. (2009) Douglas Bates, Martin Maechler, Ben Bolker, Steven Walker, Rune Haubo Bojesen Christensen, Henrik Singmann, Bin Dai, Fabian Scheipl, Gabor Grothendieck, Peter Green, et al. 2009\.  Package ‘lme4’.  _URL http://lme4. r-forge. r-project. org_ (2009).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023\.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ (2023).
  * Chi et al. (1989) Michelene TH Chi, Miriam Bassok, Matthew W Lewis, Peter Reimann, and Robert Glaser. 1989.  Self-explanations: How students study and use examples in learning to solve problems.  _Cognitive science_ 13, 2 (1989), 145–182.
  * Choi et al. (2022) Kabdo Choi, Hyungyu Shin, Meng Xia, and Juho Kim. 2022.  Algosolve: Supporting subgoal learning in algorithmic problem-solving with learnersourced microtasks. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Christensen (2018) Rune Haubo B Christensen. 2018.  Cumulative link models for ordinal regression with the R package ordinal.  _Submitted in J. Stat. Software_ 35 (2018).
  * Denny et al. (2022) Paul Denny, Sami Sarsa, Arto Hellas, and Juho Leinonen. 2022.  Robosourcing Educational Resources–Leveraging Large Language Models for Learnersourcing.  _arXiv preprint arXiv:2211.04715_ (2022).
  * Doroudi et al. (2016) Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016.  Toward a learning science for complex crowdsourcing tasks. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. 2623–2634.
  * Durkin and Rittle-Johnson (2012) Kelley Durkin and Bethany Rittle-Johnson. 2012.  The effectiveness of using incorrect examples to support learning about decimal magnitude.  _Learning and Instruction_ 22, 3 (2012), 206–214.
  * Fiorella and Mayer (2016) Logan Fiorella and Richard E Mayer. 2016.  Eight ways to promote generative learning.  _Educational Psychology Review_ 28 (2016), 717–741.
  * Fukaya (2013) Tatsushi Fukaya. 2013.  Explanation generation, not explanation expectancy, improves metacomprehension accuracy.  _Metacognition and learning_ 8 (2013), 1–18.
  * Glassman et al. (2016) Elena L Glassman, Aaron Lin, Carrie J Cai, and Robert C Miller. 2016.  Learnersourcing personalized hints. In _Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing_. 1626–1636.
  * Guo et al. (2020) Philip J Guo, Julia M Markel, and Xiong Zhang. 2020.  Learnersourcing at scale to overcome expert blind spots for introductory programming: A three-year deployment study on the python tutor website. In _Proceedings of the Seventh ACM Conference on Learning@ Scale_. 301–304.
  * Jupyter (2022) Jupyter 2022.  https://jupyter.org/.
  * Khosravi et al. (2021) Hassan Khosravi, Gianluca Demartini, Shazia Sadiq, and Dragan Gasevic. 2021.  Charting the design and analytics agenda of learnersourcing systems. In _LAK21: 11th international learning analytics and knowledge conference_. 32–42.
  * Kim et al. (2015) Juho Kim et al. 2015\.  _Learnersourcing: improving learning with collective learner activity_.  Ph. D. Dissertation. Massachusetts Institute of Technology.
  * Loibl and Leuders (2019) Katharina Loibl and Timo Leuders. 2019.  How to make failure productive: Fostering learning from errors through elaboration prompts.  _Learning and Instruction_ 62 (2019), 1–10.
  * Metcalfe (2017) Janet Metcalfe. 2017.  Learning from errors.  _Annual review of psychology_ 68 (2017), 465–489.
  * Pankiewicz and Baker (2023) Maciej Pankiewicz and Ryan S Baker. 2023.  Large Language Models (GPT) for automating feedback on programming assignments.  _arXiv preprint arXiv:2307.00150_ (2023).
  * Phung et al. (2023) Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023.  Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models.  _arXiv preprint arXiv:2302.04662_ (2023).
  * Rittle-Johnson and Star (2011) Bethany Rittle-Johnson and Jon R Star. 2011.  The power of comparison in learning and instruction: Learning outcomes supported by different types of comparisons.  In _Psychology of learning and motivation_. Vol. 55. Elsevier, 199–225.
  * Shute (2008) Valerie J Shute. 2008.  Focus on formative feedback.  _Review of educational research_ 78, 1 (2008), 153–189.
  * Singh et al. (2022) Anjali Singh, Christopher Brooks, and Shayan Doroudi. 2022.  Learnersourcing in theory and practice: synthesizing the literature and charting the future. In _Proceedings of the Ninth ACM Conference On Learning@ Scale_. 234–245.
  * Singh et al. (2021) Anjali Singh, Christopher Brooks, Yiwen Lin, and Warren Li. 2021.  What’s In It for the Learners? Evidence from a Randomized Field Experiment on Learnersourcing Questions in a MOOC. In _Proceedings of the Eighth ACM Conference on Learning@ Scale_. 221–233.
  * Thurlings et al. (2013) Marieke Thurlings, Marjan Vermeulen, Theo Bastiaens, and Sjef Stijnen. 2013.  Understanding feedback: A learning theory perspective.  _Educational Research Review_ 9 (2013), 1–15.
  * Yan et al. (2023) Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. 2023.  Practical and ethical challenges of large language models in education: A systematic literature review.  _arXiv preprint arXiv:2303.13379_ (2023).


# Paicos: A Python package for analysis of (cosmological) simulations
performed with Arepo

Thomas Berlok  Niels Bohr Institute, University of Copenhagen, Denmark
0000-0003-0466-603X Léna Jlassi  Leibniz-Institut für Astrophysik Potsdam,
Germany  0009-0007-9039-294X Ewald Puchwein  Leibniz-Institut für Astrophysik
Potsdam, Germany  0000-0001-8778-7587 Troels Haugbølle  Niels Bohr Institute,
University of Copenhagen, Denmark  0000-0002-9422-8684

(31 January 2024)

## Summary

Cosmological simulations evolve dark matter and baryons subject to
gravitational and hydrodynamic forces (Vogelsberger et al., 2020). The
simulations start at high redshift and capture hierarchical structure
formation where small structures form first and later assemble to larger
structures (Springel et al., 2005). The Arepo code is a versatile finite-
volume code which can solve the magnetohydrodynamic equations on an
unstructured Voronoi mesh in a cosmologically comoving frame (Springel, 2010;
Weinberger et al., 2020).

Here we present Paicos, a new object-oriented Python package for analyzing
simulations performed with Arepo. Paicos strives to reduce the learning curve
for students and researchers getting started with Arepo simulations. As such,
Paicos includes many examples in the form of Python scripts and Jupyter
notebooks (Kluyver et al., 2016) as well as an online documentation describing
the installation procedure and recommended first steps.

Paicos’ main features are automatic handling of cosmological and physical
units, computation of derived variables, 2D visualization (slices and
projections), 1D and 2D histograms, and easy saving and loading of derived
data including units and all the relevant metadata.

Paicos relies heavily on well-established open source software libraries such
as NumPy (Harris et al., 2020; van der Walt et al., 2011), Scipy (Virtanen et
al., 2020), h5py (Collette, 2013), Cython (Behnel et al., 2011) and astropy
(Astropy Collaboration, 2013) and contains features for interactive data
analysis inside Ipython terminals (Perez & Granger, 2007) and Jupyter
notebooks (Kluyver et al., 2016), e.g., tab completion of data keywords and
Latex rendering of data units. Paicos also contains a number of tests that are
automated using pytest (Krekel et al., 2004), CircleCi and GitHub workflows.

## Statement of need

The Arepo code stores its data output as HDF5 files, which can easily be
loaded as NumPy arrays using h5py. However, data visualization of the
unstructured mesh used in Arepo is non-trivial and keeping track of the units
used in the data outputs can also be a tedious task. The general purpose
visualization and data analysis software package yt111See http://yt-
project.org/. (Turk et al., 2011) and the visualization package py-sphviewer
(Benitez-Llambay, 2015) are both able to perform visualizations of Arepo
simulations. Paicos provides an alternative, which is specifically tailored to
Arepo simulations. It is in this regard similar to swiftsimio (Borrow &
Borrisov, 2020), which was developed specifically for SWIFT simulations
(Schaller & others, 2023).

We have developed Paicos because we identified a need for an analysis code
that simultaneously fulfills the following requirements: 1) is specifically
written for analysis of Arepo simulations discretized on a Voronoi mesh 2)
provides safeguards against errors related to conversions of cosmological and
physical units 3) facilitates working with large data sets by supporting the
saving and loading of reduced/derived data including units and metadata 4)
contains enough functionality to be useful for practical research tasks while
still being light-weight and well-documented enough that it can be installed,
used and understood by a junior researcher with little or no assistance.

## Overview of key Paicos features

The key Paicos features, which were implemented to fulfill the above
requirements, are as follows:

  * •

Functionality for reading cell/particle and group/subhalo catalog data as
saved by Arepo in the HDF5 format.

  * •

Unit-handling via astropy with additional support for the $a$ and $h$ factors
that are used in cosmological simulations.222See e.g. Pakmor & Springel (2013)
and Berlok (2022) for detailed discussions and a derivation of the comoving
magnetohydrodynamic equations used in cosmological simulations including
magnetic fields.

  * •

Functionality for automatically obtaining derived variables from the variables
present in the Arepo snapshot (e.g. the cell volume, temperature, or magnetic
field strength). Within Jupyter notebooks or an Ipython terminal: Tab
completion to show which variables are available for a given data group
(either by directly loading from the HDF5-file or by automatically calculating
a derived variable).

  * •

Functionality for saving and loading data including cosmological and physical
units in HDF5-files that automatically include all metadata from the original
Arepo snapshot (i.e. the Header, Config and Param groups).

  * •

Functionality for creating slices and projections and saving them in a format
that additionally includes information about the size, center and orientation
of the image.

  * •

Functionality for creating 1D and 2D histograms (e.g. radial profiles and
$\rho$-$T$ phase-space plots) of large data sets (using OpenMP-enabled
Cython).

  * •

Functionality for user customization, e.g. adding new units or functions for
computing derived variables.

  * •

Functionality for selecting only a part of a snapshot for analysis and for
storing this selection as a new reduced snapshot.

Finally, we also make public GPU implementations of projection
functionalities, which are much faster than the OpenMP-parallel CPU
implementations described above. The GPU implementations include a mass-
conserving SPH-like projection and a ray tracing implementation using a
bounding volume hierarchy (BVH) in the form of a binary radix tree for nearest
neighbor searches. These GPU implementations use Numba (Lam et al., 2015) and
CuPy (Okuta et al., 2017) and require that the user has a CUDA-enabled GPU.
Our binary tree implementation follows the the GPU-optimized tree-construction
algorithm described in (Karras, 2012) and the implementation of it found in
the publicly available Cornerstone Octree GPU-library (Keller et al., 2023).
Using our SPH-like implementation on an Nvidia A100 GPU, it takes less than a
second to project 100 mio. particles onto an image plane with $4096^{2}$
pixels. This speed enables interactive data exploration. We provide an example
IPython widget illustrating this feature. Finally, we note that the returned
images include physical units and can be used for scientific analysis.

Paicos is hosted on GitHub at https://github.com/tberlok/paicos. We strongly
encourage contributions to Paicos by opening issues and submitting pull
requests.

## Acknowledgements

We thank the JOSS referees, Kyle Oman and Terrence Tricco, and the editor,
Josh Borrow, for constructive feedback that greatly helped improve Paicos. We
are grateful to Christoph Pfrommer for support and advice. We thank Rüdiger
Pakmor, Rosie Talbot and Timon Thomas for useful discussions as well as
Matthias Weber, Lorenzo Maria Perrone, Arne Trabert and Joseph Whittingham for
beta-testing Paicos. We are grateful to Volker Springel for making the Arepo
code available and to the main developers of arepo-snap-util (Federico
Marinacci and Rüdiger Pakmor), a non-public code which we have used for
comparison of projections and slices. TB gratefully acknowledges funding from
the European Union’s Horizon Europe research and innovation programme under
the Marie Skłodowska-Curie grant agreement No 101106080. LJ acknowledges
support by the German Science Foundation (DFG) under grant “DFG Research Unit
FOR 5195 – Relativistic Jets in Active Galaxies”. LJ and TB acknowledge
support by the European Research Council under ERC-AdG grant
PICOGAL-101019746. The authors gratefully acknowledge the Gauss Centre for
Supercomputing e.V. (www.gauss-centre.eu) for funding this project by
providing computing time on the GCS Supercomputer SuperMUC at Leibniz
Supercomputing Centre (www.lrz.de). The Tycho supercomputer hosted at the
SCIENCE HPC center at the University of Copenhagen was used for supporting
this work.

## References

reAstropy Collaboration. (2013). Astropy: A community Python package for
astronomy. _Astronomy and Astrophysics_ , _558_.
https://doi.org/10.1051/0004-6361/201322068

preBehnel, S., Bradshaw, R., Citro, C., Dalcin, L., Seljebotn, D. S., & Smith,
K. (2011). Cython: The Best of Both Worlds. _Computing in Science and
Engineering_ , _13_(2), 31–39. https://doi.org/10.1109/MCSE.2010.118

preBenitez-Llambay, A. (2015). _Py-sphviewer: Py-SPHViewer v1.0.0_ (Version
v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.21703

preBerlok, T. (2022). Hydromagnetic waves in an expanding universe -
cosmological MHD code tests using analytic solutions. _Monthly Notices of the
RAS_ , _515_(3), 3492–3511. https://doi.org/10.1093/mnras/stac1882

preBorrow, J., & Borrisov, A. (2020). Swiftsimio: A python library for reading
SWIFT data. _Journal of Open Source Software_ , _5_(52), 2430.
https://doi.org/10.21105/joss.02430

preCollette, A. (2013). _Python and HDF5_. O’Reilly.

preHarris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen,
P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R.,
Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Rio, J.
F., Wiebe, M., Peterson, P., … Oliphant, T. E. (2020). Array programming with
NumPy. _Nature_ , _585_(7825), 357–362.
https://doi.org/10.1038/s41586-020-2649-2

preKarras, T. (2012). Maximizing parallelism in the construction of BVHs,
octrees, and k-d trees. _Proceedings of the Fourth ACM SIGGRAPH / Eurographics
Conference on High-Performance Graphics_ , 33–37.
https://doi.org/10.2312/EGGH/HPG12/033-037

preKeller, S., Cavelan, A., Cabezon, R., Mayer, L., & Ciorba, F. M. (2023).
Cornerstone: Octree Construction Algorithms for Scalable Particle Simulations.
_arXiv e-Prints_ , arXiv:2307.06345. https://doi.org/10.48550/arXiv.2307.06345

preKluyver, T., Ragan-Kelley, B., Pérez, F., Granger, B., Bussonnier, M.,
Frederic, J., Kelley, K., Hamrick, J., Grout, J., Corlay, S., Ivanov, P.,
Avila, D., Abdalla, S., Willing, C., & team, J. development. (2016). Jupyter
notebooks – a publishing format for reproducible computational workflows. In
F. Loizides & B. Scmidt (Eds.), _Positioning and power in academic publishing:
Players, agents and agendas_ (pp. 87–90). IOS Press.
https://doi.org/10.3233/978-1-61499-649-1-87

preKrekel, H., Oliveira, B., Pfannschmidt, R., Bruynooghe, F., Laugher, B., &
Bruhin, F. (2004). _Pytest 7.4_. https://github.com/pytest-dev/pytest

preLam, S. K., Pitrou, A., & Seibert, S. (2015). Numba: A LLVM-based python
JIT compiler. _Proceedings of the Second Workshop on the LLVM Compiler
Infrastructure in HPC_. https://doi.org/10.1145/2833157.2833162

preOkuta, R., Unno, Y., Nishino, D., Hido, S., & Loomis, C. (2017). CuPy: A
NumPy-compatible library for NVIDIA GPU calculations. _Proceedings of Workshop
on Machine Learning Systems (LearningSys) in the Thirty-First Annual
Conference on Neural Information Processing Systems (NIPS)_.
http://learningsys.org/nips17/assets/papers/paper_16.pdf

prePakmor, R., & Springel, V. (2013). Simulations of magnetic fields in
isolated disc galaxies. _Monthly Notices of the RAS_ , _432_(1), 176–193.
https://doi.org/10.1093/mnras/stt428

prePerez, F., & Granger, B. E. (2007). IPython: A System for Interactive
Scientific Computing. _Computing in Science and Engineering_ , _9_(3), 21–29.
https://doi.org/10.1109/MCSE.2007.53

preSchaller, M., & others. (2023). Swift: A modern highly-parallel gravity and
smoothed particle hydrodynamics solver for astrophysical and cosmological
applications. _arXiv e-Prints_ , arXiv:2305.13380.
https://doi.org/10.48550/arXiv.2305.13380

preSpringel, V. (2010). E pur si muove: Galilean-invariant cosmological
hydrodynamical simulations on a moving mesh. _Monthly Notices of the RAS_ ,
_401_(2), 791–851. https://doi.org/10.1111/j.1365-2966.2009.15715.x

preSpringel, V., White, S. D. M., Jenkins, A., Frenk, C. S., Yoshida, N., Gao,
L., Navarro, J., Thacker, R., Croton, D., Helly, J., Peacock, J. A., Cole, S.,
Thomas, P., Couchman, H., Evrard, A., Colberg, J., & Pearce, F. (2005).
Simulations of the formation, evolution and clustering of galaxies and
quasars. _Nature_ , _435_(7042), 629–636. https://doi.org/10.1038/nature03597

preTurk, M. J., Smith, B. D., Oishi, J. S., Skory, S., Skillman, S. W., Abel,
T., & Norman, M. L. (2011). yt: A Multi-code Analysis Toolkit for
Astrophysical Simulation Data. _The Astrophysical Journal Supplement Series_ ,
_192_ , 9. https://doi.org/10.1088/0067-0049/192/1/9

prevan der Walt, S., Colbert, S. C., & Varoquaux, G. (2011). The NumPy Array:
A Structure for Efficient Numerical Computation. _Computing in Science and
Engineering_ , _13_(2), 22–30. https://doi.org/10.1109/MCSE.2011.37

preVirtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T.,
Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der
Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R.
J., Jones, E., Kern, R., Larson, E., … SciPy 1. 0 Contributors. (2020). SciPy
1.0: fundamental algorithms for scientific computing in Python. _Nature
Methods_ , _17_ , 261–272. https://doi.org/10.1038/s41592-019-0686-2

preVogelsberger, M., Marinacci, F., Torrey, P., & Puchwein, E. (2020).
Cosmological simulations of galaxy formation. _Nature Reviews Physics_ ,
_2_(1), 42–66. https://doi.org/10.1038/s42254-019-0127-2

preWeinberger, R., Springel, V., & Pakmor, R. (2020). The AREPO Public Code
Release. _Astrophysical Journal, Supplement_ , _248_(2), 32\.
https://doi.org/10.3847/1538-4365/ab908c

p