# Game Elements to Engage Students Learning the Open Source Software
Contribution Process

Italo Santos1, Katia Romero Felizardo12, Marco A. Gerosa1, and Igor
Steinmacher1 Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>1Northern Arizona University,
Flagstaff, AZ, USA
2Federal University of Technology, Paraná, PR, Brazil

###### Abstract

Contributing to OSS projects can help students to enhance their skills and
expand their professional networks. However, novice contributors often feel
discouraged due to various barriers. Gamification techniques hold the
potential to foster engagement and facilitate the learning process.
Nevertheless, it is unknown which game elements are effective in this context.
This study explores students’ perceptions of gamification elements to inform
the design of a gamified learning environment. We surveyed 115 students and
segmented the analysis from three perspectives: (1) cognitive styles, (2)
gender, and (3) ethnicity (Hispanic/LatinX and Non-Hispanic/LatinX). The
results showed that Quest, Point, Stats, and Badge are favored elements, while
competition and pressure-related are less preferred. Across cognitive styles
(persona), gender, and ethnicity, we could not observe any statistical
differences, except for Tim’s GenderMag persona, which demonstrated a higher
preference for storytelling. Conversely, Hispanic/LatinX participants showed a
preference for the Choice element. These results can guide tool builders in
designing effective gamified learning environments focused on the OSS
contributions process.

keywords: gamification, game elements, human factors, cognitive styles, human-
computer interaction.

##  I Introduction

Open Source Software (OSS) projects are not just code repositories, they are
evolving ecosystems where learning and collaboration converge. OSS initiatives
receive a worldwide network of contributors partly due to the growing
recognition of the value that experience with OSS projects offers to
professionals [1]. For students, in particular, contributing to OSS projects
is an effective way to develop practical skills, gain real-life experience,
and boost their employability [2, 3, 4].

Given the importance of providing students with practical skills, educators
integrate OSS projects into their curriculum to expose students to authentic
software development challenges [5, 6, 7]. However, students new to OSS,
especially those from diverse backgrounds, including different cognitive
styles, genders, and ethnicities, might feel demotivated from participating
due to barriers that prevent contributions from newcomers [8, 9, 10, 11].
Engaging and motivating students to contribute to OSS projects is a challenge
[12, 13, 14].

A potential strategy to engage students is employing gamification, which
increases motivation by integrating game elements into a system [15]. Through
gamification, students are more likely to remain committed, find joy in their
contributions, and stand to benefit educationally [16, 17, 18]. For example,
Diniz et al. [12] implemented gamification strategies to increase motivation
and facilitate collaboration among undergraduate students in OSS projects
using GitLab. They evidenced that game elements, such as quests and points,
kept students engaged and aided in orienting and motivating them to contribute
to the projects. Nevertheless, no studies have examined the appropriateness of
a comprehensive set of game elements designed to foster student engagement in
learning the OSS contribution process.

Our study investigates students’ perceptions of game elements for this
purpose. We particularly focus on understanding how the perception changes
across diverse cognitive styles, gender identities, and ethnic backgrounds.

The following research questions guided our investigation:

Research Question 1 Which game elements do students prefer to engage in
learning the OSS contribution process?

Research Question 2 How do students’ preferences of game elements differ
according to their cognitive style, gender, and ethnicity?

To answer our research questions, we surveyed 115 students and identified
their preferences for various game elements. We designed a survey to collect
responses from students majoring in computer science and related fields (e.g.,
applied computer science, software engineering, information systems, and
computer engineering). The findings from this survey can inform future
targeted experimental research, where specific game elements can be
implemented and their effects directly measured. While integrating actual game
elements into the open-source tools would be informative, this study primarily
aims to explore the receptiveness to such mechanisms. Understanding whether
and which game elements are perceived as valuable is a necessary precursor to
their practical application. Gathering potential end users’ perceptions before
implementing a tool not only contributes to usability and satisfaction but
also offers valuable insights from the user-centered design perspective.

Our findings suggest that game elements characterized by performance-oriented
features, such as Stats, Maps, Levels, Points, Progress Bars, and Badges, are
preferred for student engagement. These elements not only provide informative
feedback on performance but also contribute to the development of a sense of
progression and achievement. With a few exceptions, we did not find much
difference across demographics. We expect that our insights on students’
preferences for game elements will help the research community create gamified
learning environments focused on the OSS contribution process.

##  II Background and Related Work

According to Ghai and Tandon [19], gamification has become a popular topic
among educators due to recent technological advancements and a growing
interest in innovative educational methods. Gamification gained popularity due
to its ability to engage and inspire students, helping them achieve a state of
flow throughout the learning process [20]. Dichev and Dicheva [15] describe
gamification as a strategy to enhance student motivation and engagement in
educational settings by integrating game elements. Gamification leverages game
elements’ inherent motivational appeal and immersive qualities to encourage
student participation in the learning process. Deterding et al. [21] suggest
that gamification simplifies complex material, making learning more
approachable and facilitating a deeper understanding of the subject matter.
They consider that games can transform educational experiences, making them
more interactive, enjoyable, and effective for students.

Zhao et al. [22] focused on encouraging contributions to OSS through a network
algorithm called OpenRank and a monthly updated contribution leaderboard. The
results show that gamified leaderboards can impact OSS developers towards
better collaboration and a more harmonious community atmosphere with social
connections. Kapp [23] salients the transformative power of gamification,
observing that, when implemented effectively, it can captivate and educate
students. Huotari and Hamari [24] further assert that gamification, when
utilized as an educational tool, extends beyond merely expanding student
knowledge; it also enhances their collaborative and communicative skills. In
education, gamification introduces game-based rules, immersive player
experiences, and cultural contexts, which collectively influence and modify
student behavior [25].

Recent studies have explored the application of gamified environments to
software engineering education. Su [26] created a gamified framework to
examine the impact of gamification on teaching software engineering. The
findings indicated that students were more motivated by gamified teaching
methods, which enhanced academic outcomes. Similarly, Sheth et al. [27]
demonstrated that introducing gamification to a software engineering course
could boost student engagement in various aspects of the development process,
including documentation, bug reporting, and testing activities.

Diniz et al. [12] implemented gamification strategies to motivate and support
student collaboration in OSS projects using GitLab. Their findings highlighted
that game elements, such as quests and points, kept students engaged and aided
in orienting and motivating them to contribute to OSS projects. Toscani et al.
[28] demonstrated that gamification can effectively engage a diverse range of
newcomers. Moreover, gamification has been successfully applied to various
domains within software engineering education, including agile process
training [29], mutant testing [30], and the learning of design patterns [18],
Graphical User Interface (GUI) testing [31], cybersecurity training [32],
exploratory testing [33], and design patterns [18], demonstrating its wide-
ranging applicability and potential to enrich the educational experience in
this field.

Our study enhances the existing body of research by exploring student
perceptions regarding implementing game elements in a gamified learning
environment specifically designed to educate about the OSS contribution
process. We uncover student preferences for specific game elements to present
a nuanced understanding of how various game elements can be considered among
student profiles, thereby informing the development of more inclusive and
effective gamified educational tools.

##  III Research Method

This study investigates students’ perceptions of game elements in the context
of learning the OSS contribution process. In this section, we detail our
method.

###  III-A Survey design

Our survey consisted of four main sections: (Part I) scenario explanation;
(Part II) game elements; (Part III) GenderMag facets; and (Part IV)
demographics. We used open, multiple-choice, and 5-Likert scale questions
(from 1– strongly disagree to 5–strongly agree). We implemented the survey in
Qualtrics111www.qualtrics.com.

Below, we briefly explain the survey sections:

Part I – Scenario Explanation: Respondents were directed to a consent page
that explains the study’s objectives, the rules regarding confidentiality, the
estimated time commitment required to complete the survey, and contact
information for the researchers. After that, to guide participants in
providing relevant and context-aware responses, we offer a scenario to
consider while answering questions related to game elements in the survey: You
are a student in a university class using an interactive environment to learn
how to contribute to Open Source Software (OSS) projects. You must accomplish
tasks such as exploring projects, collaborating with fellow learners,
reviewing pull requests, making non-code contributions (e.g., documentation,
translation, etc.), and establishing connections with the community.

Part II – Game Elements: This section was designed to collect information on
the game elements students perceive as engaging within a gamified learning
environment focused on teaching the process of contributing to OSS. We used
Toda et al. [34] taxonomy as our framework, which focuses on gamification
strategies in educational contexts. The game elements included in our survey
were selected from the taxonomy during design sessions, which focused on
representing the game elements in the context of the OSS contribution process.
During these sessions, we considered the trade-offs of including additional
dimensions and implementations with the need to keep the instrument short.
Toda et al.’s taxonomy is structured into five dimensions [34], as outlined
below:

  * •

Performance – This dimension is related to responses from the environment that
serve as feedback mechanisms for the learner, including elements such as
points, progress bars, levels, stats, and acknowledgments, i.e., badges.

  * •

Ecological – The elements of this dimension, including choice, economy, and
time pressure, act as properties of the environment to engage learners to
follow a desired behavior.

  * •

Social – Concerning this dimension, the game elements involve interactions
among learners within the environment, including cooperation, reputation,
social pressure, and competition, i.e., leaderboards, private rank, and
competitive tasks.

  * •

Personal – The elements of this dimension focus on the individual learner’s
engagement with the environment, featuring aspects such as objectives (e.g.,
quests), puzzles, novelty, and renovation.

  * •

Fictional – Regarding this dimension, the elements connect the user to the
environment through user-oriented narratives and environment-oriented
storytelling, intertwining their experience with the context.

Table I describes the game elements and the questions used in the survey.

Still, for each game element, we included an optional open-ended question on
the survey — Please share any additional thoughts or details to justify your
rating. — This was done to gather insights into why participants gave specific
ratings for each game element. Furthermore, we asked participants to rank the
top three most relevant they would like to see while using a gamified learning
environment.

Table I: Game elements and their corresponding survey questions | Taxonomy
---
dimensions
| Game
---
elements
Survey questions
Performance | Point |  | After completing tasks, you receive experience points proportionally to your performance. You will likely come back
---
because you like gaining experience points.
Level |  | As you complete tasks, you gain levels, and increasing your level unlocks new and more complex tasks (e.g., you can
---
only submit a contribution if you are at level “5”). You will likely come back
because you like increasing your level.
Badges |  | You receive a badge as an award for successfully meeting specific requirements (e.g., a “Flash” badge when you finish
---
tasks before their time limit, etc.). You will likely come back because you
like conquering more badges.
Progress Bar |  | You can see a progress bar with the accumulated experience points and the remaining points required to attain the
---
next level. You will likely come back because you like visualizing how many
tasks you still have to complete to keep
progressing.
Map |  | As you advance in your learning journey, you see a visual map of your learning progress, including completed and
---
remaining tasks you must accomplish. You will likely come back because you
like visualizing an overall map with all
the tasks you have (and have not) completed.
Stats |  | When you interact with the learning environment, you can visualize your learning results and other relevant
---
information (e.g., current level, experience points (XP), tasks completed,
badges, etc.). You will likely come back
because you like visualizing your overall stats.
Ecological | Choice |  | You can choose the tasks you want to work on in the learning environment, such as editing a file or reviewing a
---
peer’s contribution. You will likely come back because you like choosing a
task to work on.
Economy |  | You can do extra tasks (e.g., quizzes) to accumulate special items (e.g., gems), and you can trade the special items
---
received for advantages, such as help from the instructor/assistant to
complete a task. You will likely come back
because you like trading items for advantages.
| Time
---
Pressure
| You have to complete a task (e.g., submit the assignment) in the given
timeframe. Once the task has been opened,
---
the clock starts counting backward. If you complete the task before the
specific time, you will be rewarded. You will
likely come back because you like doing tasks with time pressure.
Social | Cooperation |  | You can perform collaborative tasks, i.e., tasks involving other peers. You will likely come back because you like
---
working on collaborative tasks.
| Social
---
Pressure
| The learning environment exposes your task results for other learners to
peer-review. You will likely come back
---
because you like to show your work to other learners.
| Public
---
Leaderboard
| In the learning environment, you see a public leaderboard and can compare
your performance to others (e.g., a public
---
leaderboard with a ranking of all learners in the environment). You will
likely return because you like improving
your performance based on the public leaderboard.
Private Rank |  | You see your position in a private rank compared to your peers (e.g., three learners have more points than you; you
---
are 5 points behind the 3rd position). You will likely return because you like
improving your position in the
ranking.
| Competitive
---
Task
| You engage in tasks competitively with other learners to determine who
achieves the best results (e.g., you have to
---
finish a before all other learners). You will likely come back because you
like completing tasks and competing
against other learners.
Personal | Quest |  | The environment offers quests to guide your learning journey. Each quest involves tasks such as editing a file.
---
You will likely come back because you like accomplishing quests.
Puzzle |  | You can perform challenging tasks related to the content learned (e.g., contribute to a project outside of the learning
---
environment) to measure your learning. You will likely come back because you
like being challenged with tasks
related to the content learned.
Renovation |  | You can redo a task that you fail, such as committing your changes with clearer commit messages, to increase your
---
score or experience points. You will likely come back because you like redoing
a failed task.
Novelty |  | The learning environment offers new content, such as unlocking new tasks, including novelty, as you advance in
---
your learning process. You will likely come back because you like having new
content.
Fictional | Narrative |  | The learning environment presents a dystopic narrative (e.g., about an individual who needs to learn about the OSS
---
process to improve the quality of life in society). You must interact with the
environment to advance in the storyline.
You will likely come back because you like having a narrative.
Storytelling |  | The learning environment shows the storytelling using images and text related to the narrative. You will likely come
---
back because you like storytelling.

Part III – GenderMag Facets: In this section, participants were asked how they
behave when they approach unknown technology. The questions were used to
evaluate participants’ GenderMag [35] facets. GenderMag is a systematic
inspection method that captures individual differences in how people solve
problems and use software features. GenderMag is based on research showing
that individual differences in cognitive styles (facets) cluster by gender
[36, 37, 38]. The method encapsulates the facets into personas (Abi and Tim)
related to (i) Motivation: Abis are motivated to use technology for what they
can accomplish with it, whereas Tims are often motivated by their enjoyment of
technology per se; (ii) Information processing styles: Abis process new
information comprehensively–gathering fairly complete information before
proceeding–but Tims use selective information processing–following the first
promising information, then backtracking if needed; (iii) Computer self-
efficacy: relates to a person’s confidence about succeeding at a specific
task, which influences their use of cognitive strategies, persistence, and
strategies for coping with obstacles. Abis have lower computer self-efficacy
as compared to their peers; (iv) Risk aversion: Abis are risk-averse when
trying out new features as compared to Tims, which impacts their decisions
about which feature sets to use; and (v) Learning: by Process vs. by
Tinkering: Abis prefer process-oriented learning, whereas Tims, like to
experiment (“tinker”) with software features new to them. Each cognitive style
has advantages, but it is also at a disadvantage when not supported by the
software.

In summary, Abi and Tim are in the opposite spectrum of facet values, with the
Abi persona aligned with facet values that women tend to favor and Tim
embodying facet values typically favored by men. We selected five questions
from the complete GenderMag to make it more manageable by shortening the
number of questions. We selected these questions by applying the variance
inflation factor (VIF) on the data of a previous study [37] that collected
participants’ information using the complete GenderMag questionnaire to reduce
the Multicollinearity [39]. Based on participants’ answers, we could
characterize participants’ cognitive styles according to the facets of each
GenderMag persona (e.g., Tim or Abi).

Part IV – Demographics: Participants were asked to provide information about
age, the country where they live, gender, ethnicity, the year in college they
are enrolled in, if they have taken a course about OSS, and years of
experience working in software development, games, and gamified learning
environments.

The survey was anonymous. The order of the game elements that appear in the
response options was randomized to mitigate response order effects [40]. All
open questions were optional to increase the response rate by making
respondents more comfortable [41]. Furthermore, an attention check question
was also included to ensure that participants read instructions carefully
[42]: “This is an attention check; please mark the option Strongly Disagree.”

###  III-B Pilot

After we designed and proofread the survey, we tested it in multiple browsers
and devices. We invited five participants to pilot the survey to ensure the
survey language was appropriate to our target population, collect feedback,
and measure the survey response time. We had three pilot sessions with
undergraduate computer science students that fit the respondent profile. In
those sessions, we validated if the students could understand the questions
well. The participants suggested some language improvements to make the survey
more straightforward. One example was the initially proposed question for the
level element: You will likely come back because you like gain levels. During
the pilot, the participants mentioned that the word increase is more common in
a game context than gain. After the improvements made in the survey from this
first interaction, we conducted the pilot study with two open-source
specialists. We received more feedback about the content presented in the
survey. Based on the feedback, we made some changes to improve our questions,
dropping some questions to reduce the survey length and including some new
questions to validate that the participants are part of our target population.

###  III-C Recruitment

The respondents were recruited using Prolific222www.prolific.co, a
crowdsourcing platform where researchers can channel their surveys [43].
Prolific keeps the demographic information of all its users to facilitate the
pre-selection process. We could choose some pre-screening characteristics to
narrow down the population to our relevant populations. We focused on
undergrad students taking any CS-related major. We created four recruitment
surveys on Prolific, with the following extra criteria based on our population
of interest: (i) Men and Hispanic/LatinX; (ii) Non-Men and Hispanic/LatinX;
(iii) Men and Non-Hispanic/LatinX; (iv) Non-Men and Non-Hispanic/LatinX. We
focused on Hispanic/LatinX population because it is chronically
underrepresented in computer science and OSS [44]. Our goal was to explore how
cultural differences might correlate with preferences for certain game
elements. We aimed for a balanced distribution to compare responses across
different groups.

We included screening questions at the beginning of our survey to check that
the respondents still hold the same demographic information as when they
filled out their information to Prolific. Therefore, we asked the same
questions asked by the prolific platform that we used to filter the
population. After answering the pre-screening validation section, respondents
could access the complete survey (in case they were still part of the group of
interest).

The survey was conducted from February 15 to March 10, 2024, and participants
were paid $3.0 through the Prolific platform after completing the survey. We
received 158 responses that were not blank. Following a thorough review,
detailed in the subsequent subsection, we identified 115 valid responses for
our analysis.

###  III-D Filtering

We carefully reviewed and filtered our data to consider only valid responses.
We dropped answers that failed the attention check question (five cases) and
checked for answers with the same choice for all Likert scale questions (0
removed). We removed participants who did not complete the survey (16 cases)
and those who failed the prescreening questions (22 cases). We then analyzed
the time to complete the survey to remove lower outliers (0 removed). After
this filtering process, we end up with 115 valid responses for analysis in our
dataset.

###  III-E Data analysis

1) Likert-scale items: We asked the level of agreement of participants using
Likert-scale items (ranging from 1 representing strongly disagree to 5
representing strongly agree) for a set of game elements. Then, participants
ranked the top three game elements they would like to see in a learning
environment proposed. We used descriptive statistics and visually analyzed the
data. We segmented our sample based on cognitive styles (persona: Abi and
Tim), gender (man and minorities — women and non-binary), and ethnicity
(Hispanic/LatinX and Non- Hispanic/LatinX) and checked the odds ratios to
evaluate the interest of subgroups and Mann-Whitney U test to analyze the
differences in game element preferences across them.

2) Open questions: We used open questions to ask for more information about
participants’ perceptions of the game elements. We qualitatively analyzed
participants’ comments following open coding procedures [45]. The process was
conducted using continuous comparison and discussion until reaching a
consensus. Two researchers jointly analyzed the sets of answers to establish
common ground, discussing the applied codes and disagreements until reaching a
consensus. Finally, a third researcher inspected the classification.

The replication package, with the anonymized dataset, instruments, and
scripts, is available for public access
333https://figshare.com/s/5b5a16dcb17d16781bae.

##  IV Participant Demographics

The demographics of our participants are summarized in Table II. Our
demographics show a gender distribution with the majority identifying as men
(73%), followed by women (31%), and a smaller representation of non-binary
respondents (5%). Concerning ethnic identity among our respondents, we have
49% of respondents as Hispanic/LatinX, with the remaining 51% not identifying
as Hispanic/LatinX. As expected, the age distribution of our survey
participants indicates a young demographic, with 48% being 24 or below, 39%
between the ages of 25 and 34, and smaller percentages distributed among older
age groups. In terms of academic major, the largest group of respondents were
studying computer science (43%), followed by other computing-related
majors/programs (17%), information technology (15%), and software engineering
(13%), among others. Looking at their academic journey, 37% were in their
$4^{th}$ or $5^{th}$ year, indicating a senior level of study among the
respondents. The survey also touched upon the GenderMag personas, where 54%
aligned with the persona “Abi” and 46% with “Tim”, indicating a diverse range
of user perspectives.

Table II: Personal characteristics of respondents (N=115) Demographics | # | %
---|---|---
Gender
Man | 73 | 64%
Woman | 36 | 31%
Non-binary | 6 | 5%
Ethnicity
Hispanic/LatinX | 56 | 49%
Not Hispanic/LatinX | 59 | 51%
Age
24 or below | 56 | 48%
25 to 34 | 45 | 39%
35 to 44 | 10 | 9%
45 to 54 | 3 | 3%
55 to 64 | 1 | 1%
Year of Study
1st year (freshman) | 18 | 16%
2nd year (sophomore) | 22 | 19%
3rd year (junior) | 27 | 23%
4th or 5th year (senior) | 42 | 37%
Graduate studies | 6 | 5%
GenderMag Persona
Abi | 62 | 54%
Tim | 53 | 46%

##  V Results

###  V-A Students’ perception about game elements

To answer RQ1: Which game elements do students prefer to engage in learning
the OSS contribution process?, we asked participants Likert-scale items to
rate their preference. Participants also ranked their top three game elements
and expressed their rationale through open questions.

Students preferences for game elements. Figure 1 shows the answers to the
Likert-scale items. Our findings indicate that the respondents agree (or
strongly agree) that Performance dimension elements contribute to their
engagement. Those elements refer to stats and maps (80%), levels (79%), points
(76%), progress bars (75%), and badges (70%). Participants underscore the
significance of such feedback in enhancing task completion, as stated by P13:
“Gaining experience to gain levels makes me feel like I’m working towards
something important.”. Toda et al. [34] emphasize that the performance
dimension is critical in providing feedback for students.

Figure 1: Responses to the 5-point Likert-scale items for the game elements.
The left hand (yellow) shows levels of disagreement, the middle (grey) shows
neutral, and the right (green) shows levels of agreement.

Additionally, respondents agree that Personal dimension elements contribute to
their engagement with the environment. Those elements include novelty (82%),
puzzles (79%), quests (77%), and renovation (77%), with P41 explaining that:
“I appreciate having new content introduced in the learning environment
because it keeps me engaged and motivated by offering fresh challenges and
opportunities for growth.”.

Within the Ecological dimension, 72% of respondents agree that the choice
element allows students to tailor their experience by optimizing their
interaction with the environment based on their preferences and available
time. For instance, P10 highlighted the practicality of this element:
“choosing tasks is useful, so you can measure the cost-benefit of doing them
based on the time available.”. Furthermore, while 65% of participants agree
(or strongly agree) that they enjoy trading items for advantages, 21% disagree
(or strongly disagree). P13 expressed enjoyment and a sense of achievement
from collecting items. In contrast, P31 raised concerns about fairness,
suggesting, “I agree the use of special items to trade for special advantages
outside a learning environment, but these advantages should be available to
everyone in a learning environment and not only to those who can pay for it”.

Relative to the Social dimension and their elements, while most participants
perceive the value of Private Rank, Competitive Task, and Public Leaderboard
elements, with agreement rates at 69%, 62%, and 59%, respectively, a minority
expressed reservations, with 15%, 25%, and 19% disagreeing. Student feedback
reveals diverse perspectives. On private ranks, one student (P47) states: “if
I know how I am doing compared to others privately it will inspire me to do
better” suggesting a motivational benefit. In contrast, P44 resists the
competitive approach, stating, “I don’t view education as a competition.”.
Regarding competitive tasks, P88 comprehends that “The competition aspect
would make me get more involved.” while P78 express discomfort, noting, “I
would not like to participate in competitions. They make me nervous, and the
opponent is always overly confident.”. Public leaderboards also draw mixed
feelings. P42 notes motivation in competition, saying, “This creates
competition, which is often a great motivator.” Meanwhile, P50 articulates
concern about the implications of comparing grades publicly, manifesting: “I
would likely be a bit competitive, but I don’t like the idea of a public
leaderboard in a class if it’s comparing grades.”.

The Fictional dimension, encompassing Storytelling and Narrative, aims to
bridge the user with the environment, enhancing the learning experience by
embedding it within a broader context. However, opinions on these elements are
divided, with roughly 21% and 22% of respondents showing ambivalence despite
an average agreement rate of 58%. For Storytelling, one participant (P44)
admits that integrating related images and text makes learning more memorable
and engaging. In contrast, another participant (P77) thinks such elements can
detract from the learning focus. As for Narrative, some see it as a tool to
forge meaningful connections, with P42 noting its potential to make content
more relatable and encourage more profound commitment. However, P23 criticizes
it for feeling juvenile and counterproductive, suggesting narratives be more
aligned with real-world scenarios and professional language, thereby
preserving the learning’s relevance and respectability.

Overall, the elements Time Pressure, Social Pressure, and Cooperation received
the highest disagreement rate from participants (42%, 28%, and 22%,
respectively). For instance, P52 cited stress induced by time constraints,
while P104 voiced discomfort with their work being visible to peers, leading
to overthinking. P74’s reluctance towards cooperative tasks was influenced by
past negative experiences, often with unequal workload distribution.

Ranked game elements. Participants identified the top three elements they
preferred for inclusion in the gamified learning environment. The results are
visually represented in a stacked bar plot, as shown in Figure 2. This plot
categorizes the game elements by their rankings—from 1 to 3, with 1 being the
highest priority.

Figure 2: Responses to the top 3 ranked game elements ordered by rank 1. Each
bar represents the percentage of respondents who ranked a particular game
element as their top three preferences.

Reflecting on the elements that received the highest rankings in Rank 1, it is
clear that those from the performance dimension—namely Point, Stats, Badge,
and Level—stand out, garnering approximately 17.5%, 16.7%, 14.9%, and 12.3% of
the first-place votes, respectively. Stats, in particular, demonstrate a
strong preference among participants, achieving significant percentages across
the top three ranks, marking an inclination towards performance-related game
elements.

Remarkably, the quest element, capturing 11.4% of the top-rank votes, stands
out as the singular element consistently preferred as the top choice,
suggesting distinctive initial favoritism among participants. Meanwhile,
ProgressBar and Economy showcase a uniform distribution of preferences across
all three rankings (23% and 19.5%, respectively).

Other elements, including Puzzle (11.5%), Collaboration (9.7%), Renovation
(16.1%), and Novelty (18.8%), exhibit moderate levels of preference, with
their distribution across the three rankings indicating diverse views on their
significance and attractiveness. This observation is intriguing given that, in
Figure 1, Novelty emerged as the element with the highest level of agreement.

Additionally, it is worth noting that the Map element, despite receiving a
higher rating in Figure 1, was absent from participants’ rankings. The
omission may suggest that participants preferred the ProgressBar over the Map
as a progression mechanism.

Student’s perceptions. As previously mentioned, in our qualitative analysis,
we adopted open coding to identify related content across participants’
responses, with Engagement emerging universally across all game elements.
Another noteworthy category we unveiled, Fun, was referenced for nearly all
elements, excluding Progress Bar, Public Leaderboard, Private Rank, Social
Pressure, and Renovation. These exceptions suggest that aspects such as the
visual progress indicators from the progress bar, the competition, and the
pressure within the gamified setting are not universally perceived as
enjoyable. The finding was surprising, especially since Renovation was highly
rated on our scale (Figure 1). However, the discrepancy could be clarified by
participant P67’s remark: “This is where it would start to get bothersome. I
know I need help learning. Just not sure this would work for my type of
personality.” The comment suggests that while the concept of Renovation is
appealing, its application may not align with everyone’s learning preferences
or personality types, reflecting the nuanced impact of gamified elements on
individual user experience.

Autonomy emerged only when participants commented about the Choice element,
highlighting the significance of providing students with decision-making. P44
elaborated on its value, stating, “ I like having the ability to focus on the
things I enjoy doing, or feel as though I do a good job at.”. The affirmation
highlights how autonomy through choice enhances user engagement by allowing
personal interests and strengths to guide their actions.

We also identified that participants like the Sense of Progress associated
with game elements. This highlights the role of gamification in advancing
students’ perceptions of progression. However, this perception is less
associated with Public Leaderboard and Puzzle elements. It is inferred that
the Leaderboard may predominantly evoke a sense of competition. At the same
time, puzzles might be viewed more as a mechanism for assessing learning, as
indicated by P38: “It would feel REALLY good to get real-world experience and
challenge myself.”. Additionally, alongside puzzles, other elements were also
recognized by participants for their role in enhancing learning, including
Novelty, Renovation, Narrative, and Storytelling, as outlined by P54: “I think
this helps me learn, to receive feedback and improve on a specific task.”.

Elements centered around competition (such as Public Leaderboard, Private
Rank, and Competitive Task) and those focusing on Cooperation and Social
Pressure demonstrate mixed reactions among participants. While some exhibit an
apparent fondness for the competitive aspects, others show a strong dislike
due to the potential stress or discomfort they could cause. Social Pressure
arises as highly polarizing, with many respondents rejecting it, primarily
because of its association with peer evaluation.

Research Question 1 Which game elements do students prefer to engage in
learning the OSS contribution process? Answer: Elements such as Quest, Point,
Stats, and Badge emerged as clear favorites. In contrast, elements associated
with competition and pressure were viewed less favorably.

Table III: Odds ratios per characteristics of respondents |  | Abi vs.
---
Tim
| Men vs.
---
Minorities
| Hispanic/LatinX vs.
---
Non-Hispanic/LatinX
Quest | 1.0 | 1.6 | 1.3
Point | 1.8 | 1.1 | 1.1
Level | 0.6 | 0.8 | 1.7
Badge | 0.5 | 1.1 | 1.2
ProgressBar | 0.7 | 0.8 | 1.0
Map | 0.7 | 0.7 | 1.3
Stats | 0.7 | 1.1 | 2.0
Choice | 0.8 | 1.0 | 2.2*
Economy | 1.2 | 0.9 | 0.9
TimePressure | 1.7 | 0.8 | 1.2
Cooperation | 0.9 | 1.1 | 1.2
Renovation | 0.6 | 1.5 | 0.8
Puzzle | 0.6 | 1.3 | 1.1
Narrative | 0.5 | 1.4 | 0.9
Storytelling | 0.3** | 1.2 | 0.9
Novelty | 0.8 | 1.0 | 0.6
SocialPressure | 0.9 | 1.5 | 1.3
PublicLeaderboard | 1.6 | 1.1 | 0.9
PrivateRank | 0.9 | 0.9 | 1.2
CompetitiveTask | 0.7 | 0.6 | 0.7
Significance codes: * $p<0.10$, ** $p<0.05$.
| Note: Odds ratio greater than 1 means the first segment has greater
---
chances of reporting higher preference in the game element than the
second. A Ratio less than 1 means the opposite. The preference was
coded from the survey Likert questions.

###  V-B Students’ preferences of game elements concerning diverse
characteristics

To answer RQ2: How do students’ preferences of game elements differ according
to their cognitive style, gender, and ethnicity?, we assessed the odds ratios
to evaluate the preferences of each subgroup, as observed in Table III. The
preference levels were classified as high when respondents chose agree or
strongly agree and as low when selections ranged from neutral, disagree, or
strongly disagree.

Our findings from the odds ratio test show no significant statistical
differences in game element preferences among most groups. Only participants
who align with Tim’s GenderMag personality traits have a higher preference for
game elements such as Storytelling (0.3) than Abi. Additionally,
Hispanics/LatinX have greater odds of engagement in a gamified learning
environment that offers the Choice element (2.2x) than non-Hispanics/LatinX.

We conducted the Mann-Whitney U test to analyze the differences in game
element preferences across various groups, including persona, gender, and
ethnicity. This nonparametric statistical test is beneficial for comparing two
independent groups without assuming normal distribution in the data [46]. The
test results for comparing game element preferences by persona yielded a
p-value of 0.5937, by gender a p-value of 0.6691, and by ethnicity a p-value
of 0.19. The result does not offer significant statistical differences in game
element preferences among the groups.

Research Question 2 How do students’ preferences of game elements differ
according to their cognitive style, gender, and ethnicity? Answer: No
significant statistical differences were observed across persona, gender, and
ethnicity in most cases. The exceptions were participants who exhibited Tim’s
GenderMag personality traits, showing a preference for Storytelling, and
Hispanic/LatinX individuals, who were more likely to prefer the Choice
element.

##  VI Discussion

This section explores the insights derived from our research findings and
identifies opportunities for further exploration.

Enhancing engagement. Gamification has been widely implemented in educational
settings and teaching methodologies to boost student engagement and motivation
by integrating game elements [15]. Our findings indicate that game elements
with performance-oriented features, such as Stats, Maps, Levels, Points,
Progress Bars, and Badges, are essential for engaging students. They offer
performance feedback and foster a sense of advancement. Furthermore, elements
from the personal dimension, including Novelty, Puzzles, Quests, and
Renovation, significantly contribute to engagement by presenting new
challenges and content, which suggests an enhancement in learning activities.
Given that motivation and engagement are central themes in gamification
research [47], a gamified approach incorporating these elements is well-
positioned to increase student engagement in learning the OSS contribution
process.

Diverse preferences among game elements. Zahedi et al. [48] affirm that few
studies demonstrate the differences between men and women regarding the
performance impact of gamification; those that exist also show mixed results.
One gender study demonstrated that gamification (points, badges, levels)
improved male students’ performance, but no improvement was observed for
female students [49]. In accordance with these previous studies, our findings
show no significant statistical differences across persona, gender, and
ethnicity for most cases, except Storytelling for Tim’s persona and Choice for
Hispanic/Latino. Additionally, the limited sample of empirical studies that
explicitly explored gender are centered on the impacts of video games and not
gamifcation [48]. Therefore, future research could explore why certain game
elements appeal more to specific populations and the reasons for these
preferences.

Exploring the gamification considering self-determination theory. Self-
determination theory, established by Deci and Ryan [50], has been used to
conceptualize gamification. This theory identifies three psychological needs,
including competence or the desire for mastery, reflecting the innate need to
feel proficient and successful, driving people toward skill enhancement and
pursuing challenges that foster growth [51, 52]. Our findings suggest that
various game elements significantly contribute to the perception of
advancement (e.g., Stats, Maps, Levels, Points, Progress Bars, Badges, and
Quests), raising the question of whether they are aligned with the
psychological need for competence. Additionally, future studies can
investigate how game elements can be strategically implemented to satisfy
competence among the students, thereby harnessing a potent source of
motivation for learners.

Autonomy is another essential psychological need, and it embodies the
intrinsic desire for self-direction and the ability to choose paths freely and
meaningfully. According to participants’ open responses, our findings pinpoint
the choice element as the only satisfying autonomy need. The segmented
analysis indicates that participants identifying as Tim, Man, Hispanic/LatinX
ranked the Choice element highly. This observation prompts further exploration
into how autonomy can be facilitated within gamified environments to enrich
both the learning experience and outcomes by granting learners the freedom to
navigate and create their educational journey.

Reevaluating competition in gamified learning environment. Previous studies
have claimed the double-edged nature of leaderboards and competitive elements
in educational settings, noting their potential to boost competition but also
to hinder overall performance, cooperation, problem-solving and even foster
negative classroom dynamics, thereby increasing demotivation [47]. Upward
comparisons among students have been shown to potentially lead to adverse
effects and diminished academic self-concept [53]. Our findings echo this
ambivalence towards competitive game elements such as Public Leaderboards,
Private Rankings, and Competitive Tasks, which emerged as the least favored
among participants (see Figure 2). While some literature accentuates the
appeal of individual and team competition, other research points to its
detrimental impact on practical activities. For example, Nevin et al. [54]
stated that the ability to compete individually and in teams appealed to many
students. Simultaneously, Çakıroğlu et al. [55] reported that competition came
to the forefront as a negative element in practical activities. The divergent
perspectives on the same game elements suggest a complex relationship between
competition and motivation, highlighting the need for further investigation.

Gamification in OSS. The applicability of game elements to actual OSS
contributions is crucial. Concrete examples of how gamification could be
integrated into OSS practices include using progress bars for tracking
contributions, leaderboards for incentivizing participation, and badges for
recognizing achievements. Another possibility would be to address the barriers
of newcomers needing a deep understanding of the codebase by using game
elements such as quests through incremental and smaller manageable
contributions like bug reporting and documentation updates. Barriers that
prevent contributions from newcomers [8] demotivate students in OSS,
especially those from diverse backgrounds, including different cognitive
styles, genders, and ethnicities. A survey by Trinkenreich et al. [11]
highlights existing barriers in OSS communities, including toxic culture,
stereotyping, and uninclusive practices. Gamification can engage and attract a
diverse group of contributors. However, it is important to recognize that
while gamification can enhance engagement, it may not address deeper systemic
issues impacting OSS communities. We believe that gamification can complement
efforts to address these systemic issues, but it does not replace other
initiatives.

Implications. Our research emphasizes the need for designers to carefully
select and implement game elements that resonate with diverse student
preferences to enhance engagement. We showed that Performance-related and
Personal elements are well-accepted by participants from different backgrounds
(crosscutting gender, ethnicity, and cognitive styles). Therefore, they may
impact the engagement of more students in a more equal and inclusive
environment. Moreover, educators can use our findings to gamify the learning
experience, the OSS contribution process, and beyond. Understanding that
students have diverse preferences for game elements, educators can tailor
their teaching strategies and incorporate a mix of performance and personal
dimension elements to meet these preferences. Researchers can also replicate
our study and further explore our data, which is available in our replication
package.

##  VII Limitations

Sampling bias. As detailed in Section III, we used Prolific to recruit
participants, and we attempted to balance our sampling with the following
criteria: (i) Men/LatinX; (ii) Non-Men/LatinX; (iii) Men/Non-LatinX; and (iv)
Non-Men/Non-LatinX. However, we did not receive enough responses from non-men.
Moreover, we recognize that our sample may carry biases that are not
immediately apparent. Therefore, our conclusions are only valid for our
participants. Future research should aim for a more balanced and globally
representative sample to enhance the universality of the findings.
Additionally, we segmented the dataset to examine students’ preferences behind
certain game elements. The process involved dividing the dataset into smaller,
more manageable groups for focused analysis. While segmenting the data can
sometimes present challenges for statistical analysis due to reduced group
sizes, each group contained a minimum of 40 responses. The volume of data is
adequate for conducting Chi-Square Tests [56], the statistical significance
test we employed in our odds ratio analysis for RQ2.

Response biases. We introduced a preliminary context-setting stage to mitigate
the impact of such biases. The context was designed to provide a scenario for
participants, encouraging them to anchor their responses within a more
objective framework to capture more accurate and meaningful reflections of
their experiences.

Construct validity. To enhance construct validity, we based our survey on the
game elements taxonomy proposed by Toda et al. [34]. Moreover, we employed
pilot studies to test and collect feedback about our instrument to mitigate
this threat. The iterative studies enabled us to refine and validate our
survey tool, ensuring its effectiveness and reliability in capturing the
intended constructs. Another concern in our study is the participants’ prior
familiarity with the game elements discussed in the survey. To address this
issue, we provided a contextualized explanation of each game element, ensuring
a more informed response process. Regarding the hypothetical nature of the
survey and the potential gap between these scenarios and real-life OSS
contributions, we recognize this paper is an initial step toward developing a
gamified learning environment to support students’ learning about the OSS
contribution process. The survey helps to understand perceived value and user
receptiveness, which are relevant from the user-centered design perspective.

Subjectivity. We employed qualitative procedures to classify the open
questions responses. These procedures are subject to
subjectivity/interpretation bias. To mitigate this potential bias, we adopted
a multi-faceted approach involving the collaboration of multiple researchers.
The team engaged in continuous comparative analysis and reached conclusions
through a process of negotiated agreement. Each team member has extensive
experience in qualitative methods and OSS.

##  VIII Conclusion

We surveyed 115 participants, asking which game elements would boost
engagement in a gamified learning environment for teaching the OSS
contribution process. Enhancing student engagement has the potential to
improve the contribution experience and expand the contributor base, making
OSS projects more accessible and appealing to a diverse range of contributors.

Our findings highlight the importance of integrating game elements that offer
students constructive feedback on their performance and foster a sense of
progression and achievement. In future work, we aim to design and develop a
gamified environment with such game elements.

## Acknowledgments

The National Science Foundation (NSF) partially supports this work under grant
numbers 2236198, 2247929, 2303042, and 2303612. Katia Romero Felizardo is
funded by a research grant from the Brazilian National Council for Scientific
and Technological Development (CNPq), Grant $302339/2022-1$.

## References

  * [1] M. Gerosa, I. Wiese, B. Trinkenreich, G. Link, G. Robles, C. Treude, I. Steinmacher, and A. Sarma, “The shifting sands of motivation: Revisiting what drives contributors in open source,” in _ICSE 2021_. IEEE, 2021.
  * [2] C. Jergensen, A. Sarma, and P. Wagstrom, “The onion patch: migration in open source ecosystems,” in _19th ESEC/FSE 2011_. ACM, 2011.
  * [3] V. Goduguluri, T. Kilamo, and I. Hammouda, “Kommgame: A reputation environment for teaching open source software,” in _IFIP_. Springer, 2011, pp. 312–315.
  * [4] J. O. Silva, I. Wiese, D. M. German, C. Treude, M. A. Gerosa, and I. Steinmacher, “Google summer of code: Student motivations and contributions,” _JSS_ , vol. 162, p. 110487, 2020.
  * [5] G. Pinto, C. Ferreira, C. Souza, I. Steinmacher, and P. Meirelles, “Training software engineers using open-source software: the students’ perspective,” in _ICSE-SEET 2019_. IEEE, 2019.
  * [6] G. H. L. Pinto, F. Figueira Filho, I. Steinmacher, and M. A. Gerosa, “Training software engineers using open-source software: the professors’ perspective,” in _CSEE &T_. IEEE, 2017, pp. 117–121.
  * [7] C. Ferreira, C. Souza, G. Pinto, I. Steinmacher, and P. Meirelles, “When students become contributors: leveraging oss contributions in software engineering courses,” in _Proceedings of the XXXII SBES_ , 2018, pp. 260–269.
  * [8] I. Steinmacher, T. Conte, M. Gerosa, and D. Redmiles, “Social barriers faced by newcomers placing their first contribution in open source software projects,” in _ACM CSCW 2015_ , 2015.
  * [9] C. Mendez, H. S. Padala, Z. Steine-Hanson, C. Hilderbrand, A. Horvath, C. Hill, L. Simpson, N. Patil, A. Sarma, and M. Burnett, “Open source barriers to entry, revisited: a sociotechnical perspective,” in _ICSE 2018_ , 2018.
  * [10] S. H. Padala, C. J. Mendez, L. F. Dias, I. Steinmacher, Z. S. Hanson, C. Hilderbrand, A. Horvath, C. Hill, L. D. Simpson, M. Burnett _et al._ , “How gender-biased tools shape newcomer experiences in OSS projects,” _IEEE TSE_ , 2020.
  * [11] B. Trinkenreich, I. Wiese, A. Sarma, M. Gerosa, and I. Steinmacher, “Women’s participation in open source software: a survey of the literature,” _ACM TOSEM_ , 2022.
  * [12] G. C. Diniz, M. A. G. Silva, M. A. Gerosa, and I. Steinmacher, “Using gamification to orient and motivate students to contribute to OSS projects,” in _IEEE/ACM 10th CHASE_. IEEE, 2017, pp. 36–42.
  * [13] M. Müller, C. Schindler, and W. Slany, “Engaging students in open source: Establishing foss development at a university,” in _HICSS_ , 01 2019.
  * [14] C. V. Flach and D. S. Feitosa, “Teaching and promoting engagement with oss: Yet another experience report,” in _Proceedings of the XXXVII SBES_ , 2023, pp. 534–543.
  * [15] C. Dichev and D. Dicheva, “Gamifying education: what is known, what is believed and what remains uncertain: a critical review,” _International journal of educational technology in higher education_ , vol. 14, pp. 1–36, 2017.
  * [16] O. Pedreira, F. García, N. Brisaboa, and M. Piattini, “Gamification in software engineering–a systematic mapping,” _IST_ , vol. 57, pp. 157–168, 2015.
  * [17] D. Dicheva, C. Dichev, G. Agre, and G. Angelova, “Gamification in education: A systematic mapping study,” _ET &S_, vol. 18, no. 3, pp. 75–88, 2015.
  * [18] A. Bartel and G. Hagel, “Gamifying the learning of design patterns in software engineering education,” in _EDUCON_. IEEE, 2016, pp. 74–79.
  * [19] A. Ghai and U. Tandon, “Integrating gamification and instructional design to enhance usability of online learning,” _Education and Information Technologies_ , vol. 28, no. 2, pp. 2187–2206, 2023.
  * [20] K. Huotari and J. Hamari, “A definition for gamification: anchoring gamification in the service marketing literature,” _Electronic markets_ , vol. 27, no. 1, pp. 21–31, 2017.
  * [21] S. Deterding, R. Khaled, L. E. Nacke, D. Dixon _et al._ , “Gamification: Toward a definition,” in _CHI_ , vol. 12. Vancouver, 2011, pp. 1–79.
  * [22] S. Zhao15, X. Xia25, B. Fitzgerald, X. Li, V. Lenarduzzi, D. Taibi, R. Wang, W. Wang, and C. Tian, “Motivating open source collaborations through social network evaluation: A gamification practice from alibaba,” in _IEEE/ACM 46th ICSE-SEIP_ , 2024.
  * [23] K. M. Kapp, _The gamification of learning and instruction: game-based methods and strategies for training and education_. John Wiley & Sons, 2012.
  * [24] K. Huotari and J. Hamari, “Defining gamification: a service marketing perspective,” in _16th MindTrek_ , 2012, pp. 17–22.
  * [25] J. J. Lee and J. Hammer, “Gamification in education: What, how, why bother?” _Academic exchange quarterly_ , vol. 15, no. 2, p. 146, 2011.
  * [26] C.-H. Su, “The effects of students’ motivation, cognitive load and learning anxiety in gamification software engineering education: a structural equation modeling study,” _Multimedia Tools and Applications_ , vol. 75, pp. 10 013–10 036, 2016.
  * [27] S. K. Sheth, J. S. Bell, and G. E. Kaiser, “Increasing student engagement in software engineering with gamification,” in _4th SSE_ , 2012.
  * [28] C. Toscani, D. Gery, I. Steinmacher, and S. Marczak, “A gamification proposal to support the onboarding of newcomers in the flosscoach portal,” in _17th IHC_ , 2018, pp. 1–10.
  * [29] C. R. Prause, J. Nonnen, and M. Vinkovits, “A field experiment on gamification of code quality in agile development.” in _PPIG_ , 2012, p. 17.
  * [30] J. M. Rojas and G. Fraser, “Teaching mutation testing using gamification,” in _ECSEE_ , 2016.
  * [31] G. Garaccione, T. Fulcini, and M. Torchiano, “Gerry: a gamified browser tool for gui testing,” in _Gamify’22_ , 2022, pp. 2–9.
  * [32] C. DeCusatis, E. Alvarico, and O. Dirahoui, “Gamification of cybersecurity training,” in _Gamify’22_ , 2022, pp. 10–13.
  * [33] S. Ozturk, “Gamification of exploratory testing process,” in _Gamify’22_ , 2022, pp. 14–17.
  * [34] A. M. Toda, A. C. Klock, W. Oliveira, P. T. Palomino, L. Rodrigues, L. Shi, I. Bittencourt, I. Gasparini, S. Isotani, and A. I. Cristea, “Analysing gamification elements in educational environments using an existing gamification taxonomy,” _Smart Learning Environments_ , vol. 6, no. 1, pp. 1–14, 2019.
  * [35] M. Burnett, S. Stumpf, J. Macbeth, S. Makri, L. Beckwith, I. Kwan, A. Peters, and W. Jernigan, “Gendermag: A method for evaluating software’s gender inclusiveness,” _Interacting with Computers_ , 2016.
  * [36] E. Murphy-Hill, A. Elizondo, A. Murillo, M. Harbach, B. Vasilescu, D. Carlson, and F. Dessloch, “Gendermag improves discoverability in the field, especially for women: An multi-year case study of suggest edit, a code review feature,” in _IEEE/ACM 46th ICSE_ , 2024, pp. 1–12.
  * [37] I. Santos, J. F. Pimentel, I. Wiese, I. Steinmacher, A. Sarma, and M. A. Gerosa, “Designing for cognitive diversity: Improving the github experience for newcomers,” in _2023 IEEE/ACM 45th ICSE-SEIS_. IEEE, 2023, pp. 1–12.
  * [38] C. Hilderbrand, C. Perdriau, L. Letaw, J. Emard, Z. Steine-Hanson, M. Burnett, and A. Sarma, “Engineering gender-inclusivity into software: ten teams’ tales from the trenches,” in _ICSE 2020_. ACM, 2020.
  * [39] G. James, D. Witten, T. Hastie, R. Tibshirani _et al._ , _An introduction to statistical learning_. Springer, 2013, vol. 112.
  * [40] P. J. Lavrakas, _Encyclopedia of survey research methods_. Sage publications, 2008.
  * [41] T. Punter, M. Ciolkowski, B. Freimut, and I. John, “Conducting on-line surveys in software engineering,” in _2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings._ IEEE, 2003, pp. 80–88.
  * [42] A. Danilova, A. Naiakshina, S. Horstmann, and M. Smith, “Do you really code? designing and evaluating screening questions for online surveys with programmers,” in _IEEE/ACM 43rd ICSE_. IEEE, 2021, pp. 537–548.
  * [43] D. Russo, “Recruiting software engineers on prolific,” in _RoPES’22_ , 2022.
  * [44] I. Villegas Molina, A. Montalvo, and A. G. Soosai Raj, “Us latines in computing: A review of the literature,” in _55th ACM SIGCSE_ , 2024, pp. 1381–1387.
  * [45] A. Strauss and J. Corbin, _Basics of qualitative research techniques_. Sage Publications, 1998.
  * [46] N. Feltovich, “Nonparametric tests of differences in medians: comparison of the wilcoxon–mann–whitney and robust rank-order tests,” _Experimental Economics_ , vol. 6, pp. 273–297, 2003.
  * [47] R. Leitão, M. Maguire, S. Turner, and L. Guimarães, “A systematic evaluation of game elements effects on students’ motivation,” _Education and Information Technologies_ , pp. 1–23, 2022.
  * [48] L. Zahedi, J. Batten, M. Ross, G. Potvin, S. Damas, P. Clarke, and D. Davis, “Gamification in education: A mixed-methods study of gender on computer science students’ academic performance and identity development,” _Journal of Computing in Higher Education_ , vol. 33, pp. 441–474, 2021.
  * [49] L. Z. Pedro, A. M. Lopes, B. G. Prates, J. Vassileva, and S. Isotani, “Does gamification work for boys and girls? an exploratory study with a virtual learning environment,” in _30th ACM SAC_ , 2015, pp. 214–219.
  * [50] E. L. Deci and R. M. Ryan, “The general causality orientations scale: Self-determination in personality,” _Journal of research in personality_ , vol. 19, no. 2, pp. 109–134, 1985.
  * [51] R. M. Ryan and E. L. Deci, “Intrinsic and extrinsic motivations: Classic definitions and new directions,” _Contemporary educational psychology_ , vol. 25, no. 1, pp. 54–67, 2000.
  * [52] C.-L. Hsu and H.-P. Lu, “Why do people play on-line games? an extended tam with social influences and flow experience,” _Information & management_, vol. 41, no. 7, pp. 853–868, 2004.
  * [53] P. Dijkstra, H. Kuyper, G. Van der Werf, A. P. Buunk, and Y. G. van der Zee, “Social comparison in the classroom: A review,” _Review of educational research_ , vol. 78, no. 4, pp. 828–879, 2008.
  * [54] C. R. Nevin, A. O. Westfall, J. M. Rodriguez, D. M. Dempsey, A. Cherrington, B. Roy, M. Patel, and J. H. Willig, “Gamification as a tool for enhancing graduate medical education,” _Postgraduate medical journal_ , vol. 90, no. 1070, pp. 685–693, 2014.
  * [55] Ü. Çakıroğlu, B. Başıbüyük, M. Güler, M. Atabay, and B. Y. Memiş, “Gamifying an ict course: Influences on engagement and academic performance,” _Computers in human behavior_ , vol. 69, pp. 98–107, 2017.
  * [56] O. Sureiman, “Conceptual model on application of chi-square test in education and social sciences,” _Educational Research and Reviews_ , vol. 8, no. 15, 2013.