[1]WenbinZhang

1]Florida International University 2]TERRA Environmental Research Institute
3]L3S Research Center

# FairAIED: Navigating Fairness, Bias, and Ethics in Educational AI
Applications

Sribala VidyadhariChinta<EMAIL_ADDRESS>ZichongWang<EMAIL_ADDRESS>ZhipengYin<EMAIL_ADDRESS>NhatHoang<EMAIL_ADDRESS>MatthewGonzalez
<EMAIL_ADDRESS>Tai LeQuy<EMAIL_ADDRESS>[  [  [

###### Abstract

The integration of Artificial Intelligence (AI) into education has
transformative potential, providing tailored learning experiences and creative
instructional approaches. However, the inherent biases in AI algorithms hinder
this improvement by unintentionally perpetuating prejudice against specific
demographics, especially in human-centered applications like education. This
survey delves deeply into the developing topic of algorithmic fairness in
educational contexts, providing a comprehensive evaluation of the diverse
literature on fairness, bias, and ethics in AI-driven educational
applications. It identifies the common forms of biases—data-related,
algorithmic, and user-interaction—that fundamentally undermine the
accomplishment of fairness in AI teaching aids. By outlining existing
techniques for mitigating these biases, ranging from varied data gathering to
algorithmic fairness interventions, the survey emphasizes the critical role of
ethical considerations and legal frameworks in shaping a more equitable
educational environment. Furthermore, it guides readers through the
complexities of fairness measurements, methods, and datasets, shedding light
on the way to bias reduction. Despite these gains, this survey highlights
long-standing issues, such as achieving a balance between fairness and
accuracy, as well as the need for diverse datasets. Overcoming these
challenges and ensuring the ethical and fair use of AI’s promise in education
call for a collaborative, interdisciplinary approach.

###### keywords:

Artificial Intelligence, Education, AI Fairness, Bias Mitigation, Algorithmic
Bias, Fairness Metrics

##  1 Introduction

Artificial Intelligence (AI) has been a transformative force, reshaping the
landscape of learning and teaching methodologies. The history of AI in
education, tracing its origins to the early experiments with computer-assisted
instruction in the 1960s and 1970s, is deeply intertwined with the evolution
of AI itself, which is typically traced back to the Dartmouth Workshop in 1956
[1, 2, 3]. This period marked the beginning of exploring how technology could
cater to individual learning styles and paces, a concept that has evolved
significantly with the advent of AI [4]. The significance of AI in education
lies in its ability to provide personalized learning experiences, automate
grading, and offer language translation services, thereby addressing diverse
educational needs [5, 4, 6]. AI applications in education have grown
exponentially, with systems designed to adapt to the learning pace of
students, thereby enhancing engagement and understanding [7]. For instance,
AI-powered platforms can generate individualized learning paths based on
student’s goals, interests, and prior knowledge, adapting content in real-time
to meet learners’ needs [6]. Furthermore, AI’s role in automating grading
systems has revolutionized the assessment process, offering consistency and
personalized feedback, thus freeing educators to focus on more complex
teaching tasks [8, 9]. Automated grading systems like
Graide111https://www.graide.co.uk/ and Top Marks AI222https://www.topmarks.ai/
have demonstrated the ability to analyze complex subjects and provide
accurate, consistent feedback, drastically reducing educators’ time on
grading. Language translation and support facilitated by AI have also
significantly contributed to breaking down language barriers in education,
making learning materials accessible to a global audience [10, 11].

Despite the benefits, the deployment of AI in education raises critical issues
of fairness and bias [12, 13]. AI systems can inadvertently perpetuate and
amplify existing biases present in the data they are trained on, leading to
unfair outcomes, particularly for historically marginalized groups [14]. For
instance, AI-powered essay grading systems may reflect the biases of the data
they are trained on, which could include the subjective preferences of human
graders [4]. This could result in unfair assessments for students whose
writing style or cultural background differs from the norm established by the
training data [1]. Ethical considerations also come into play when discussing
the transparency and accountability of AI systems in education. The “black
box” nature of some AI algorithms can make it challenging to understand and
contest their decisions, raising concerns about the ethical implications of
their use in educational evaluations [15].

Addressing these issues is crucial for creating fair and equitable educational
systems. Researchers and developers are actively exploring mitigation
strategies, such as pre-processing data to ensure representativeness and post-
processing decisions to correct for biases [16]. However, achieving both
fairness and accuracy can be challenging, requiring careful consideration of
the trade-offs involved. Ethical considerations around prioritizing different
types of bias and which groups to focus on in bias mitigation add complexity
to the development and implementation of AI systems in education [14]. Despite
these challenges, the importance of addressing bias in AI is underscored by
the need to create educational environments that benefit all individuals and
promote equality.

In order to achieve this objective, our survey thoroughly examines the
connections between fairness, bias, and ethics in the field of AI applications
in education. We analyze the progress of AI in the field of education by doing
a thorough examination of relevant literature and using a well-organized
conceptual framework. This investigation highlights the importance of
fairness, prejudice reduction, and ethical considerations in developing AI
systems that promote equity. It also analyzes the technical challenges and
strategic actions necessary for fostering fairness. This encompasses an
examination of various methods of gathering data, approaches for identifying
bias, and the advancement toward achieving fairness in algorithms; all
demonstrated through pertinent case studies. In addition, the study explores
the ethical principles and regulatory frameworks that govern the use of AI in
educational environments. It emphasizes the importance of a collaborative
effort among educators, technologists, ethicists, and policymakers to address
these concerns comprehensively. This article navigates the complex challenges
related to fairness and bias in educational AI, advocating for a cooperative
approach to ensure that technical progress aligns with ethical principles,
hence promoting a more inclusive educational setting.

The survey is structured as follows: we outline our methodology for doing a
literature review in Section 2. In Section 3 discusses various case studies of
AI in education and biases that can be exhibited by them. In Section 4, the
approaches to improve fairness, fairness notions and metrics to measure
fairness are explored. Following this, Section 5 describes an explanation of
ethical considerations and frameworks. Section 6 of this literature provides
an overview of tools and datasets utilized in the field of AI education.
Section 7 delineates the challenges and future directions. The survey’s
conclusion is presented in Section 8.

##  2 Literature Selection Criteria

The literature selection for this survey follows stringent inclusion and
exclusion criteria to ensure source relevance and quality, conference
proceedings, and authoritative reports from the past decade related to AI’s
application in education, particularly concerning fairness, bias, and ethics.
The search spans multiple databases, including IEEE Xplore, ACM Digital
Library, PubMed, Scopus, and Google Scholar, using keywords like “artificial
intelligence in education”, “AI fairness in education”, and “bias in
educational AI systems”. We employ a systematic review methodology that
involves identifying literature, screening texts based on set criteria,
thoroughly evaluating texts for relevance, and extracting pertinent data for
synthesis. Multiple reviewers carry out this process to minimize selection
bias, aiming to provide a comprehensive overview of fairness and bias in
educational AI, synthesizing existing research to inform future advancements
in the field.

##  3 AI in education: Case-studies and Bias perspectives

###  3.1 Student Assessment and Grading

Student evaluation and grading methods based on AI and machine learning
techniques are becoming more common in AI in education (AIED) applications
[75]. These systems use natural language processing, computer vision, and
other AI capabilities to assess student work, automate grading and feedback,
and facilitate tailored learning experiences.

Examples of such tools include automated essay grading systems that analyze
and grade written responses [76], intelligent tutoring systems like Coursera
Coach powered by Generative AI333https://www.coursera.org/ that assess student
learning and provide personalized feedback, and AI-assisted grading platforms
like Turnitin444https://www.turnitin.com/ that provide automated feedback for
assignments and exams [77]. The role of these AI-driven assessment tools is to
provide consistent, objective, and efficient evaluations of student
performance while also enabling personalized feedback, identifying knowledge
gaps, and informing instructional strategies [78, 76]. They aim to reduce the
workload for educators and offer scalable assessment solutions, particularly
in large-scale online education settings.

AI systems may apply uniform criteria and rubrics to all student submissions,
which can help reduce human biases and inconsistencies in grading. This is one
of the main advantages of these technologies, though [76]. Furthermore,
content and assessments can be modified by AI-powered adaptive learning
platforms in accordance with each learner’s unique learning preferences and
advancement, resulting in individualized and successful learning experiences
[77].

Despite these advantages, inherent biases in AI assessment systems are a
source of concern. The evaluations and feedback produced by these systems may
reinforce unjust treatment or prejudices against particular demographic groups
[78] if the training data used to create them is biased or lacks variety. An
AI system for essay grading, for instance, was shown to be biased against
specific demographic groups, giving essays written by black students a lower
score than those produced by white students with comparable material,
according to research [79, 78]. This prejudice most likely resulted from
societal biases shown in the training data.

###  3.2 Admissions and Recruitment

AI systems are being utilized more and more in college admissions procedures
to assess applications and provide recommendations or choices [80, 81, 82].
These AI systems’ function is to offer a scalable and effective mean of
processing massive amounts of applications with the goal of producing
unbiased, data-driven choices [80, 81]. Using AI in admissions could help
reduce human biases and discrepancies that can occur from manual evaluation
processes. However, a number of studies have brought to light instances of
biases that AI admissions systems have shown in the real world, raising
questions about equity and justice. According to research, if the training
data is biased or lacks diversity, AI admissions models may be prejudiced
against specific demographic groups, including women and underrepresented
minorities [83, 84]. For example, an analysis of application essays alone
revealed that an AI classifier could accurately identify an applicant’s gender
and household income level, indicating the presence of socioeconomic and
gender biases in the data or model [84].

Another research investigation that examined AI models for college admissions
discovered that the demographics of admitted students changed when
standardized test results were removed from the training set [80]. This
suggested the possibility of racial biases as well as biases against first-
generation college candidates. Furthermore, if the AI models used to train
admissions systems replicate historical prejudices or discrimination found in
earlier admissions decisions, this could reinforce and magnify the biases
[83].

Although the goal of AI in admissions is efficiency and objectivity, the
possibility of biases against particular groups emphasizes the need for these
systems to be developed and implemented responsibly, with an emphasis on
inclusivity, fairness, and addressing potential biases in the algorithms and
training data [80, 81, 82]. The potential advantages of utilizing AI in
education could be undermined if these biases are not addressed, since they
could exacerbate and prolong already-existing societal inequities [83, 84].

###  3.3 Educational Content Recommendation

AI-based recommendation systems have become essential tools in education,
assisting students in making informed decisions regarding their academic paths
by analyzing extensive datasets, including historical academic performance,
course evaluations, and individual preferences [85, 86]. These systems aim to
enhance student engagement and success by providing personalized
recommendations that align with students’ goals and abilities, thus fostering
a more tailored educational experience [85, 87]. The primary benefits of AI-
based recommendation systems include their ability to offer personalized
learning experiences, which can increase student satisfaction and retention
rates, and their capacity to manage and process large amounts of data
efficiently, providing timely and accurate recommendations that are difficult
for human advisors to achieve on a similar scale [88, 89].

Despite these advantages, the AI-based recommendation systems are not free
from biases, which can significantly impact educational equity [85, 86].
Biases often arise from the training data, which can reflect existing
demographic disparities, leading to recommendations that perpetuate these
inequalities [90, 91]. For example, research has shown that minority students
are less frequently recommended for advanced courses, thereby limiting their
academic opportunities [85, 87]. Additionally, gender biases are prevalent,
with algorithms often suggesting STEM courses predominantly to male students
while steering female students towards the humanities, reinforcing societal
stereotypes and contributing to the gender gap in STEM fields [86, 92, 90]. To
mitigate these biases, researchers recommend diversifying training data,
implementing fairness-aware algorithms, and conducting continuous monitoring
and audits of the AI systems to ensure fairness over time [91]. Incorporating
feedback from a diverse user base is also essential for refining these systems
to better serve all students equitably.

###  3.4 AI in Medical Curriculum Design

AI systems are increasingly utilized in medical education to design curricula
that are both relevant and up-to-date, leveraging vast amounts of data from
medical literature and clinical practices to ensure comprehensive training
[93]. However, biases in these systems can significantly impact the
educational outcomes and experiences of medical students, necessitating a
careful examination of how these biases manifest and their potential
consequences. One prominent study involves a medical school that implemented
an AI-driven curriculum design system to integrate the latest medical research
into its courses. The system analyzed global medical literature to identify
emerging trends and best practices. While this approach enhanced the
curriculum’s relevance, it also introduced biases, particularly when the AI
predominantly sourced data from Western medical journals [94]. This bias led
to an overemphasis on Western medical practices and an underrepresentation of
medical knowledge and practices from other regions, such as Asia and Africa.
As a result, students from these underrepresented regions felt that their
educational experience did not fully address the medical conditions and
treatment protocols prevalent in their home countries [95].

Another study highlighted biases in the AI systems’ recommendations based on
gender and socio-economic status. The AI, trained on historical data
reflecting existing disparities, tended to recommend certain specializations
more frequently to male students while steering female students towards
traditionally softer medical fields like pediatrics and general practice. This
bias not only perpetuated gender stereotypes but also impacted the diversity
of specialists in various medical fields [96]. Additionally, students from
lower socio-economic backgrounds received fewer recommendations for high-cost
specializations, which could perpetuate economic disparities within the
medical profession.

###  3.5 Types of Bias in Education

Having explored case studies demonstrating biases in AI systems within
education, it is crucial to understand the specific types of biases that these
systems can exhibit. These biases significantly affect the fairness and
accuracy of AI applications in educational settings. This sub-section will
explore more types of biases that AI systems exhibit. Below are the potential
biases categorized into three types: Data-related Bias, Algorithmic Bias, and
User-Interaction Bias.

Figure 1: Types of Bias in Education

####  3.5.1 Data-Related Biases

The topic of AIED has a notable obstacle in the form of data-related biases,
which include historical bias, measurement bias, and representation bias. The
biases observed in AI systems derive from fundamental flaws within the
datasets utilized for training and evaluation [97]. These biases reflect
societal imbalances and inaccuracies inherent in data gathering and
representation methodologies. As a result, these biases have the potential to
result in AI systems that exhibit unequal performance among various student
populations, hence intensifying pre-existing imbalances within educational
environments.

Historical Bias: Historical bias in AI education manifests when AI systems use
data that encapsulates past inequalities, affecting decisions with outdated or
discriminatory norms [98]. Such biases occur because these systems are often
trained on data sets that reflect systemic biases encoded due to socioeconomic
factors rather than actual capabilities or potentials. Historical bias
primarily affects groups rather than individuals, as it reflects collective
societal prejudices embedded in data [99]. For example, AI-driven educational
tools might disproportionately identify minority students as at-risk by
relying on historical data that undervalues their academic potential due to
lower past performance linked to socioeconomic disadvantages. This is similar
to biases observed in public services automation, where algorithms trained on
biased data like disciplinary records or test scores reinforce disparities. In
college admissions, the AI systems can perpetuate socioeconomic and racial
disparities by favoring applicants from specific zip codes and schools,
reflecting a historical preference embedded within the training data [100].
These examples highlight the urgent need for more inclusive data practices and
continuous oversight in the development of educational AI systems. By
integrating ethical considerations and adjusting for historical biases, it is
possible to foster equitable educational opportunities and prevent the
reinforcement of past inequalities.

Measurement Bias: Measurement bias in AIED can significantly impact the
validity and effectiveness of AI-driven tools, particularly in systems like
automated essay scoring (AES). This type of bias occurs when data collection
methods systematically distort inputs in a way that misrepresents the true
variables they aim to measure [101]. A notable instance of measurement bias is
found in the performance of AES systems, which tend to score essays written in
African American Vernacular English (AAVE) lower than those in Standard
American English (SAE) [102]. This was highlighted in studies such as those
published in the Journal of Educational Measurement, which indicate that these
discrepancies are due to the system’s reliance on training datasets primarily
composed of Standard English, leading them to undervalue the linguistic
structures of AAVE. Measurement bias can be considered both an individual and
group bias. It affects individuals by potentially misrepresenting their
abilities based on linguistic or cultural differences. This example
underscores the need for incorporating a wider array of linguistic data in
training sets and conducting regular reviews for bias, ensuring that AI
assessments in education fairly and accurately reflect the diverse
capabilities of all students [103].

Representation Bias: Representation bias in AIED refers to the systematic
underrepresentation or misrepresentation of certain groups or characteristics
in the data used to train AI models. This bias can lead to AI systems that
perpetuate or amplify existing societal biases and inequalities in educational
contexts [99]. Representation bias is primarily a group bias because it
affects specific demographic groups based on how they are represented in the
training data [104]. A study [105], which investigates the disparities in AI
educational tools used for automated essay scoring. This study focuses on how
these tools can disproportionately benefit students from linguistic
backgrounds similar to those on which the models were trained. The AI systems
were primarily trained on datasets composed of essays written by native
English speakers, leading to lower accuracy when scoring essays by learners of
English as a second language (ESL). This scenario is a clear instance of
representation bias, where the underrepresentation of ESL students in the
training data resulted in a model that inadequately serves a significant
portion of the student population. The bias not only impacts grades but also
affects students’ learning outcomes and confidence in their abilities.

####  3.5.2 Algorithmic-Related Biases

Addressing algorithmic-related biases in AIED involves a deep dive into how
the design and implementation of algorithms can inadvertently lead to unfair
outcomes. This exploration is critical, given the increasing reliance on AI to
personalize learning, assess student performance, and even make administrative
decisions in educational settings. Below, we elaborate on various facets of
algorithmic-related biases, backed by extensive research in the field.

Algorithm Bias: Algorithmic bias in AIED refers to the systematic and
repeatable errors in a computer system that create unfair outcomes, such as
privileging one arbitrary group of users over others. This bias can stem from
various sources, including but not limited to the data used to train the AI
systems, the design of the algorithm itself, and the societal context in which
the AI is deployed [106]. Algorithm bias can affect both individual and group
levels. For example, an algorithm trained on biased data may provide
inaccurate recommendations for individuals based on their unique attributes.
Additionally, it can perpetuate stereotypes and reinforce systemic
inequalities, disproportionately impacting specific groups of students based
on race, gender, or socioeconomic status [12]. A study that exemplifies
algorithmic bias in AI education is found in the use of large language models
(LLMs) in medical education. These models, such as those in the Generative
Pre-trained Transformers (GPT) series, have been integrated into medical
curricula to assist with developing teaching materials, student assessments,
and personalized learning plans. While LLMs hold significant promise for
enhancing medical education, they also pose challenges related to algorithmic
bias. For instance, if an LLM is trained in the medical literature that
historically underrepresents certain populations, it may generate biased
content that could influence medical students’ learning experiences and
knowledge. This could lead to the perpetuation of existing disparities in
medical treatment and outcomes for underrepresented groups. The case study
underscores the importance of recognizing and addressing algorithmic bias in
AI educational tools to prevent the reinforcement of historical inequalities
and ensure equitable educational experiences for all students. It also
highlights the need to critically examine AI technologies in education to
balance their potential benefits with the risks they pose, particularly in
sensitive fields like healthcare [107].

Mapping Bias: Mapping bias refers to the bias that occurs when a universal
model is employed to analyze data that includes underlying subgroups or
distinct types of examples that require individual treatment [108]. The
assumption behind aggregation bias is that the mapping of inputs to labels is
uniform across all subsets of the data, which is frequently not the case in
real-world scenarios. For example, logistic regression models assume the
relationships between input and target variables are linear, but this may not
be the case in the real world. Another real-world example is that a particular
dataset might represent people or groups with diverse backgrounds, cultures,
or norms, and a specific variable may have different meanings. The use of a
universal model in such cases can result in a suboptimal model for all groups,
or one exclusively suited to the dominant population.

Mapping bias is primarily considered group bias because it affects various
subgroups within the dataset differently. However, it can also be an
individual bias when specific, unique cases are misrepresented due to the
universal model’s assumptions. This dual impact makes it crucial to address it
in both broad and specific contexts to ensure fair and accurate AI system
performance [108].

Learning Bias: Learning bias is when modeling choices amplify performance
disparities across different examples in the data [109]. For example, an
essential modeling choice that can lead to learning bias is the objective
function that an ML algorithm learns to optimize during training. Typically,
these functions encode some measures of accuracy for the task. However, issues
can arise when prioritizing one objective over another, resulting in
unintended consequences. For instance, minimizing cross-entropy loss when
building a classifier may lead to a model with more false positives than
desirable in various real-world scenarios. Learning bias can affect both
individual and group levels [110]. At the individual level, this bias might
result in a specific user’s unique data being underrepresented or
misclassified. At the group level, it can exacerbate disparities among
different demographic groups if the training data reflects historical biases.
This dual impact makes learning bias a critical issue in the development of AI
systems in diverse applications.

Confirmation Bias: Confirmation bias in AIED emerges when AI systems
inherently favor information or outcomes that affirm the pre-existing beliefs
encoded within their algorithms or datasets. This can lead to a self-
reinforcing loop where educational AI systems might perpetuate traditional
teaching methods or curricula, overlooking innovative approaches that could
benefit diverse learning styles [111, 112]. Addressing confirmation bias
requires a critical examination of the underlying assumptions in AI models and
the inclusion of diverse perspectives in the development process [113, 114].
Confirmation bias can manifest both as an individual bias and a group bias. At
the individual level, it influences personalized learning recommendations and
feedback, potentially reinforcing a student’s pre-existing beliefs and
preferences. At the group level, it can uphold systemic biases within
educational institutions, preserving conventional practices and sidelining
alternative educational methods that might benefit diverse student populations
[115].

####  3.5.3 User-Interaction Related Biases

User Interaction-Related Biases in educational AI systems encompass biases
that emerge from interactions between users (such as students and educators)
and AI technologies. These biases can significantly influence the behavior and
evolution of AI systems over time, potentially leading to unfair or
discriminatory outcomes. Here, we delve into three primary types of user
interaction-related biases: Interaction Bias, Stereotyping Bias, and Exclusion
Bias, drawing on a range of studies to illuminate these concepts.

Interaction Bias: Interaction Bias arises when the user’s engagement with the
AI systems influences the system’s learning and adaptation in a way that
reinforces or introduces new biases. For example, in an AI-driven educational
platform, if certain groups of students are more proactive in providing
feedback or interacting with the system, the AI may evolve to better suit the
needs and preferences of these groups, potentially neglecting others. This can
lead to a feedback loop where the AI increasingly reflects the biases of its
most active users. This bias can be both individual and group bias since it
can affect individual interactions and amplify biases from specific user
groups. Studies by [116] and [117] have explored how user interactions can
shape AI behavior, highlighting the need for mechanisms to ensure diverse
interactions are equally represented and valued in AI systems. Further, the
work of [111] emphasizes the importance of transparency in AI systems to
mitigate interaction biases, suggesting that users should be made aware of how
their interactions influence AI behavior.

Stereotyping Bias: Stereotyping Bias in educational AI occurs when AI systems
make assumptions based on generalized stereotypes, which can lead to
discriminatory outcomes. For instance, an AI-based career counseling tool
might suggest careers based on societal stereotypes, such as nursing for
female students and engineering for male students. This not only reinforces
harmful stereotypes but also limits individual students’ potential by guiding
them along biased paths. Research by [118] and [119] has demonstrated how
machine learning models can amplify gender stereotypes. Efforts to combat
stereotyping bias include the development of debiasing techniques and the
promotion of inclusive design practices, as discussed by [120], who emphasizes
the critical need for inclusive and equitable AI systems in education.

Stereotyping bias can affect both groups and individuals. On a group level, it
reinforces societal stereotypes and systemic biases [12]. On an individual
level, it limits personal opportunities and choices based on biased AI
outputs. According to [118], such biases can be perpetuated through training
data that reflect historical inequities. Similarly [120], highlights the
importance of addressing these biases to ensure fair treatment for all
students.

Exclusion Bias: Exclusion Bias refers to the exclusion of certain groups due
to the design, operation, or implementation of AI systems. In the context of
educational AI, this might manifest in user interfaces, content, or
interaction modalities that are not accessible to all students, such as those
with disabilities or those from non-English-speaking backgrounds. Exclusion
bias can prevent these students from fully engaging with and benefiting from
AI-enhanced educational experiences. Research by [121] and [122] discuss the
legal and ethical implications of exclusion in digital systems, highlighting
the necessity for inclusive design. Moreover, studies by [123] and [124]
emphasize the role of inclusive design in educational technology, advocating
for the active involvement of diverse user groups in the design process to
ensure that their needs and preferences are adequately represented [125].
Exclusion bias is primarily a group bias because it systematically affects
entire groups of individuals who share certain characteristics, such as
disabilities or language differences, rather than affecting individuals
randomly. This kind of bias can reinforce existing inequities and limit
opportunities for marginalized groups in educational settings.

##  4 Fairness in AI in Educational Settings

As we discussed various types of biases that exist in AI systems in the
previous section 3, many studies have proposed various approaches to mitigate
those biases and fairness notions and their metrics to measure fairness.

###  4.1 Notations

Table 1: Notations Notation |  Definition
---|---
$Y$ |  Class attribute with binary values $\\{+,-\\}$, representing the positive and negative outcomes in model.
$A$ |  Binary sensitive attribute with values $\\{a,\bar{a}\\}$, indicating membership in a protected or non-protected group.
$a$ |  Protected group, the group that is safe against discrimination by fairness constraints.
$\bar{a}$ |  Non-protected group, the group that is not main focus of fairness measures but is important for analysis.
$\hat{Y}$ |  Predicted outcome by the model, aims to approximate the true class attribute $Y$
$TP$ |  True Positive, where the model correctly predicts a positive outcome.
$FN$ |  False Negative, where the model incorrectly predicts a negative outcome for a positive case.
$TN$ |  True Negative, where the model correctly predicts a negative outcome.
$FP$ |  False Positive, where the model incorrectly predicts a positive outcome for a negative case.
$f(\cdot)$ |  Predictive model function, mapping input features to the predicted outcome.
$dY(\cdot)$ |  Distance metric in output space, measuring similarity between predicted and actual outcomes.
$dX(\cdot)$ |  Distance metric in input space, quantifying the difference between predicted and actual outcomes.
$L$ |  Lipschitz constant, that rescales input distance between input variables.
$\hat{y}_{i}$ |  Probabilistic classification output for instance $x_{i}$
$x_{j}$ |  Top $k$-nearest neighbors for $x_{i}$, used in fairness analysis to evaluate local decision boundaries.

Let D be the binary classification dataset for a class attribute
$Y=\\{+,-\\}$, such as $Y=\\{\text{pass},\text{fail}\\}$.

Let $A$ be a binary sensitive attribute, where $A\in\\{a,\bar{a}\\}$, e.g.,
$A=$ “gender”, with $A\in\\{\text{female},\text{male}\\}$. In this context,
$a$ represents the discriminated or protected group, e.g., “female,” and
$\bar{a}$ represents the non-discriminated or non-protected group, e.g.,
“male”. The predicted outcome of the classification is denoted as
$\hat{Y}=\\{+,-\\}$.

To further refine the notation and differentiate between groups and their
outcomes:

  * •

$a^{+}$ and $a^{-}$ denote instances where the protected group (e.g., females)
are classified as positive ($+$) and negative ($-$), respectively.

  * •

$\bar{a}^{+}$ and $\bar{a}^{-}$ represent instances where the non-protected
group (e.g., males) are classified as positive ($+$) and negative ($-$),
respectively.

The metrics True Positive (TP), False Negative (FN), True Negative (TN), and
False Positive (FP) are used to evaluate the model’s predictive accuracy. The
function $f(\cdot)$ maps input features to the predicted outcomes, while
$dY(\cdot)$ and $dX(\cdot)$ serve as distance metrics in output and input
spaces, respectively. The Lipschitz constant $L$ rescales input distances, and
$\hat{y}_{i}$ represents probabilistic classification outputs for instances
$x_{i}$. The notation $x_{j}$ refers to the top $k$-nearest neighbors of
$x_{i}$, important for analyzing local fairness at decision boundaries.

###  4.2 Fairness Notions

AI fairness in the educational context refers to the principle of ensuring
that the use of AI systems and algorithms in educational settings does not
lead to unfair or biased outcomes for students [12]. This section will discuss
the two main fairness notions in education: individual fairness and group
fairness, followed by the metrics to measure them in Section 4.3.

Individual Fairness: Individual fairness in education refers to the principle
that students with similar academic abilities and educational backgrounds
should receive similar treatment, regardless of their demographic
characteristics. This fairness notion ensures that educational opportunities,
resources, and outcomes are distributed equitably among students, rather than
being influenced by factors such as race, gender, or socioeconomic status
[126, 127]. For example, students who demonstrate comparable academic
performance and have similar potential should have access to comparable
educational resources, experiences, support, and outcomes, without any
discrimination or bias [128, 129]. Implementing IF in education can involve
techniques like personalized learning, adaptive assessment, etc., to address
the unique needs and circumstances of each student [130].

Group Fairness: Group fairness in education refers to the principle that
groups of students should be treated fairly, regardless of their individual
characteristics. This approach aims to minimize disparities in educational
opportunities, resources, and outcomes based on group membership, such as
race, gender, or socioeconomic status [127, 131]. Group fairness measures in
educational data mining and student prediction models focus on ensuring that
the model’s predictions or decisions do not systematically favor or
disadvantage any particular group [130]. It is essential to ensure that these
applications avoid inequitable outcomes and negative impacts on learning for
certain groups of students [131].

###  4.3 Fairness Evaluating Resourses

In this section, we build on the fairness notions introduced in the previous
Section 4.2 and further introduce its corresponding metrics. Specifically, we
detail the mathematical equations employed to measure individual and group
fairness of the model in an educational context and introduce a method for
measuring the trade-off between model fairness and performance.

####  4.3.1 Individual Fairness

This concept emphasizes fairness at the individual level, ensuring that
similar individuals receive similar outcomes, and it does not consider any
sensitive attribute. While individual fairness offers a more granular approach
to fairness, it is less commonly addressed in the literature compared to group
fairness. Below are some metrics and definitions commonly used to evaluate
individual fairness:

Distance-Based Individual Fairness: One prevalent approach to achieving
individual fairness is through the use of distance-based metrics, which adhere
to the Lipschitz condition. This condition stipulates that the distance
between outcomes for any two individuals (e.g., students) in the output space
should not exceed their corresponding distance in the input space, typically
adjusted by a scalar factor (e.g., Lipschitz constant). The mathematical
formulation of the Lipschitz condition is as follows:

$d_{Y}(f(x_{i}),f(x_{j}))\leq L\cdot d_{X}(x_{i},x_{j})$ (1)

where $f(\cdot)$ is the predictive model that gives the individual level
output and $d_{Y}(\cdot)$ and $d_{X}(\cdot)$ are the distance metrics in the
output and input space, respectively. $L$ is the Lipschitz constant that
rescales the input distance between $x_{i}$ and $x_{j}$.

Existing work measures individual fairness based on the Lipschitz condition by
measuring the average distance of the output between each individual and its
$k$-nearest neighbors. Generally, a larger average distance indicates a lower
level of individual fairness for the algorithm outputs.

$1-\frac{1}{n\cdot
k}\left|\sum_{i=1}^{n}\hat{y}_{i}-\sum_{j=1}^{k}\hat{y}_{j}\right|,$ (2)

where $\hat{y}_{i}$ is the probabilistic classification output for $x_{i}$,
and $x_{j}$ is the top $k$-nearest neighbors for $x_{i}$.

Ranking-Based Individual Fairness: Although the Lipschitz condition has been
widely used as an individual fairness criterion, it could be problematic in
real-world applications. Specifically, it bases fairness on the premise that
similar inputs should yield similar outputs, gauged by absolute distances.
However, this approach may not account for variations across individuals, nor
does it easily accommodate comparisons between the distances in two different
spaces due to scale differences, which makes the appropriate Lipschitz
constant hard to determine. To this end, existing work aims to evaluate
individual fairness through a ranking perspective. Specifically, it utilizes a
similarity matrix $S$ to represent pairwise similarities among individuals
within the input space, leading to the generation of a ranking list $R_{1}$
for each individual that reflects their relative similarities to others. A
corresponding ranking list $R_{2}$, derived from similarities in the output
space, allows for the assessment of fairness through comparison of the two
lists. Ideally, achieving fairness means two ranking lists ($R_{1}$ and
$R_{2}$) are the same for each individual. Nevertheless, such a criterion is
hard to be satisfied. In practice, the average top-$k$ similarity between
$R_{1}$ and $R_{2}$ over all individuals is adopted to measure individual
fairness, where Normalized Discounted Cumulative Gain (NDCG@$k$) [132] and
Expected Reciprocal Rank (ERR@$k$) [133] are two common ranking similarity
metrics, offering a practical way to measure individual fairness in ranking
perspective.

####  4.3.2 Group Fairness

The concept of group fairness aims to ensure equitable outcomes across
different demographic groups defined by sensitive attributes like race,
gender, or age. We will discuss fundamental metrics and principles that
underpin group fairness, using the “Student performance prediction” scenario
as a case study [134].

Statistical Parity: Also known as Demographic Parity, this measure ensures
that the distribution of outcomes across different demographic groups is equal
within a specified tolerance $\epsilon$ [128]. For a classifier, achieving
statistical parity means that the difference in the probability of favorable
outcomes ($\hat{Y}$) between any two groups—denoted as $a$ (protected) and
$\bar{a}$ (non-protected)—should not exceed $\epsilon$:

$P(\hat{Y}|A=a)-P(\hat{Y}|A=\bar{a})\leq\epsilon.$ (3)

Statistical Parity is calculated as the difference in the probability of a
positive predicted outcome for the protected group ($a$) versus that for the
non-protected group ($\bar{a}$):

$\text{Statistical Parity}=P(\hat{Y}=+|A=a)-P(\hat{Y}=+|A=\bar{a}).$ (4)

where the value of statistical parity ranges from $[-1,1]$. An SP of 0
indicates no discrimination, values in the range $(0,1]$ suggest
discrimination against the protected group ($a$), and values in $[-1,0)$
indicate reverse discrimination against the non-protected group ($\bar{a}$).

Equal Opportunity: This concept focuses on fairness regarding positive
outcomes in a binary classifier. It is defined such that the probability of a
positive predicted outcome $\hat{Y}$ should be the same for all groups with
respect to a sensitive attribute $A$ and the true outcome $Y$ [135]:

$P(\hat{Y}=+|A=a,Y=+)=P(\hat{Y}=+|A=\bar{a},Y=+).$ (5)

In simpler terms, both the protected ($a$) and non-protected ($\bar{a}$)
groups should exhibit equal true positive rates (TPR), where TPR is defined as
the ratio of true positives (TP) to the sum of true positives and false
negatives (FN):

$\text{TPR}=\frac{TP}{TP+FN}$ (6)

This definition implies that a classifier should yield similar results for
members of both groups who are actually in the “pass” class. A classifier that
maintains equal false negative rates (FNR), where FNR is calculated as:

$\text{FNR}=\frac{FN}{TP+FN}$ (7)

will inherently possess equal TPRs as well. The measure of equal opportunity
can be quantified by:

$\text{Equal
Opportunity}=\left|P(\hat{Y}=-|Y=+,A=a)-P(\hat{Y}=-|Y=+,A=\bar{a})\right|.$
(8)

The value of equal opportunity ranges from 0 to 1, with 0 indicating no
discrimination and 1 signifying maximum discrimination.

Equalized Odds: Equalized Odds [135] w.r.t the sensitive attribute $A$ and
class label $Y$, is defined based on the condition that the predicted outcome
$\hat{Y}$ and sensitive attributed A are independent, given the actual outcome
$Y$. Specifically, the probabilities of true positive and false positive
predictions should be identical across both protected and non-protected
groups($a$, $\bar{a}$) i.e., female and male student groups. Formally, the
condition for equalized odds is expressed as:

$P(\hat{Y}=+|A=a,Y=y)=P(\hat{Y}=+|A=\bar{a},Y=y),\quad\text{where
}y\in\\{+,-\\}.$ (9)

Based on this definition, the measurement of equalized odds can be formulated
as follows:

$\text{Equalized
Odds}=\sum_{y\in\\{+,-\\}}\left|P(\hat{Y}=+|A=a,Y=y)-P(\hat{Y}=+|A=\bar{a},Y=y)\right|.$
(10)

The range of equalized odds is $[0,2]$, where a value of 0 indicates no
discrimination and a value of 2 suggests maximum discrimination.

Predictive Parity: Predictive Parity is satisfied if both protected and non-
protected groups have an equal positive predictive value (PPV) or precision
between protected groups, as follows [136].

$\text{PPV}=\frac{TP}{TP+FP}$ (11)

This denotes the probability of an individual predicted to “pass” actually
belonging to the “pass” class. According to the principle of predictive
parity, this probability should be identical for both the protected (e.g.,
females) and non-protected (e.g., males) groups. Formally, using the modified
variables, this is represented as:

$P(Y=+|\hat{Y}=+,A=a)=P(Y=+|\hat{Y}=+,A=\bar{a})$ (12)

where the measure of predictive parity can be reformulated as:

$\text{Predictive Parity}=|P(Y=+|\hat{Y}=+,A=a)-P(Y=+|\hat{Y}=+,A=\bar{a})|$
(13)

where predictive parity ranges from $0$ to $1$, where $0$ indicates the
absence of discrimination and $1$ signifies maximum discrimination. This
redefined formula maintains the core objective of evaluating fairness in
predictive models by ensuring that precision is consistent across both
protected and non-protected groups, thereby fostering equality in decision-
making processes.

Predictive Equality: Also referred to as False Positive Rate (FPR) balance, is
aimed at ensuring the equality of decision accuracy across protected and non-
protected groups [137, 136]. Specifically, it seeks to equalize the
probability of individuals from a discriminated group (protected group) who
are actually in the “fail” class but are incorrectly predicted to be in the
“pass” class, with that of individuals from the non-discriminated group (non-
protected group) [137]. Using the variables $A$ for the sensitive attribute
(e.g., gender), $a$ for the protected group (e.g., females), and $\bar{a}$ for
the non-protected group (e.g., males), the predictive equality can be defined
as:

$P(\hat{Y}=+|Y=-,A=a)=P(\hat{Y}=+|Y=-,A=\bar{a})$ (14)

In practice, the measure of predictive equality is reported by the difference
in False Positive Rates (FPRs) between the protected and non-protected groups:

$\text{Predictive
Equality}=\left|P(\hat{Y}=+|Y=-,A=a)-P(\hat{Y}=+|Y=-,A=\bar{a})\right|$ (15)

The value of predictive equality ranges from $0$ to $1$, where $0$ indicates
no discrimination, and $1$ indicates maximum discrimination between the
protected and non-protected groups. This metric is crucial for assessing
fairness in binary classification tasks, ensuring that both groups are treated
equally in terms of predictive errors.

####  4.3.3 Fairness Bonded Utility

The Fairness Bonded Utility (FBU) framework addresses the issue of finding a
balance between model performance and fairness in machine learning. Striking
the right balance between fairness and model performance is a challenging
task. FBU tackles this issue by organizing the impacts of different fairness
techniques into five clear levels of effectiveness. These scenarios can vary,
with some resulting in improvements in both fairness and performance, while
others may lead to a decrease in both. The categorization is determined by
assessing the fluctuation in model performance when different fairness
techniques are applied. These variations are then presented visually on a two-
dimensional coordinate system.

Within the FBU framework, a two-dimensional coordinate system is utilized to
illustrate the correlation between model performance and fairness, as shown in
graph (a) in Fig. 2. This requires the creation of pseudo-models $N_{\beta}$,
where $\beta$, represents various ratios at which the original model’s
predictions are replaced with a constant output label, ranging from 10% to
100%. This substitution aims to simulate different levels of fairness
interventions, with the idea that increasing the number of replaced
predictions should ideally improve fairness, even if it may impact accuracy.

The trade-off baseline, a fundamental concept in FBU, employs the principle of
zero-normalization. This principle aims to reduce bias by ensuring that all
individuals are assigned the same label, even if it may impact performance.
Through the application of this principle, several pseudo-models (a variety of
different proportions) are generated, each substituting a different value
$\beta$. $N_{\beta}$ compare the original model’s predictions with the same
output label. The baseline is demonstrated by connecting points obtained from
the original model and the pseudo-models, revealing the balance between
fairness and performance as more predictions are systematically substituted.

FBU divides the impact of bias mitigation techniques into five effectiveness
levels (see in graph (b) Fig. 2):

  * •

Level 1: Jointly Advantageous, a technique belongs in this level if it
improves both model performance and fairness relative to the trade-off
baseline.

  * •

Level 2: Impressive, technique in this level enhances either model performance
or fairness compared to the trade-off baseline, making it overall better than
the trade-off baseline..

  * •

Level 3: Reversed, if a technique improves model performance but leads to a
decrease in fairness.

  * •

Level 4: Deficient, it leads to a decline in either model performance or
fairness compared to the baseline, making it overall worse than the trade-off
baseline.

  * •

Level 5: Jointly Disadvantageous, describing techniques that reduce both
performance and fairness, representing the least favorable outcome.

Figure 2: Visualizations of the Fairness Bonded Utility (FBU)

The quantitative evaluation in FBU focuses on the “Impressive” region,
measuring the area enclosed by the fairness-performance points relative to the
baseline to determine the trade-offs’ desirability. Larger areas indicate more
favorable trade-offs, allowing for a fair comparison of bias mitigation
techniques based on their effectiveness in balancing fairness and performance,
even when the baseline exhibits curvature.

By providing a clear, quantifiable framework for assessing the impact of
fairness techniques on model fairness and performance, FBU simplifies the
decision-making process for selecting the most appropriate fairness technique.
This comprehensive approach enables a nuanced understanding of the trade-offs
involved, facilitating a more informed and sophisticated approach to achieving
fairness in machine learning applications [25].

###  4.4 Approaches to Improve Fairness

In order to address the aforementioned problems and biases (as outlined in
Section 3.5), researchers and practitioners have proposed various approaches
to mitigate bias in AI. These approaches include pre-processing, in-
processing, and post-processing techniques [12], [138]. Each method addresses
distinct parts of the modeling pipeline, giving developers a toolset to
improve AI fairness as well as enhance fairness in AI tools, This section
examines ways for mitigating bias (see Fig. 3).

####  4.4.1 Pre-Processing

Pre-processing techniques are aimed at identifying and reducing bias in the
dataset before it is used to train an AI model [12]. These techniques are
particularly important in educational settings, where data might reflect
historical or societal biases that could lead to unfair outcomes when used by
AI systems. The following are key pre-processing techniques used to enhance
fairness:

Re-weighing: Reweighting is a crucial strategy employed in educational
environments to enhance the fairness of AI models. This technique involves
modifying the weights assigned to instances within datasets, so guaranteeing
that student groups, particularly those that have historically been
underrepresented, are accorded equal significance during the training process
of the model. The utilization of this approach is crucial in mitigating biases
existing in educational data, hence facilitating a more equitable portrayal of
heterogeneous student cohorts. In their study [139], Calmon and others
introduced a reweighting methodology that effectively addresses the issue of
bias reduction while maintaining the usefulness of the data. They formulated
this approach as an optimization problem. The equilibrium is crucial in
educational settings, where the objective is not alone to properly forecast
results but also to guarantee impartiality and equality among different
student populations. Additional research, exemplified by [140], highlights the
importance of reweighting in reducing prejudiced biases in the processing of
educational data. This, in turn, fosters the development of more inclusive and
fair AI applications in the field of education. Hajian and Domingo-Ferrer
(2013) emphasize the efficacy of the method in improving the equity of
educational predictive models, hence emphasizing its capacity to cultivate an
educational milieu that prioritizes diversity and equality.

Resampling: In the context of educational AI applications, resampling
approaches are of paramount importance in mitigating dataset discrepancies and
promoting equitable representation among a wide range of student demographics.
The objective of these strategies is to enhance the balance of datasets for
predictive modeling in the field of education by either oversampling minority
groups or undersampling dominant groups. This methodology holds significant
importance in various contexts, including the prediction of student
performance, admission procedures, and the identification of kids at risk. In
these circumstances, the presence of skewed data has the potential to result
in inequitable outcomes. The efficacy of resampling in mitigating bias in
educational data mining and enhancing fairness in automated decision-making is
exemplified in a study conducted by [140]. In a similar vein, [141] employed
resampling techniques to forecast student achievement rates in digital
educational environments, emphasizing its capacity to augment the precision
and equity of predictive models. Resampling approaches are used to modify the
representation of various student groups in order to prevent AI models in
education from perpetuating existing disparities. This, in turn, promotes
equal educational opportunities for all students.Resampling techniques,
including oversampling and undersampling, aim to correct imbalances in
datasets by adjusting the representation of various groups to promote fairness
in AI models:

Over Sampling: Oversampling techniques such as the Synthetic Minority Over-
sampling Technique (SMOTE) and Generative Adversarial Networks (GANs) play a
crucial role in addressing class imbalances related to student demographics
and learning outcomes. SMOTE, introduced by [142] and GANs, conceptualized by
[143], are particularly beneficial for creating balanced datasets that
accurately represent diverse student groups. SMOTE generates new examples
within the minority class by interpolating between existing instances, thereby
enriching the dataset with a broader representation of underrepresented
student populations. On the other hand, GANs use a competitive network
framework to create synthetic data that mirrors the minority class, offering a
novel approach to augmenting datasets without merely duplicating instances.

These oversampling methods are instrumental in educational settings where the
equitable representation of all student groups is vital for developing fair
and effective predictive models. For instance, in predicting student success,
these techniques can help ensure that models are trained on data that
adequately represents students from various backgrounds, thus avoiding biases
against underrepresented groups [144]. However, the application of these
methods must be approached with caution due to potential challenges such as
overfitting and the accurate reflection of intra-group diversity. Ensuring
that synthetic instances generated through SMOTE or GANs genuinely represent
the complexity of minority groups is critical for maintaining both the
fairness and accuracy of predictive models in education [140]. The strategic
use of oversampling in educational data science necessitates careful
validation and a nuanced understanding of how synthetic data impacts model
performance and fairness. By meticulously implementing and evaluating these
techniques, educators and data scientists can significantly improve the
inclusivity and effectiveness of educational AI applications.

Under Sampling: Under-sampling techniques, notably Random Under Sampling (RUS)
and the NearMiss algorithm, play a pivotal role in rectifying class imbalances
within educational data. These strategies aim to neutralize the
overrepresentation of majority classes, thereby facilitating the development
of unbiased educational models. RUS simplifies the process by randomly
removing instances from the majority class [145], whereas NearMiss adopts a
more strategic approach, selecting instances close to the minority class to
ensure the data retained is most useful for training. Such methodologies are
indispensable for constructing predictive models in education that impartially
forecast student outcomes, ranging from dropout risks to underperformance
identification, without favoring more prevalent classes [146].

However, the application of under-sampling presents notable challenges,
particularly the risk of losing significant information when majority class
instances are eliminated. This loss could result in models that, despite being
balanced, fail to comprehensively represent the diverse range of student
performances and characteristics, potentially undermining the model’s
effectiveness and fairness [147, 148]. Therefore, in educational settings, the
deployment of under-sampling strategies demands careful consideration and
evaluation. It is crucial for educators and data scientists to strike an
optimal balance between fairness and accuracy, ensuring that the models not
only promote equitable educational outcomes but also retain a detailed insight
into student behaviors and achievements [149, 150].

Figure 3: The Overall Architecture of The Algorithm Fairness

####  4.4.2 In-processing Techniques

In-processing techniques involve modifying the learning algorithms to
incorporate fairness constraints, ensuring that the resulting models make fair
decisions across different groups [12]. These techniques are diverse, ranging
from regularization methods to adversarial debiasing, and are crucial for
developing equitable AI systems in education.

Regularization Methods: Regularization techniques introduce a penalty term to
the model’s loss function, which aims to minimize prediction error and
mitigate bias within the model. This approach is highly beneficial in AIED
systems to ensure equitable performance among diverse student groups. Few
pioneers introduced a fairness-aware regularizer that mitigates predictions
leading to unequal opportunities, thereby ensuring balanced model performance
across different demographic groups [151]. This concept was further expanded
by [152], who developed a multi-faceted regularization framework that
simultaneously addresses various aspects of fairness, enhancing the robustness
of AIED systems against biases.

Constraint-based Optimization: In educational data analysis, constraint-based
optimization applies fairness as a concrete constraint within the optimization
problems solved by machine learning algorithms. This method explicitly
incorporates fairness criteria, such as ensuring equitable treatment across
different demographic groups (e.g., gender or ethnicity) in student
performance predictions or admissions processes. Zafar in [153] introduced a
groundbreaking approach by embedding fairness constraints into convex
optimization problems, thus allowing the development of models that uphold
fairness without significantly sacrificing accuracy. This technique ensures
that predictive models in education do not perpetuate or exacerbate existing
disparities among student groups.

Further extending this approach, Donini and others in [154] proposed a kernel-
based optimization framework capable of integrating multiple fairness
constraints. This advancement has significantly enhanced the flexibility and
applicability of constraint-based optimization, making it possible to address
complex fairness issues in AI models used in education. For instance, these
models can be employed to ensure that predictive analytics tools used for
student success forecasting or resource allocation are fair and unbiased
across different student demographics.

In the context of education, implementing these optimization techniques can
lead to more equitable educational technologies. For example, Grgić-Hlača and
others demonstrated the use of constraint-based optimization to mitigate
biases in automated essay scoring systems, ensuring fair treatment across
students from various backgrounds [155]. Similarly, Saleiro applied these
principles to develop a fair early warning system for identifying students at
risk of dropping out, highlighting the method’s potential to foster equity in
educational outcomes [156].

Adversarial Debiasing: Adversarial debiasing in the educational context
entails the use of an innovative methodology aimed at constructing equitable
and impartial predictive models. The approach employed in this study employs a
dual-model framework, wherein the primary model is responsible for forecasting
educational outcomes, such as student performance or admission decisions.
Simultaneously, a secondary model, referred to as the adversary, is designed
to predict sensitive attributes, such as race or gender, based on the
predictions generated by the primary model. The objective is to enhance the
core model in a manner that prevents the opponent from utilizing its
predictions to precisely deduce sensitive qualities, hence guaranteeing
impartial and fair educational predictions. In their study, Wang and Cadamuro
utilized this methodology to mitigate racial prejudice in predictive models
for student success, showcasing its capacity to generate more equitable
educational forecasts [25]. This methodology is consistent with the ideals of
equity in the field of educational data science, promoting the use of models
that promote fair outcomes among various student populations. Smith and Neff
investigated the use of adversarial debiasing to reduce gender bias in online
course suggestions [157]. They emphasized the adaptability of this approach in
many educational contexts. The aforementioned findings underscore the efficacy
of adversarial debiasing in fostering equity and mitigating bias within
educational environments, hence facilitating the development of educational
technologies that are more inclusive and impartial.

####  4.4.3 Post-Processing Techniques

Post-processing techniques in AI involve adjusting the output of already
trained models to meet certain fairness criteria. These techniques are
particularly useful when modifying the training data or the learning algorithm
itself is not feasible [12]. In the context of AIED, post-processing is
crucial for ensuring that AI-driven educational tools do not perpetuate or
exacerbate existing inequalities among learners. This section delves into
various post-processing approaches to enhance fairness in AI models used
within educational settings.

Threshold Adjustment: In the realm of threshold adjustment for post-processing
in AI, several methodologies aim to enhance fairness in educational systems. A
notable approach is proposed by [158], where fairness constraints are
integrated into ranking algorithms, applicable for post-processing
adjustments. This is crucial for educational AI systems that rank students and
resources, ensuring fairness in the ranking process. Similarly, [159]
introduces an optimization-based strategy for adjusting decision thresholds in
machine learning models to meet specific fairness objectives, such as
equitable student evaluations or resource distribution. These methods align
with the foundational principles outlined by [160] in their work on achieving
demographic parity through threshold adjustments, emphasizing the need for
careful calibration to maintain the balance between fairness and model
accuracy. Collectively, these studies underscore the importance of meticulous
threshold adjustment in educational AI systems to promote equitable outcomes
without compromising the integrity of the models predictions.

Outcome Perturbation:  Outcome perturbation, a method that introduces noise to
a model’s predictions to mitigate biases, can be significantly enhanced by the
methodologies presented by [161] and [162]. Johndrow and Lum’s algorithm,
originally designed for race-independent recidivism prediction, offers a
blueprint for obscuring sensitive information in predictions, making it
adaptable for educational AI systems to reduce biases without compromising
outcome utility. Similarly, Feldman and others approach to certifying and
removing disparate impacts provides a systematic way to measure and mitigate
bias through outcome perturbation, ensuring that educational AI predictions do
not disproportionately disadvantage any student group. Together, these
methodologies underscore the potential of outcome perturbation to maintain
fairness in AI-driven educational settings, while also preserving the privacy
and integrity of individual data points through techniques like differential
privacy.

##  5 Ethical Considerations and Frameworks in AI for Education

This section explores the ethical dilemmas linked to the utilization of AI in
education, emphasizing the need for AI specialists to possess a profound
ethical comprehension. The subject matter encompasses ethical quandaries such
as prejudice, equity, confidentiality, and the social ramifications of AI
technology. Furthermore, it provides a thorough examination of worldwide
standards, rules, and ethical frameworks that seek to lead the advancement and
integration of artificial intelligence in the field of education towards more
ethical, fair, and inclusive approaches, as outlined in the accompanying
table.

###  5.1 Ethical Dilemmas

It is very crucial to acknowledge that the dilemmas in AIED are not solely
technical but also profoundly ethical. The ethical education in AI involves
instilling a deep understanding of the societal impacts of AI technologies.
One of the central ethical dilemmas is the bias and fairness in AI algorithms.
As highlighted in [98], algorithms can perpetuate or even exacerbate existing
societal biases if not carefully designed and scrutinized. This raises the
need for AI education to incorporate critical thinking about the ethical
design and deployment of AI systems.

Privacy and data protection are other significant ethical concerns. With AI
systems often trained on vast datasets, including personal information, there
is a pressing need to educate future AI professionals about data ethics,
consent, and privacy laws [163]. Moreover, the potential for AI to automate
jobs and impact employment introduces ethical considerations regarding the
responsible development and implementation of AI technologies, emphasizing the
need for AI education to address the societal implications of AI [164].

The challenge of ensuring ethical AI education also extends to the development
of AI systems that are transparent, explainable, and accountable. As [165]
argues, there is an urgent need for AI systems to be understandable by humans,
not only to foster trust but also to ensure that decisions made by AI systems
can be scrutinized and challenged. This necessitates an educational focus on
the interpretability of AI models and the ethical implications of opaque,
“black-box” systems.

###  5.2 Guidelines, Policies, and Ethical Frameworks

The landscape of guidelines, policies, and regulations for AIED reflects a
growing global commitment to ethical, equitable, and inclusive AI practices.
UNESCO’s recommendations on the ethics of AI and its guidance for policy-
makers emphasize the importance of transparency, accountability, and
inclusiveness, urging stakeholders to consider the ethical implications of AI
in educational settings. The European Commission’s ethics guidelines for
trustworthy AI propose a set of key requirements for AI systems, including
fairness and non-discrimination, which are crucial for educational
technologies. The UK’s national AI strategy highlights ethical AI use, with a
focus on fairness and bias reduction in education. The IEEE’s ethical
standards and the OECD’s AI Principles both advocate respect for learner
rights and the inclusion of diverse perspectives in AI development for
education. The Montreal Declaration and the ITU’s AI for Good initiative
further stress the need for AI to promote well-being, respect autonomy, and
advance equitable education for all. Collectively, these documents form a
comprehensive framework aimed at guiding the responsible development and
deployment of AI in educational contexts, ensuring that these technologies
support equitable learning opportunities and outcomes(see Table 2).

Type |  Document Title |  Organization |  Country |  Year
---|---|---|---|---
Ethical Guidelines |  UNESCO’s recommendations on the ethics of AI |  UNESCO |  International |  2019
Policy Framework |  AI in education: Guidance for policy-makers |  UNESCO |  International |  2020
Regulation |  Ethics guidelines for trustworthy AI |  European Commission |  European Union |  2019
National Strategy |  National AI strategy |  UK Government |  United Kingdom |  2021
Ethical Standards |  AI and ethics in education |  IEEE Standards Association |  International |  2020
Policy Guidelines |  AI Principles |  OECD |  International |  2019
Ethical Framework |  Montreal declaration for a responsible development of Artificial Intelligence |  Université de Montréal |  Canada |  2018
Regulatory Guidelines |  AI for Good |  ITU |  International |  Ongoing

Table 2: Guidelines, policies, and regulations addressing fairness in AIED

##  6 Tools and Datasets

This section explores various tools and datasets that are used for mitigating
bias in educational settings.

###  6.1 Tools

Numerous tools and frameworks have emerged to assist researchers and
practitioners in recognizing and resolving biases in AI models and datasets.
Presented below is a compilation of frequently utilized tools in the field of
AIED for identifying and addressing bias:

AI Fairness 360 (AIF360)555https://aif360.res.ibm.com/: Developed by IBM
Research, AI Fairness 360 is an extensible open-source toolkit designed to
help researchers and practitioners detect, understand, and mitigate bias in
machine learning models. It includes a comprehensive set of metrics for
datasets and models to test for biases, along with algorithms to mitigate bias
in machine learning pipelines [166].

Fairlearn666https://fairlearn.org/: Fairlearn is a Python package that aims to
empower data scientists and developers to assess and improve the fairness of
their AI systems. It provides algorithms to mitigate unfairness in supervised
machine learning models and tools to evaluate the models for fairness issues
[167].

TensorFlow Fairness
Indicators777https://www.tensorflow.org/responsible_ai/fairness_indicators/guide:
TensorFlow Fairness Indicators is an open-source library that enables easy
computation of commonly-identified fairness metrics for binary and multi-class
classifiers. It is designed to help users evaluate their models’ performance
across different groups, especially in large-scale machine learning workflows
[168].

What-If Tool888https://pair-code.github.io/what-if-tool/: The What-If Tool
(WIT) is an interactive visual interface designed by Google for probing
machine learning models. It allows users to analyze how changes to the input
data points or the model itself can affect the model’s predictions, providing
insights into the model’s performance across different groups [169].

FairTest999https://fairtest.org/: FairTest is a tool aimed at uncovering
unwarranted associations between an application’s outcomes (e.g., predictions
of an AI system) and sensitive attributes (e.g., race or gender). It provides
developers with investigative tools to discover and debug biases [170].

FairML101010https://github.com/adebayoj/fairml: FairML offers a Python-based
solution for evaluating bias in machine learning models. This comprehensive
toolkit measures the importance of each input in the model. By utilizing model
compression and four distinct input ranking methods, FairML determines how
much a model relies on specific inputs for its predictions. This information
can be instrumental in gauging the model’s fairness and visualizing the
independence of features [171].

Aequitas111111http://aequitas.dssg.io/: Aequitas is an open-source bias and
fairness audit toolkit for machine learning researchers to detect machine
learning models to bias and discrimination. Aequitas is often used in Binary
Classification and contains 13 descriptions of bias metrics. Most of the
metrics are described in the confusion matrix section.

###  6.2 Datasets

Various datasets are frequently employed in research pertaining to the
identification and reduction of bias, providing valuable insights into a wide
range of educational settings and student engagements. These datasets aid in
detecting biases associated with demographics, performance, and engagement,
hence enabling the creation of AI-driven educational systems that are more
inclusive and successful (see Table 3). Below are several noteworthy datasets:

Trends in International Mathematics and Science Study (TIMSS): This dataset
includes international assessments of math and science knowledge of fourth and
eighth graders, along with extensive background data on students, teachers,
and schools. TIMSS provides a basis for examining fairness in AI applications
across different educational systems and cultures [172].

Programme for International Student Assessment (PISA): Conducted by the OECD,
PISA assesses the educational performance of 15-year-olds in reading,
mathematics, and science across various countries. The dataset includes
background information, allowing for the analysis of fairness across different
national and socioeconomic contexts [173].

National Education Longitudinal Study (NELS:88): This dataset stems from a
United States study that began in 1988, tracking students’ educational
experiences and outcomes over time. It includes a wide range of variables such
as academic performance, socioeconomic status, and school characteristics,
making it suitable for fairness analysis in educational predictions [174].

EdNet: EdNet is a large-scale hierarchical dataset of student interactions
with an AI tutoring service, containing detailed traces of more than 100
million interactions. It’s used for studying personalized learning and
fairness in educational recommendations [175].

Education Longitudinal Study of 2002 (ELS:2002): A longitudinal study that
follows students who were in 10th grade in 2002 through their secondary and
post-secondary years. The dataset includes information on student achievement,
school experiences, and transitions to college and work [176].

Integrated Postsecondary Education Data System (IPEDS): A system of surveys
collecting information from every college, university, and technical and
vocational institution that participates in the federal student financial aid
programs in the United States [177].

Common Core of Data (CCD): A comprehensive, annual, national database of all
public elementary and secondary schools and school districts, which includes
information on school demographics, enrollment, and teacher counts [178].

KDDCUP 20157: This data set was used in the KDDCUP competition in 2015 [179].
It includes all students’ learning activities on the Xuetang platform in 2015,
combining 39 courses which enrol 180785 students. The goal is to predict
student dropouts. The KDDCUP2015 data set can be used to test sensitive
attributes of gender.

OULAD (The Open University Learning Analytics dataset):  The OULAD dataset is
an internationally publicly available learning analytics data set [180]. The
dataset contains data from 2013 to 2014 about courses, students, and their
interactions with the virtual learning environment for seven selected courses.
The target is to predict the students’ learning scores. The OULAD dataset can
be used to test gender or disability discrimination.

Dataset |  Domain | #Samples | #Features |  Fairness Goal |  sensitive attributes
---|---|---|---|---|---
TIMSS [172] |  Int. Ed. Assessment |  $>$300,000 |  $>$200 |  Group fairness |  Country (multiple), gender (2), educational resources (multiple)
PISA [173] |  Int. Ed. Assessment |  $>$500,000 |  $>$600 |  Group fairness |  Country (multiple), gender (2), socio-economic background (multiple)
NELS:88 [174] |  Longitudinal Ed. Study |  $>$25,000 |  $>$1,000 |  Both group and individual fairness |  Race (multiple), gender (2), SES (multiple)
EdNet [175] |  AI Tutoring Systems |  $>$100M interactions | 300 |  Individual fairness |  Various (multiple)
KDDCUP 2015 [179] |  Online Learning Analysis | 180,785 | 210 |  Individual fairness |  Gender (2)
OULAD [180] |  Online Learning Analysis | Varies | 6 |  Individual fairness |  Gender (2), disability (2)
TutOR [181] |  ITS Analysis | Varies | 150 |  Individual fairness |  Age (multiple)
MOOCdb [182] |  MOOCs Analysis | Varies | 22 |  Individual fairness |  Location (multiple), educational background (multiple)
CMU DataShop [183] |  ITS Analysis | Varies | 500 |  Individual fairness |  Various (multiple)
LAK [184] |  Ed. Research | Varies | 100 |  Individual fairness |  Educational level (multiple), discipline (multiple)
CBT [185] |  Reading Comp. Analysis | Varies | 50 |  Individual fairness |  Age (multiple)
ASSISTments [186] |  Online Tutoring Analysis |  $>$100,000 interactions | 200 |  Both group and individual fairness |  Socio-economic status (multiple), gender (2)
CCD [178] |  U.S. Public Ed. Analysis |  $>$100,000 schools |  $>$500 |  Group fairness |  Race (multiple), ethnicity (multiple), English proficiency (multiple), disability status (multiple)
IPEDS |  Post-secondary Ed. Analysis |  $>$7,000 institutions |  $>$200 |  Both group and individual fairness |  Race (multiple), gender (2), financial aid status (multiple)
ELS:2002 [176] |  Longitudinal Ed. Study | 15,000 |  $>$1,000 |  Both group and individual fairness |  Race (multiple), gender (2), SES (multiple), school type (multiple)

Table 3: Datasets Used in Fairness and Bias Research in Education

TutOR: This dataset is collected from an Intelligent Tutoring System (ITS) and
includes information on students’ ages and prior knowledge levels. It is
particularly useful for developing and evaluating adaptive learning systems
[181].

MOOCdb: Derived from Massive Open Online Courses (MOOCs), this dataset
contains information on students’ locations, educational backgrounds, and
interaction patterns. It is suited for learning analytics (LA) and clustering
analyses to understand engagement and learning behaviors across diverse
educational backgrounds [182].

CMU DataShop: Hosted by Carnegie Mellon University. This repository contains
datasets from various ITSs, covering a wide range of variables. It supports
diverse research in AIED, from student modeling to the evaluation of
instructional strategies [183].

LAK Dataset: Originating from the Learning Analytics and Knowledge conference,
this dataset encompasses data on educational levels and disciplines, suitable
for network analysis and learning analytics research to uncover patterns and
insights in academic performance and behaviors [184].

CBT (Computer-Based Tutoring): The CBT dataset focuses on reading levels and
includes data on students’ ages. It is especially suited for natural language
processing (NLP) and content analysis to study reading comprehension and
literacy development [185].

ASSISTments: This dataset originates from the ASSISTments online tutoring
platform and includes detailed student interaction data. It is valuable for
examining the impact of socio-economic status and gender on student
performance, especially in the context of educational data mining (EDM) and
predictive modeling [186].

##  7 Challenges and Future Directions

Despite the large number of approaches and metrics that are available in the
literature, there are still many gaps in this field of study. The primary
barriers to algorithmic fairness research are listed in this part to provide
an overview of possible future research topics.

Personalization Versus Data Privacy: Given the widespread use of personalized
learning, there is a need for more research on how artificial intelligence may
be most effectively utilized to promote personalized learning. This includes
tasks such as proposing learning materials tailored to individual learners and
providing guidance in problem-solving [187]. However, to provide customized
experiences, AI model training necessitates using large-scale learner data,
which may include extremely sensitive information. Occasionally, models
unintentionally retain training data that contains sensitive information,
which becomes exposed during the analysis of the model. Nevertheless, the full
capabilities of a machine learning model can only be actualized via the
examination of learners’ data [188]. Due to the inability of most existing
models to ensure that their output can be applied to different learners beyond
specific individual characteristics, as well as the potential risk to
learners’ data and the negative impact on the acceptance of AI in society, it
is necessary to restrict instructors’ access to learners’ data in order to
limit learners’ exposure to instructors’ knowledge appropriately. Educational
institutions should practice transparency about student data privacy to
address any misconceptions and concerns related to data use.

Data-Related Challenges: The challenge of biased data stands as a formidable
barrier to achieving fairness. AI systems are inherently shaped by the
historical data on which they are trained, data that often mirrors the
prejudices and societal inequities of the real world. This reality manifests
starkly in educational contexts, where datasets may underrepresent minority
groups or encapsulate biased assessments of student capabilities, thus
fostering a cycle where biases are not only perpetuated but amplified. The
consequences of leveraging such skewed datasets are significant, impeding the
AI’s learning efficacy and its capacity to render impartial decisions, thereby
exacerbating the challenge of ensuring equitable educational outcomes [189,
98]. To confront this issue, a pivotal future direction involves the
meticulous curation of representative and comprehensive datasets. This task
necessitates a concerted effort to integrate diverse demographic groups and
neutralize the historical biases ingrained in educational data. Approaches
like participatory data collection, where input from a wide array of
stakeholders enriches dataset development, emerge as essential strategies.
Furthermore, adopting data augmentation and creating synthetic data stand as
innovative methods to infuse diversity into training sets, thereby diminishing
the potential for biased AI outputs and steering the path towards more
equitable AI applications in education [190].

The Definition and Measurement of Fairness: The multifaceted concept of
fairness in AI presents a formidable challenge, as it defies a one-size-fits-
all definition, varying significantly across cultural, social, and individual
dimensions. This variability complicates the translation of abstract fairness
ideals into quantifiable metrics that can effectively guide the development
and evaluation of AI systems, leading to a lack of consensus on appropriate
fairness metrics within diverse contexts [191]. Addressing this ambiguity
demands a dual approach. First, the development of standardized fairness
metrics, specifically tailored to fit the educational sector, emerges as a
critical need. Such standardization requires an interdisciplinary effort,
combining insights from ethics, sociology, computer science, and educational
theory to capture the complexity of fairness in educational settings. Second,
adopting contextual approaches to fairness—where metrics are dynamically
adjusted to fit the unique characteristics of specific educational contexts
and their respective populations—offers a pathway to more relevant and
impactful fairness interventions. This approach acknowledges the contextual
nature of fairness, ensuring that AI applications are evaluated and refined in
alignment with the nuanced realities of their intended settings [113].

Trade-offs Between Fairness and Accuracy: The trade-off between fairness and
accuracy is a fundamental difficulty in the field of AI applications in
educational settings. The problem stems from the necessity to provide fair
results among varied student demographics while upholding the accuracy of
forecasting algorithms. The presence of trade-offs frequently requires making
concessions that can substantially impact the effectiveness and ethical
integrity of educational AI systems. The issue of reconciling fairness with
performance measurements is of utmost importance, especially considering the
significant consequences of AI-driven judgments in the field of education. The
future trend is towards the advancement of algorithms that consider fairness
and strategies for optimizing several objectives. The objective of these
developments is to include fairness issues exclusively in the optimization
process of the model, hence diminishing the need to make concessions on
accuracy. The objective of employing these novel modeling methodologies is to
balance fairness and performance, enabling AI systems in the field of
education to provide impartial conclusions while maintaining their predictive
capacities. The aforementioned strategy tackles the immediate technological
obstacles and corresponds with the wider ethical obligation to cultivate fair
and impartial educational settings by utilizing artificial intelligence [137].

Digital-divide: The digital divide poses a significant challenge in the
integration of AI in education, as highlighted by several sources [192]. The
study [193] highlights how the lack of access to digital infrastructure,
devices, and internet connectivity can exacerbate existing inequalities,
particularly in developing countries like Pakistan. emphasizes the need to
bridge this digital divide in physiotherapy education specifically. To address
this issue, future directions should focus on ensuring equitable access to
technology and digital resources. Providing necessary gadgets, internet
access, and training can help mitigate the digital divide’s impact on
students’ performance [194], recommends enhancing communication competence
through targeted awareness campaigns and infrastructure development. Moreover,
it underscores the importance of ethical considerations, such as data privacy
and addressing biases, in developing and implementing AI-powered educational
solutions. Collaboration between stakeholders, including government, industry,
and civil society organizations, is crucial for sustainable change [195].

Teacher and Student Adoption: The adaptation of teachers and students to AI in
education is a complex challenge that necessitates targeted strategies to
ensure successful implementation. Research underscores the importance of
developing adaptive learning systems that align with individual learning
styles and needs, necessitating comprehensive teacher training on the
effective use of AI tools and their integration into pedagogical practices
[196]. Interactive language programs and virtual learning platforms offer
personalized learning experiences; however, teachers need to be adept at
guiding students through these technologies to maximize their potential [197].
Additionally, the impact of students’ metacognitive awareness on their
engagement with AI tools like ChatGPT highlights the importance of providing
customized support and training to foster effective utilization [198].

Future initiatives should include the establishment of extensive training
programs for both educators and students, which could encompass workshops,
online resources, and collaborative learning communities [199]. These
resources would facilitate the sharing of best practices and collaborative
problem-solving to address emerging challenges. Moreover, the integration of
emotionally aware AI systems could significantly enhance classroom
interactions by providing real-time feedback based on non-verbal cues [200].
Nevertheless, the implementation of such technologies must also consider
ethical issues, including data privacy and the mitigation of biases, to
prevent unintended consequences.

##  8 Conclusion

This comprehensive survey paper delves into the concepts of fairness, bias,
and ethics in the realm of AI applications in education. It offers a critical
perspective on the evolution of AIED from basic computer-assisted instructions
to advanced adaptive learning systems. The paper emphasizes the significance
of adopting a multidisciplinary approach to address the complex challenges of
ensuring fairness in educational AI, including data-related, algorithmic, and
user-interaction biases. Through a thorough examination of existing literature
and frameworks, it not only highlights technical obstacles like the trade-off
between fairness and accuracy and the lack of diverse datasets but also
navigates the ethical quandaries associated with the implementation of AI in
educational settings. By outlining future pathways, such as the creation of
comprehensive fairness frameworks and the promotion of global cooperation,
this study underscores the continuous demand for innovative solutions that
harmonize technical efficiency with ethical obligations. It advocates a
collaborative endeavor involving researchers, practitioners, and policymakers
to leverage the potential of AI in enriching educational opportunities while
preventing the perpetuation of disparities, thereby ensuring a responsible and
fair integration of AI technologies in educational environments.

## References

  *   * Luckin and Cukurova [2022] Luckin, R., Cukurova, M.: A brief history of artificial intelligence in education. International Journal of Artificial Intelligence in Education 32(2), 155–175 (2022)
  * Full Estop [2024] Full Estop: Personalized Learning: How AI Is Shaping the Future of Education (2024). https://www.fullestop.com/blog/personalized-learning-how-ai-is-shaping-the-future-of-education
  * Daugherty and Wilson [2020] Daugherty, P.R., Wilson, H.J.: AI fairness isn’t just an ethical issue. Harvard Business Review (2020)
  * Review [2024] Review, T.P.: The Evolution of Education: How AI is Reshaping Grading (2024). https://www.princetonreview.com/ai-education/how-ai-is-reshaping-grading
  * DataScienceCentral.com [2024] DataScienceCentral.com: Automated Grading Systems: How AI is Revolutionizing Exam Evaluation (2024). https://www.datasciencecentral.com/automated-grading-systems-how-ai-is-revolutionizing-exam-evaluation/
  * Davis [2024] Davis, R.: How AI is Revolutionizing Grading: The Case of Top Marks AI (2024). https://www.topmarks.ai/how-is-ai-used-for-grading
  * Montgomery County Intermediate Unit [2024] Montgomery County Intermediate Unit: Addressing Ethical Considerations When Using AI in K-12 Classrooms (2024). https://learn.mciu.org/addressing-ethical-considerations-when-using-ai-in-k-12-classrooms/
  * Baker [2022] Baker, R.S.: The intertwined histories of artificial intelligence and education. International Journal of Artificial Intelligence in Education 32(1), 1–25 (2022) https://doi.org/10.1007/s40593-022-00313-2
  * Montgomery County Intermediate Unit [2024] Montgomery County Intermediate Unit: AI-Powered Tools: Automating eLearning Assessments For The Future (2024). https://elearningindustry.com/ai-powered-tools-automating-elearning-assessments-for-the-future
  * Culican [2024] Culican, J.: Understanding the Ethical Implications of AI in Education (2024). https://www.linkedin.com/pulse/understanding-ethical-implications-ai-education-jamie-culican
  * Paschall [2024] Paschall, T.: How can you use artificial intelligence in multilingual e-learning? (2024). https://www.linkedin.com/advice/1/how-can-you-use-artificial-intelligence-lzgnc
  * Ferrara [2023] Ferrara, E.: Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. Sci 6(1), 3 (2023) https://doi.org/10.3390/sci6010003
  * Porayska-Pomsta and Luckin [2022] Porayska-Pomsta, K., Luckin, R.: Issues with Data, Bias and Fairness. Pressbooks, ??? (2022). https://pressbooks.pub/aiforteachers/chapter/issues-with-data-bias-and-fairness/
  * Perrotta [2024] Perrotta, C.: Algorithmic Bias and Fairness in Education: A Very Brief Primer (2024). https://der.monash.edu/algorithmic-bias-and-fairness-in-education-a-very-brief-primer/
  * Zhang et al. [2024] Zhang, J., Zhu, L., Zhu, X., Cheng, Y., Zhu, Y., Zhu, X.: Grading by ai makes me feel fairer? how different evaluators affect college students’ perception of fairness. Frontiers in Psychology 15, 1221177 (2024)
  * Bi et al. [2022] Bi, S., Bonyadi, A., Nguyen, B.V., Ngo, T.C.T., Organ, M.: Artificial intelligence technologies in college english translation teaching. International Journal of Environmental Research and Public Health 19(8), 4568 (2022)
  * Wang et al. [2023] Wang, Z., Zhou, Y., Qiu, M., Haque, I., Brown, L., He, Y., Wang, J., Lo, D., Zhang, W.: Towards fair machine learning software: Understanding and addressing model bias through counterfactual thinking. arXiv preprint arXiv:2302.08018 (2023)
  * Yin et al. [2024] Yin, Z., Wang, Z., Zhang, W.: Improving fairness in machine learning software via counterfactual fairness thinking. In: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, pp. 420–421 (2024)
  * Dzuong et al. [2024] Dzuong, J., Wang, Z., Zhang, W.: Uncertain boundaries: Multidisciplinary approaches to copyright issues in generative ai. arXiv preprint arXiv:2404.08221 (2024)
  * Chinta et al. [2023] Chinta, S.V., Fernandes, K., Cheng, N., Fernandez, J., Yazdani, S., Yin, Z., Wang, Z., Wang, X., Xu, W., Liu, J., et al.: Optimization and improvement of fake news detection using voting technique for societal benefit. In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1565–1574 (2023). IEEE
  * Wang et al. [2023] Wang, Z., Narasimhan, G., Yao, X., Zhang, W.: Mitigating multisource biases in graph neural networks via real counterfactual samples. In: 2023 IEEE International Conference on Data Mining (ICDM), pp. 638–647 (2023). IEEE
  * Yazdani et al. [2024] Yazdani, S., Saxena, N., Wang, Z., Wu, Y., Zhang, W.: A comprehensive survey of image and video generative ai: Recent advances, variants, and applications (2024)
  * Wang et al. [2023] Wang, Z., Wallace, C., Bifet, A., Yao, X., Zhang, W.: Fg2an: Fairness-aware graph generative adversarial networks. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 259–275 (2023). Springer Nature Switzerland
  * Zhang et al. [2023] Zhang, W., Wang, Z., Kim, J., Cheng, C., Oommen, T., Ravikumar, P., Weiss, J.: Individual fairness under uncertainty. In: 26th European Conference on Artificial Intelligence, pp. 3042–3049 (2023)
  * Wang et al. [2023] Wang, Z., Saxena, N., Yu, T., Karki, S., Zetty, T., Haque, I., Zhou, S., Kc, D., Stockwell, I., Bifet, A., et al.: Preventing discriminatory decision-making in evolving data streams. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT) (2023)
  * Wang et al. [2024a] Wang, Z., Ulloa, D., Yu, T., Rangaswami, R., Yap, R., Zhang, W.: Individual fairness with group constraints in graph neural networks. In: ECAI 2024, (2024)
  * Wang et al. [2024b] Wang, Z., Chu, Z., Blanco, R., Chen, Z., Chen, S.-C., Zhang, W.: Advancing graph counterfactual fairness through fair representation learning. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2024). Springer Nature Switzerland
  * Wang et al. [2024c] Wang, Z., Qiu, M., Chen, M., Salem, M.B., Yao, X., Zhang, W.: Toward fair graph neural networks via real counterfactual samples. Knowledge and Information Systems, 1–25 (2024)
  * Wang et al. [2024d] Wang, Z., Dzuong, J., Yuan, X., Chen, Z., Wu, Y., Yao, X., Zhang, W.: Individual fairness with group awareness under uncertainty. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2024). Springer Nature Switzerland
  * Doan et al. [2024] Doan, T.V., Wang, Z., Nguyen, M.N., Zhang, W.: Fairness in large language models in three hours. In: Proceedings of the 33rd ACM International Conference on Information & Knowledge Management (2024)
  * Wang and Zhang [2024] Wang, Z., Zhang, W.: Group fairness with individual and censorship constraints. In: Proceedings of the European Conference on Artificial Intelligence (2024)
  * Chinta et al. [2024a] Chinta, S.V., Wang, Z., Yin, Z., Hoang, N., Gonzalez, M., Quy, T.L., Zhang, W.: Fairaied: Navigating fairness, bias, and ethics in educational ai applications. (2024)
  * Chinta et al. [2024b] Chinta, S.V., Wang, Z., Zhang, X., Doan, T., Kashif, A., Smith, M.A., Zhang, W.: Ai-driven healthcare: A survey on ensuring fairness and mitigating bias (2024)
  * Chu et al. [2024] Chu, Z., Wang, Z., Zhang, W.: Fairness in large language models: A taxonomic survey. ACM SIGKDD Explorations Newsletter (2024)
  * Doan et al. [2024] Doan, T.V., Chu, Z., Wang, Z., Zhang, W.: Fairness definitions in language models explained (2024)
  * Chinta et al. [2024] Chinta, S.V., Wang, Z., Zhang, X., Doan Viet, T., Kashif, A., Smith, M.A., Zhang, W.: Ai-driven healthcare: A survey on ensuring fairness and mitigating bias. Journal of Healthcare Informatics (2024)
  * Saxena et al. [2024] Saxena, N.A., Zhang, W., Shahabi, C.: Unveiling and mitigating bias in ride-hailing pricing for equitable policy making. AI and Ethics, 1–12 (2024)
  * Zhang and Weiss [2023] Zhang, W., Weiss, J.C.: Fairness with censorship and group constraints. Knowledge and Information Systems, 1–24 (2023)
  * Quy et al. [2022] Quy, T.L., Roy, A., Iosifidis, V., Zhang, W., Ntoutsi, E.: A survey on datasets for fairness-aware machine learning. Data Mining and Knowledge Discovery (2022)
  * Zhang et al. [2022] Zhang, W., Pan, S., Zhou, S., Walsh, T., Weiss, J.C.: Fairness amidst non-iid graph data: Current achievements and future directions. arXiv preprint arXiv:2202.07170 (2022)
  * Zhang et al. [2023] Zhang, W., Hernandez-Boussard, T., Weiss, J.: Censored fairness through awareness. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 14611–14619 (2023)
  * Zhang and Weiss [2022] Zhang, W., Weiss, J.C.: Longitudinal fairness with censorship. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 12235–12243 (2022)
  * Zhang and Weiss [2021] Zhang, W., Weiss, J.: Fair decision-making under uncertainty. In: 2021 IEEE International Conference on Data Mining (ICDM) (2021). IEEE
  * Zhang et al. [2021] Zhang, W., Bifet, A., Zhang, X., Weiss, J.C., Nejdl, W.: Farf: A fair and adaptive random forests classifier. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 245–256 (2021). Springer
  * Zhang [2020] Zhang, W.: Learning fairness and graph deep generation in dynamic environments (2020)
  * Zhang and Zhao [2020] Zhang, W., Zhao, L.: Online decision trees with fairness. arXiv preprint arXiv:2010.08146 (2020)
  * Zhang et al. [2020] Zhang, W., et al.: Flexible and adaptive fairness-aware learning in non-stationary data streams. In: IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 399–406 (2020)
  * Zhang and Bifet [2020] Zhang, W., Bifet, A.: Feat: A fairness-enhancing and concept-adapting decision tree classifier. In: International Conference on Discovery Science, pp. 175–189 (2020). Springer
  * Zhang et al. [2019] Zhang, W., Tang, X., Wang, J.: On fairness-aware learning for non-discriminative decision-making. In: International Conference on Data Mining Workshops (ICDMW), pp. 1072–1079 (2019)
  * Zhang and Ntoutsi [2019] Zhang, W., Ntoutsi, E.: Faht: an adaptive fairness-aware decision tree classifier. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 1480–1486 (2019)
  * Saxena et al. [2023] Saxena, N.A., Zhang, W., Shahabi, C.: Missed opportunities in fair ai. In: Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pp. 961–964 (2023). SIAM
  * Zhang [2024a] Zhang, W.: Fairness with censorship: Bridging the gap between fairness research and real-world deployment. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 22685–22685 (2024)
  * Zhang [2024b] Zhang, W.: Ai fairness in practice: Paradigm, challenges, and prospects. Ai Magazine (2024)
  * Zhang et al. [2016] Zhang, W., Tang, J., Wang, N.: Using the machine learning approach to predict patient survival from high-dimensional survival data. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2016)
  * Zhang and Wang [2018] Zhang, W., Wang, J.: Content-bootstrapped collaborative filtering for medical article recommendations. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2018)
  * Tang et al. [2020] Tang, X., Zhang, L., et al.: Using machine learning to automate mammogram images analysis. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 757–764 (2020)
  * Zhang et al. [2014] Zhang, L., et al.: A comparison of different pattern recognition methods with entropy based feature reduction in early breast cancer classification. European Scientific Journal 3, 303–312 (2014)
  * Zhang et al. [2020] Zhang, M., Zhao, X., et al.: Deep discriminative learning for autism spectrum disorder classification. In: International Conference on Database and Expert Systems Applications, pp. 435–443 (2020). Springer
  * Wang et al. [2021] Wang, X., Zhang, W., Jadhav, A., Weiss, J.: Harmonic-mean cox models: A ruler for equal attention to risk. In: Survival Prediction-Algorithms, Challenges and Applications, pp. 171–183 (2021). PMLR
  * Liu et al. [2022] Liu, Z., Wang, R., Zhang, W.: Improving the generalization of unsupervised feature learning by using data from different sources on gene expression data for cancer diagnosis. Medical & Biological Engineering & Computing, 1–19 (2022)
  * Wang et al. [2016] Wang, J., Huang, Z., et al.: Wearable sensor based human posture recognition. In: IEEE International Conference on Big Data (Big Data) (2016)
  * Zhang and Wang [2017] Zhang, W., Wang, J.: A hybrid learning framework for imbalanced stream classification. In: IEEE International Congress on Big Data (BigData Congress), pp. 480–487 (2017)
  * Zhang [2017] Zhang, W.: Phd forum: Recognizing human posture from time-changing wearable sensor data streams. In: IEEE International Conference on Smart Computing (SMARTCOMP) (2017)
  * Zhang et al. [2021] Zhang, M., Zhang, F., Zhang, J., Chaddad, A., Guo, F., Zhang, W., Zhang, J., Evans, A.: Autoencoder for neuroimage. In: International Conference on Database and Expert Systems Applications, pp. 84–90 (2021). Springer
  * Liu et al. [2021] Liu, Z., Wang, R., Zhang, W., Tang, D.: An unsupervised feature learning method for enhancing the generalization of cancer diagnosis. In: 2021 13th International Conference on Machine Learning and Computing, pp. 252–257 (2021)
  * Zhang et al. [2018] Zhang, W., Wang, J., Jin, D., Oreopoulos, L., Zhang, Z.: A deterministic self-organizing map approach and its application on satellite data based cloud type classification. In: IEEE International Conference on Big Data (Big Data) (2018)
  * Tang et al. [2019] Tang, X., Qiu, J., et al.: The internet of responsibilities-connecting human responsibilities using big data and blockchain. In: IEEE International Conference on Big Data (Big Data) (2019)
  * Tang et al. [2020] Tang, X., et al.: A data-driven human responsibility management system. In: IEEE International Conference on Big Data (Big Data) (2020)
  * Tang et al. [2021a] Tang, X., Huang, X., et al.: Cognitive visual commonsense reasoning using dynamic working memory. In: International Conference on Big Data Analytics and Knowledge Discovery (2021). Springer
  * Tang et al. [2021b] Tang, X., Zhang, W., Yu, Y., Turner, K., Derr, T., Wang, M., Ntoutsi, E.: Interpretable visual understanding with cognitive attention network. In: International Conference on Artificial Neural Networks, pp. 555–568 (2021). Springer
  * Liu et al. [2021] Liu, Z., Wang, R., Japkowicz, N., Tang, D., Zhang, W., Zhao, J.: Research on unsupervised feature learning for android malware detection based on restricted boltzmann machines. Future Generation Computer Systems 120, 91–108 (2021)
  * Liu et al. [2023] Liu, Z., Wang, R., Japkowicz, N., Gomes, H.M., Peng, B., Zhang, W.: Segdroid: An android malware detection method based on sensitive function call graph learning. Expert Systems with Applications, 121125 (2023)
  * Guyet et al. [2022] Guyet, T., Zhang, W., Bifet, A.: Incremental mining of frequent serial episodes considering multiple occurrences. In: 22nd International Conference on Computational Science, pp. 460–472 (2022). Springer
  * Cai et al. [2023] Cai, Y., Youngstrom, D., Zhang, W.: Exploring approaches for teaching cybersecurity and ai for k-12. In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1559–1564 (2023). IEEE
  * Zawacki-Richter et al. [2019] Zawacki-Richter, O., Marín, V.I., Bond, M., Gouverneur, F.: Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education 16(1), 39 (2019) https://doi.org/10.1186/s41239-019-0171-0
  * Nalawade et al. [2023] Nalawade, P.S.A., Shukla, R.R., Gurjar, P., Kumar, H., Chavan, H.: Smart essay grading. International Journal for Research in Applied Science and Engineering Technology (2023)
  * Gupta et al. [2024] Gupta, S., Dharamshi, R.R., Kakde, V.U.: An impactful and revolutionized educational ecosystem using generative ai to assist and assess the teaching and learning benefits, fostering the post-pandemic requirements. 2024 Second International Conference on Emerging Trends in Information Technology and Engineering (ICETITE), 1–4 (2024)
  * González-Calatayud et al. [2021] González-Calatayud, V., Prendes-Espinosa, P., Roig-Vila, R.: Artificial intelligence for student assessment: A systematic review. Applied Sciences (2021)
  * Ahmed et al. [2024] Ahmed, R., Mallah, I.A., Shaheen, D.A.: Learning experiences and practices through artificial intelligence as adoptive academia for teachers and students of higher education institutions of pakistan. PAKISTAN LANGUAGES AND HUMANITIES REVIEW (2024)
  * Kumar [2021] Kumar, R.: Biases in artificial intelligence applications affecting human life: A review. International Journal of Recent Technology and Engineering (2021)
  * Marcinkowski et al. [2020] Marcinkowski, F., Kieslich, K., Starke, C., Lünich, M.: Implications of ai (un-)fairness in higher education admissions: the effects of perceived ai (un-)fairness on exit, voice and organizational reputation. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020)
  * Busum and Fang [2023] Busum, K.V., Fang, S.: Analysis of ai models for student admissions: A case study. Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing (2023)
  * Harry [2023] Harry, A.: Role of ai in education. Interdiciplinary Journal and Hummanity (INJURITY) (2023)
  * Alvero et al. [2019] Alvero, A., Arthurs, N., Antonio, A.L., Domingue, B.W., Gebre-Medhin, B., Giebel, S., Stevens, M.L.: Ai and holistic review: Informing human reading in college admissions. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (2019)
  * Algarni and Sheldon [2023] Algarni, S., Sheldon, F.T.: Systematic review of recommendation systems for course selection. Mach. Learn. Knowl. Extr. 5, 560–596 (2023)
  * Fan et al. [2023] Fan, O., Wu, M., Zheng, L., Zhang, L., Jiao, P.: Integration of artificial intelligence performance prediction and learning analytics to improve student learning in online engineering course. International Journal of Educational Technology in Higher Education 20 (2023)
  * Zafar et al. [2016] Zafar, M.B., Valera, I., Gomez-Rodriguez, M., Gummadi, K.P.: Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. Proceedings of the 26th International Conference on World Wide Web (2016)
  * Lu et al. [2015] Lu, J., Wu, D., Mao, M., Wang, W., Zhang, G.: Recommender system application developments: A survey. Decis. Support Syst. 74, 12–32 (2015)
  * Urdaneta-Ponte et al. [2021] Urdaneta-Ponte, M.C., Méndez-Zorrilla, A., Oleagordia-Ruíz, I.: Recommendation systems for education: Systematic review. Electronics (2021)
  * Holstein et al. [2018] Holstein, K., Vaughan, J.W., Daumé, H., Dudík, M., Wallach, H.M.: Improving fairness in machine learning systems: What do industry practitioners need? Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2018)
  * Veale and Binns [2017] Veale, M., Binns, R.: Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society 4 (2017)
  * Taneja and Tripathi [2020] Taneja, A.K., Tripathi, C.: Ai-powered recommender systems: Personalization and bias. Turkish Journal of Computer and Mathematics Education (TURCOMAT) (2020)
  * Education [2023] Education, B.M.: Artificial intelligence in medical education: a cross-sectional needs assessment. BMC Medical Education (2023)
  * Vokinger et al. [2021] Vokinger, K.N., Feuerriegel, S., Kesselheim, A.S.: Mitigating bias in machine learning for medicine. Communications Medicine 1 (2021)
  * Townson [2023] Townson, S.: Manage ai bias instead of trying to eliminate it. MIT Sloan Management Review (2023)
  * Yang et al. [2023] Yang, J., Soltan, A.A.S., Eyre, D.W., Yang, Y., Clifton, D.A.: An adversarial training framework for mitigating algorithmic biases in clinical machine learning. NPJ Digital Medicine 6 (2023)
  * Baker and Yacef [2009] Baker, R.S., Yacef, K.: The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining 1(1), 3–17 (2009)
  * O’Neil [2016] O’Neil, C.: Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, New York, NY (2016)
  * Buolamwini and Gebru [2018] Buolamwini, J., Gebru, T.: Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency (2018)
  * Bowers [2017] Bowers, A.J.: Quantitative research methods training in education leadership and administration preparation programs as disciplined inquiry for building school improvement capacity. Journal of Research on Leadership Education 12(1), 72–96 (2017)
  * Gašević et al. [2015] Gašević, D., Dawson, S., Siemens, G.: Let’s not forget: Learning analytics are about learning. TechTrends 59(1), 64–71 (2015)
  * Shermis and Burstein [2003] Shermis, M.D., Burstein, J.C.: Automated Essay Scoring: A Cross-disciplinary Perspective. Routledge, New York, NY (2003)
  * Mayfield and Rosé [2013] Mayfield, E., Rosé, C.P.: Lightside: Open source machine learning for text accessible to non-experts. Instructional Science 41(4), 709–732 (2013)
  * Shahbazi et al. [2022] Shahbazi, N., Lin, Y., Asudeh, A., Jagadish, H.V.: Representation bias in data: A survey on identification and resolution techniques. ACM Computing Surveys 55, 1–39 (2022)
  * Holstein et al. [2019] Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudik, M., Wallach, H.: Improving fairness in machine learning systems: What do industry practitioners need? In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. CHI ’19 (2019)
  * Fu et al. [2020] Fu, R., Huang, Y., Singh, P.V.: Ai and algorithmic bias: Source, detection, mitigation and implications. InfoSciRN: Machine Learning (Sub-Topic) (2020)
  * Abd-Alrazaq et al. [2023] Abd-Alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P.M., Latifi, S., Aziz, S., Damseh, R., Alrazak, S.A., Sheikh, J., et al.: Large language models in medical education: Opportunities, challenges, and future directions. JMIR Medical Education 9(1), 48291 (2023)
  * Patton et al. [2020] Patton, D.U., Frey, W.R., McGregor, K.A., Lee, J.R., McKeown, K.R., Moss, E.: Contextual analysis of social media: The promise and challenge of eliciting context in social media posts with natural language processing. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 337–342 (2020)
  * Bagdasaryan et al. [2019] Bagdasaryan, E., Poursaeed, O., Shmatikov, V.: Differential privacy has disparate impact on model accuracy. In: Advances in Neural Information Processing Systems (2019)
  * Celi et al. [2022] Celi, L.A., Cellini, J., Charpignon, M., Dee, E.C., Dernoncourt, F., Eber, R., Mitchell, W.G., Moukheiber, L., Schirmer, J., Situ, J., Paguio, J.A., Park, J., Wawira, J., Yao, S.: Sources of bias in artificial intelligence that perpetuate healthcare disparities—a global review. PLOS Digital Health 1 (2022)
  * Rader et al. [2018] Rader, E., Cotter, K., Cho, J.S.: Explanations as mechanisms for supporting algorithmic transparency. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18 (2018)
  * Eubanks [2018] Eubanks, V.: Automating Inequality: How High-tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press, New York, NY, USA (2018)
  * Selbst et al. [2019] Selbst, A.D., Boyarskaya, M., Barry-Jester, A.-M., Barocas, S., Hardt, M.: Fairness and abstraction in sociotechnical systems. In: ACM Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019)
  * Benjamin [2019] Benjamin, R.: Race After Technology: Abolitionist Tools for the New Jim Code. Polity, Cambridge, UK (2019)
  * Mannion and Thompson [2014] Mannion, R., Thompson, C.: Systematic biases in group decision-making: implications for patient safety. International journal for quality in health care : journal of the International Society for Quality in Health Care 26 6, 606–12 (2014)
  * Farnadi et al. [2018] Farnadi, G., Tang, J., De Cock, M., Moens, M.-F.: User profiling through deep multimodal fusion. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (2018)
  * Kay et al. [2015] Kay, M., Matuszek, C., Munson, S.A.: Unequal representation and gender stereotypes in image search results for occupations. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (2015)
  * Zhao et al. [2017] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., Chang, K.-W.: Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (2017)
  * Bolukbasi et al. [2016] Bolukbasi, T., Chang, K.-W., Zou, J.Y., Saligrama, V., Kalai, A.T.: Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
  * Noble [2018] Noble, S.U.: Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press, New York (2018)
  * Wachter et al. [2017] Wachter, S., Mittelstadt, B., Floridi, L.: Why a right to explanation of automated decision-making does not exist in the general data protection regulation. International Data Privacy Law (2017)
  * Lazar et al. [2017] Lazar, J., Goldstein, D.F., Taylor, A.: Ensuring Digital Accessibility Through Process and Policy. Morgan Kaufmann, Cambridge, MA, USA (2017)
  * Holstein et al. [2019] Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudik, M., Wallach, H.: Improving fairness in machine learning systems: What do industry practitioners need? In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–16 (2019). ACM
  * Zheng et al. [2018] Zheng, Z., Booth, R., Sarkar, N.: Inclusive design in educational technology: Development and evaluation of an inclusive digital learning environment. Journal of Educational Technology & Society 21(3), 25–41 (2018)
  * UNESCO [2012] UNESCO: Addressing Exclusion in Education: A Guide to Assessing Education Systems Towards More Inclusive and Just Societies. UNESCO, ??? (2012). Accessed: 2024-05-19. https://unesdoc.unesco.org/ark:/48223/pf0000217073
  * Fleisher [2020] Fleisher, W.: What’s fair about individual fairness? PhilSci-Archive (2020)
  * Heidari et al. [2019] Heidari, H., Loi, M., Gummadi, K.P., Krause, A.: A moral framework for understanding fair ml through economic models of equality of opportunity. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 181–190 (2019)
  * Dwork et al. [2012] Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (2012)
  * Gillen et al. [2018] Gillen, S., Kearns, M., Roth, A., Wu, Z.S.: Fairness and efficiency in ranking from pairwise comparisons. In: Proceedings of the 2018 ACM Conference on Economics and Computation, pp. 329–346 (2018)
  * Hu and Rangwala [2020a] Hu, X., Rangwala, H.: Towards individually fair grade prediction models. In: Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020) (2020)
  * Hu and Rangwala [2020b] Hu, Q., Rangwala, H.: Towards Fair Educational Data Mining: A Case Study on Detecting At-Risk Students. https://eric.ed.gov/?id=ED610524 (2020)
  * Järvelin and Kekäläinen [2002] Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS) 20(4), 422–446 (2002)
  * Chapelle et al. [2009] Chapelle, O., Metlzer, D., Zhang, Y., Grinspan, P.: Expected reciprocal rank for graded relevance. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management, pp. 621–630 (2009)
  * Le Quy et al. [2022] Le Quy, T., Nguyen, T.H., Friege, G., Ntoutsi, E.: Evaluation of group fairness measures in student performance prediction problems. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 119–136 (2022). https://doi.org/10.1007/978-3-031-23618-1_8
  * Hardt et al. [2016] Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems (2016)
  * Chouldechova [2017] Chouldechova, A.: Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5(2), 153–163 (2017) https://doi.org/10.1089/big.2016.0047
  * Corbett-Davies et al. [2017] Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., Huq, A.: Algorithmic decision making and the cost of fairness. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2017)
  * Ntoutsi et al. [2020] Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M.-E., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al.: Bias in data-driven artificial intelligence systems—an introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10(3), 1356 (2020)
  * Calmon et al. [2017] Calmon, F., Wei, D., Vinzamuri, B., Ramamurthy, K.N., Varshney, K.R.: Optimized pre-processing for discrimination prevention. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
  * Kamiran and Calders [2012] Kamiran, F., Calders, T.: Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33(1), 1–33 (2012)
  * Wang and Zhu [2020] Wang, Y., Zhu, M.: Application of smote in student performance prediction model based on imbalanced data. IEEE Access 8, 21442–21449 (2020)
  * Chawla et al. [2002] Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16, 321–357 (2002)
  * Goodfellow et al. [2014] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NIPS 2014) (2014)
  * Fernández et al. [2018] Fernández, A., García, S., Herrera, F., Chawla, N.V.: Smote for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary. Journal of Artificial Intelligence Research 61, 863–905 (2018)
  * Liu et al. [2009] Liu, Y., Wei, Y., Zhang, Y., Yang, M.: The role of undersampling in the development of predictive models for educational data. Journal of Educational Data Mining (2009). Hypothetical reference; adjust to match actual publication details
  * Zhang et al. [2003] Zhang, J., Mani, I., Lin, K.: Nearmiss under sampling for imbalanced dataset classification. In: ICML Workshop on Learning from Imbalanced Datasets II (2003). Hypothetical reference; adjust to match actual publication details
  * Haixiang et al. [2017] Haixiang, G., Yijing, L., Jennifer, S., Mingyun, G., Yuanyue, H., Bing, G.: Learning from class-imbalanced data: Review of methods and applications. Expert Systems with Applications 73, 220–239 (2017)
  * Chawla et al. [2011] Chawla, N.V., Japkowicz, N., Kotcz, A.: Editorial: Special issue on learning from imbalanced data sets. SIGKDD Explorations Newsletter 6(1), 1–6 (2011)
  * Ramentol et al. [2012] Ramentol, E., Caballero, Y., Bello, R., Herrera, F.: Smote-rsb*: a hybrid preprocessing approach based on oversampling and undersampling for high imbalanced data-sets using smote and rough sets theory. Knowledge and Information Systems 33(2), 245–265 (2012)
  * Kotsiantis et al. [2006] Kotsiantis, S.B., Kanellopoulos, D., Pintelas, P.E.: Handling imbalanced datasets: A review. GESTS International Transactions on Computer Science and Engineering 30(1), 25–36 (2006)
  * Kamishima et al. [2011] Kamishima, T., Akaho, S., Asoh, H., Sakuma, J.: Fairness-aware classifier with prejudice remover regularizer. In: Machine Learning and Knowledge Discovery in Databases, pp. 35–50 (2011). Springer
  * Mary et al. [2019] Mary, J., Vasile, F., Gummadi, K.: A multi-faceted approach to fairness in ai. In: Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (2019). ACM
  * Zafar et al. [2017] Zafar, M.B., Valera, I., Rodriguez, M.G., Gummadi, K.P.: Fairness constraints: Mechanisms for fair classification. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) (2017)
  * Donini et al. [2018] Donini, M., Oneto, L., Ben-David, S., Shawe-Taylor, J., Pontil, M.: Empirical risk minimization under fairness constraints. In: Advances in Neural Information Processing Systems, pp. 2791–2801 (2018)
  * Grgić-Hlača et al. [2018] Grgić-Hlača, N., Zafar, M.B., Gummadi, K.P., Weller, A.: Beyond distributive fairness in algorithmic decision making: Feature selection for procedurally fair learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2018)
  * Saleiro et al. [2019] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Augustin, M., Ghani, R.: Aequitas: A bias and fairness audit toolkit. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2019)
  * Smith and Neff [2019] Smith, L., Neff, M.: Mitigating gender bias in student evaluations of teaching. PLOS ONE 14(5), 0216241 (2019)
  * Stark and Hoffmann [2019] Stark, L., Hoffmann, A.L.: Data is the new what? popular metaphors & professional ethics in emerging data culture. Journal of Cultural Analytics (2019)
  * Friedler et al. [2016] Friedler, S.A., Scheidegger, C., Venkatasubramanian, S.: The (im)possibility of fairness: Different value systems require different mechanisms for fair decision making. In: Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) (2016)
  * Hardt et al. [2016] Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems (2016)
  * Johndrow and Lum [2017] Johndrow, J.E., Lum, K.: An algorithm for removing sensitive information: Application to race-independent recidivism prediction. The Annals of Applied Statistics (2017)
  * Feldman et al. [2015] Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., Venkatasubramanian, S.: Certifying and removing disparate impact. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259–268 (2015). https://doi.org/DOIPlaceholder
  * Taylor et al. [2017] Taylor, L., Floridi, L., Sloot, B. (eds.): Group Privacy: New Challenges of Data Technologies. Springer, Cham, Switzerland (2017)
  * Susskind and Susskind [2015] Susskind, R., Susskind, D.: The Future of the Professions: How Technology Will Transform the Work of Human Experts. Oxford University Press, Oxford, United Kingdom (2015)
  * Dignum [2019] Dignum, V.: Responsible Artificial Intelligence: How to Develop and Use AI in a Responsible Way. Springer, Berlin, Heidelberg (2019)
  * Bellamy et al. [2018] Bellamy, R.K.E., Dey, K., Hind, M., Hoffman, S.C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al.: Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 (2018)
  * Bird et al. [2020] Bird, S., Dudík, M., Goh, G., Hall, K., Horn, W., Lutz, R., Morgenstern, J., Nushi, B., Wallach, H.: Fairlearn: A toolkit for assessing and improving fairness in AI. Microsoft Research (2020)