11institutetext: Université de technologie de Compiègne, CNRS, Heudiasyc
(Heuristics and Diagnosis of Complex Systems), CS 60319 - 60203 Compiègne
Cedex, France 22institutetext: Vivocaz, 8 B Rue de la Gare, 02200, Mercin-et-
Vaux, France

# Designing a User Contextual Profile Ontology: A Focus on the Vehicle Sales
Domain

LE Ngoc Luyen 1122 Marie-Hélène ABEL 11 Philippe GOUSPILLOU 22

###### Abstract

In the digital age, it is crucial to understand and tailor experiences for
users interacting with systems and applications. This requires the creation of
user contextual profiles that combine user profiles with contextual
information. However, there is a lack of research on the integration of
contextual information with different user profiles. This study aims to
address this gap by designing a user contextual profile ontology that
considers both user profiles and contextual information on each profile.
Specifically, we present a design and development of the user contextual
profile ontology with a focus on the vehicle sales domain. Our designed
ontology serves as a structural foundation for standardizing the
representation of user profiles and contextual information, enhancing the
system’s ability to capture user preferences and contextual information of the
user accurately. Moreover, we illustrate a case study using the User
Contextual Profile Ontology in generating personalized recommendations for
vehicle sales domain.

###### Keywords:

Ontology User Modeling Knowledge base Contextual Profile.

##  1 Introduction

Understanding and customizing experiences for users interacting with various
systems and applications is essential in today’s digital landscape. To
accomplish this, individual user profiles are integrated with their
corresponding contextual information, resulting in the creation of a user
contextual profile [14]. A user profile is a compilation of personal
information and preferences about a particular user, including demographic
data, interests, past behavior, and other attributes that define the user’s
identity within a system or application [16]. For example, the user Henri
wants to buy a vehicle for professional use and another for his family with
two children. Therefore, the user Henri can have two distinct profiles, each
catering to different needs and preferences. While contextual information for
each profile refers to the situational factors surrounding a user’s
interaction with a system or application [30]. This may include location,
time, device being used, and even the user’s current activity or emotional
state. In the example, each of the two profiles of the user Henri has unique
contextual information related to Henri’s preferences, needs, and interactions
within the application. Consequently, a user contextual profile combines the
user profile with the contextual information specific to that profile,
creating a more comprehensive understanding of the user’s needs and
preferences in various situations. For instance, when searching for a
professional vehicle, the user contextual profile may prioritize factors such
as fuel efficiency, compact size, and reliability, while for a family vehicle,
it may focus on safety features, ample seating, and cargo space.

In order to effectively model user contextual profiles, utilizing ontology-
based approaches can be advantageous. Ontologies are formal, structured
representations of knowledge within a specific domain that enable the
organization and sharing of data in a machine-readable format. Once the
ontology is designed and implemented, it can be integrated into applications
and systems, allowing them to reason over the user contextual profiles and
infer new knowledge based on the available data [32, 29, 12]. This can improve
the accuracy and relevance of recommendations, as well as facilitate more
efficient data integration and interoperability among different applications
and services.

A considerable number of studies in the field of user modeling have primarily
focused on either user profiles or user contexts [28, 25]. The aspects of
contextual information related to each profile of the user have not been
thoroughly examined. This underexplored area offers an opportunity for further
research, focusing on the integration of contextual information with different
profiles of the user to develop a deeper understanding of users’ needs and
preferences in various situations. Consequently, our work aims to address this
gap by designing a User Contextual Profile Ontology (UCPO) with a focus on the
vehicle sales domain. This ontology considers both user profiles and
contextual information on each profile, utilizing a standard ontology
development methodology. Furthermore, we provide an illustration of the use of
UCPO through a case study on generating personalized recommendations in the
vehicle sales domain.

The remainder of this article is organized as follows: Section 2 introduces
works from the literature on which our approach is based. Section 3 presents
our main contributions, outlining the methodology for designing a user
contextual profile ontology with a focus on the vehicle sales domain. In
section 4, we provide a case study on personalized recommendations based on
our developed ontology. Finally, we conclude and discuss the perspectives.

##  2 Related work

In this section, we investigate the modeling and integration of user profiles
and contextual information for specific profiles, as well as examine various
approaches to ontology design. The related works’ analysis will inform and
support the development of a user contextual profile ontology.

###  2.1 Ontological user modeling

Ontologies have proven to be highly effective in modeling user profiles and
contexts. They provide a thorough depiction of a particular domain of
interest, simplifying browsing and query refinement. Specifically, ontologies
have been observed to outperform other methods in user modeling when compared
to other methods utilized [26]. Ontological user modeling refers to the use of
ontologies for representing and managing user profiles, preferences, and
contextual information in a structured and machine-readable format. User-
related information such as profiles, preferences, and context are represented
using concepts, relationships, and attributes within a formal, hierarchical
structure. This structured representation enables more effective processing,
reasoning, and inferencing by computer systems.

Ontological user modeling has previously been investigated across numerous
research areas, demonstrating its adaptability and potential to enhance user
experiences in diverse domains such as e-learning [6], e-commerce [4],
recommender systems [8, 9, 10, 11], personalized web services [24], and
context-aware applications [22], among others [17, 5, 13]. Specifically,
Golemati et al. [16] designed an ontology that incorporates concepts and
properties essential for modeling user profiles. Their model predominantly
focuses on static user characteristics and provides a foundation for the
development of a more general, comprehensive, and extensible user model.
Adewoyin et al. [1] proposed a versatile user modeling architecture for smart
environments. They represented the characteristics of users using the behavior
concept, which encompasses all relevant aspects that can aid in effective
behavioral modeling and monitoring. Sutterer et al. [28] put forth a user
profile ontology designed specifically to depict situation-dependent sub-
profiles. This ontology can be utilized by context-aware adaptive service
platforms for mobile communication and information services, facilitating
automatic and situation-dependent personalization of such services. Skillen et
al. [25] developed a User Profile Ontology for personalizing context-aware
applications in mobile environments. Their focus lies on user behavior and
characterizing the needs of users for context-aware applications. Contrasting
with the work in [16], their ontological user modeling approach emphasizes
dynamic components for application usage.

The works mentioned highlight numerous advantages of ontological user
modeling, which is capable of capturing both static and dynamic user
characteristics related to permanent, temporary information, and user
evolution. Moreover, user contexts play a crucial role in user modeling,
contributing significantly to the creation of personalized and adaptive
systems. Most of these studies integrate user contexts and user profiles.
Nonetheless, the contextual information can be dependent on the profile and
its primary objectives. Therefore, it is necessary to distinguish contextual
information for the user from that for the profile clearly. The development of
a user contextual profile ontology is essential for clearly organizing and
distinguishing the contextual information related to users and their
respective profiles. In the following section, we will examine ontology
development methodologies to achieve this objective.

###  2.2 Ontology development methodology

Ontology development methodologies provide a structured and systematic
approach to the creation, maintenance, and evolution of ontologies. These
methodologies are specifically designed to ensure the quality, coherence, and
practicality of the resulting ontology, while also promoting its consistency
and usability. By employing ontology development methodologies, researchers
and practitioners can effectively enhance the overall performance and
applicability of the ontologies they create, making them more reliable and
valuable for their intended use cases.

Various ontology development methodologies have been proposed in the
literature, each featuring its own unique set of guidelines and principles.
Some widely adopted methodologies include: Methontology [3], Ontology
Development 101 (OD101) [19], NeON Methodology [27], Unified Process for
Ontology Building (UPON) [2], Simplified Agile Methodology for Ontology
Development (SAMOD) [20], Modular Ontology Modeling (MoMo) [23], and others.
Firstly, Methontology is one of the earliest and most widely used
methodologies. It covers the entire ontology development process, including
specification, conceptualization, formalization, integration, implementation,
and maintenance, while emphasizing the importance of documentation throughout
the development process. OD101 then is a methodology designed to guide
ontology engineering processes. Focusing on iterative development, OD101
provides step-by-step guidelines and best practices for constructing
ontologies, ensuring a systematic and efficient approach to ontology creation.
The NeON Methodology next is designed to create ontology networks, rather than
singular ontologies. This approach emphasizes the reuse and re-engineering of
existing ontologies, fostering collaboration between ontology engineers and
domain experts to facilitate a more comprehensive and efficient development
process. UPON is a methodology that incorporates concepts from the Unified
Modeling Language (UML) and the Rational Unified Process (RUP). This approach
adheres to an iterative and incremental process, encompassing four development
phases: inception, elaboration, construction, and transition. It provides a
structured framework for creating and refining ontologies, ensuring consistent
and systematic ontology development. SAMOD is a methodology that integrates
agile software development principles into ontology engineering. It provides a
flexible, iterative, and adaptable framework, fostering collaboration among
ontology engineers, domain experts, and end-users. Employing iterative
development cycles, SAMOD encompasses planning, design, implementation, and
evaluation stages, facilitating ongoing feedback and refinements throughout
the development process. Finally, MoMo is a methodology that creates and
manages modular ontologies, allowing for independent development, maintenance,
and reuse of smaller, more manageable modules. It facilitates collaboration
and focuses on specific parts of the ontology. MoMo involves module
identification, design, integration, and evaluation, resulting in a more
efficient and maintainable ontology development process. In general, the use
of an ontology development methodology can lead to more efficient ontology
development, better ontology reuse, easier ontology maintenance, and more
effective collaboration between ontology engineers and domain experts.
Furthermore, the adoption of a standardized ontology development methodology
can improve the interoperability and compatibility of ontologies, enabling
more effective data integration and knowledge sharing among different systems
and applications.

The selection of an appropriate ontology development methodology depends on
several factors, such as the complexity of the ontology, the domain of
application, the availability of resources, and the expertise of the ontology
engineers. In this work, we have opted for SAMOD to build a user contextual
profile ontology due to its flexibility, adaptability, and emphasis on
collaboration among ontology engineers, domain experts, and end-users. The
iterative and incremental process of SAMOD, combined with its ongoing feedback
and refinements, aligns with our goal of creating a comprehensive and
adaptable ontology that can evolve over time. Hence, we will describe the
development process of our user contextual profile ontology in the next
section.

##  3 Methodology for developing a user contextual profile ontology

The development of a User Context Profile Ontology is a complex process that
requires a structured and systematic approach. SAMOD methodology is based on
agile software development principles, enabling the efficient development of a
comprehensive and adaptable UCPO. Therefore, the UCPO can be made dynamic
through continuous updates and modifications to the ontology based on changes
in user profiles and contextual information. This can be achieved by
incorporating an agile approach to ontology development, where the ontology is
built and adapted incrementally, with regular feedback and updates from users.
In this section, we describe the phases of SAMOD, including kickoff, design
and implementation, and test and evaluation, which serve as a framework for
designing the UCPO.

###  3.1 Kickoff phase

The main objective of the UCPO is to establish a standardized representation
of user information and contextual information for each user profile. This
ontological user modeling can facilitate the development of personalized
systems and services, including personalized information retrieval, adaptive
user interfaces, personalized recommendations, and other applications.
Defining a standardized format for user information, context, and profile can
improve the interoperability of the UCPO and enable it to adapt to the
changing needs and preferences of the users. The scope of the UCPO is broad,
covering a wide range of user characteristics and environmental factors that
can influence user behavior and preferences. This includes, but is not limited
to, demographics, psychographics, device and network context, physical
context, social context, and temporal context. The ontology will be designed
to be flexible and adaptable, allowing for the incorporation of new or
changing user characteristics and environmental factors.

To develop the UCPO, we consult a variety of information sources and
references such as academic articles, conference proceedings, textbooks, and
online resources related to ontology engineering, user modeling, and
personalization. In selecting the information sources, it is important to
consider their relevance to the scope and goals of the UCPO, as well as the
quality and credibility of the sources. We manually reviewed and analyzed the
relevant literature to identify the key concepts and terms related to user
context and personalization. Additionally, consulting domain experts, ontology
engineers, and end-users can provide valuable insights into the specific user
characteristics and contextual factors that should be incorporated into the
ontology.

Table 1: Examples of informal competency questions tables. ID |  Questions
---|---
CQ1 | What is demographic information of the user ?
CQ2 | What is the user’s preferred vehicle type?
CQ3 | What is the user’s budget for a vehicle purchase?
CQ4 | Which particular vehicle models are favored by the user?
CQ5 | What is the user’s driving environment?
CQ6 | What is the user’s preferred vehicle brand?
CQ7 | What are the primary use cases for a particular vehicle model?
CQ8 | What is the user’s preferred vehicle transmission type

The ontology requirements can be formulated by using informal competency
questions. The set of competency questions helps to identify and define the
key concepts, relationships, and attributes that should be included in the
ontology. Competency questions serve as a guide for ontology development and
help to ensure that the resulting ontology accurately represents the user
context and profile information required for personalization. As in the table
1, we illustrate some informal competency questions related to define the key
characteristics of the user contextual profile in the vehicle sales domain,
which can then be used to inform the development of the UCPO.

Defining the goal, scope, information sources, and competency questions helps
to establish a shared understanding of the primary requirements of the UCPO
between domain experts and ontology engineers. In the next section, we will
focus on the main phases of the development process, where the key components
of the UCPO are designed and implemented to obtain a comprehensive and
adaptable version.

###  3.2 Design and implementation phase

Considered as the core of the ontology engineering process, the design and
implementation phase consists of several iterations of ontology design,
implementation, and testing until a comprehensive and adaptable ontology is
achieved. The design phase begins by identifying the key concepts and
relationships that should be included in the ontology based on the competency
questions formulated in the previous phase. SAMOD prescribes an iterative
process that aims to build the final model through a series of small steps.
One key component of this process is the use of concept modelets, which are
standalone models that describe particular aspects of the domain under
consideration. These modelets are used to provide an initial conceptualization
without being constrained by the current model available after the previous
iteration of the process.

We have organized the development of the UCPO into two separate modelets. The
first modelet focuses on representing mostly static and permanent user
characteristics. The second modelet presents both static and dynamic
information about the user profile and contextual factors that affect the user
and their profile. As shown in Figure 1, the first modelet is responsible for
organizing user demographic and social information, such as $Gender$, $Age$,
$address$, $occupation$, $Income$, $Language$, $NumberOfChildren$,
$MaritalStatus$, $Education$, and more. Gender refers to a person’s biological
identity as male or female. Age refers to the user’s age, typically measured
in years. Address refers to a person’s physical location, which can be used to
gather contextual information about the user’s environment. Occupation refers
to a person’s job or profession, which can provide insights into their daily
routine and transportation needs. Income refers to a person’s financial
status, which can affect their purchasing power and preferences. Language
refers to a person’s preferred language for communication, which can influence
the type of content and recommendations that they receive. NumberOfChildren
refers to the number of children a person has, which can impact their vehicle
preferences and needs. MaritalStatus refers to a person’s current marital
status, which can provide insights into their family structure and needs.
Education level refers to the highest level of education a person has
completed, which can be a factor in determining their interests and
preferences. In general, these types of information have been well-organized
into classes in many previous works, allowing for the description of user
information in a categorical manner. In our research, we utilized the work of
Bermudez et al. [15] for the basic classes of user information. Moreover, we
have created a class named $PersonalProfile$, which acts as a superclass for
the classes related to the demographic and social information. The
$PersonalProfile$ class can contains attributes such as first name, last name
of the user.

Figure 1: A dedicated ontology section for demographic and social information
of users, composed of the User Profile Ontology (highlighted in red boxes with
$up$ prefix), User Contextual Profile Ontology (highlighted in blue boxes with
$ucpo$ prefix), and the vehicle domain ontology (highlighted in green boxes
with $vo$ prefix).

The second modelet focuses on the representation of the structure and
relationships between a user, their user profile, and their user context. User
context refers to the circumstances, environment, and situation in which a
user interacts with a system or application. It includes a wide range of
factors such as the user’s location, time, device, preferences, goals, and
social context, among others. User context is crucial because it can have a
significant impact on the user’s behavior, preferences, and needs. As shown in
Figure 2, we investigated four particular contextual information about the
user, including $Time$, $Location$, $Activity$, and $Device$ classes. In our
design, we created two subclasses of $Context$ class: the $UserContext$ class
that describes general contextual information about the user and the
$ProfileContext$ class, which describes contextual information for each user
profile. Furthermore, we have provided the ability for each user to declare
their preferences and create various types of profiles that align with their
intended purpose while interacting with the system or application. To
structure this type of information, we have created two classes: $Preference$
and $UserProfile$.

In order to address the needs and preferences of users in the vehicle sales
domain, we have created a set of preferences that relate to the user’s desired
vehicle. This set of preferences includes several classes such as
$VehicleType$, $RouteType$, $Mileage$, $Color$, $NumberOfPlaces$, $State$,
$Budget$, and $Brand$. The $VehicleType$ class refers to the type of vehicle
the user is interested in, such as sedan, SUV, or truck. The $RouteType$ class
describes the user’s preferred driving route, such as highway or city streets.
The $Mileage$ class indicates the user’s desired mileage range. The $Color$
class refers to the user’s preferred color for their vehicle. The
$NumberOfPlaces$ class indicates the number of seats or passengers the user
wants in their vehicle. The $State$ class represents the vehicle state where
the user intends to purchase. The $Budget$ class refers to the user’s budget
for the vehicle purchase. The $Brand$ class represents the user’s preferred
vehicle brand.

It is important to note that depending on the specific domain and application,
additional preferences may be necessary to fully capture the user’s needs and
preferences. Therefore, sub-classes of $Preference$ class can be created and
customized based on the specific requirements of the vehicle sales domain and
the user base. By incorporating these preferences into the user contextual
profile ontology, we can provide a more personalized and tailored user
experience.

Figure 2: A dedicated ontology section for user preferences and contextual
information of user and each particular profile.

Using the conceptual model, we can outline the major classes, attributes, and
relationships between them (as illustrated in Figures 1 and 2). This model is
then refined through multiple iterations, taking feedback from stakeholders
and end-users into consideration to ensure that it accurately represents the
domain knowledge and requirements.

The implementation of the UCPO involves the process of actually creating and
deploying the ontology in a system or application. This requires converting
the conceptual model of the ontology into a computer-readable format using
ontology languages such as OWL or RDF. As shown in Figure 3, we have
implemented the UCPO using the Web Ontology Language (OWL) with the support of
the Protégé-OWL editor [18].

Figure 3: A snapshot of the User Contextual Profile Ontology in the Protégé-
OWL editor

After the design and implementation phase, the ontology undergoes further
refinement through multiple rounds of testing and feedback until a final
version is achieved. The ontology is then evaluated to ensure it meets the
intended goals and requirements and can be integrated with existing systems
and applications. In the next section, we present the test and evaluation
phase of the ontology.

###  3.3 Test and evaluation phase

In the final phase of the UCPO development, we focus on testing and evaluating
the produced ontology. The tests can be categorized into a model test, a data
test, and query tests. To perform the model test, we can employ evaluation
metrics such as OOPS! [21], OntoQA [31], and OntoMetrics [7] to verify the
overall consistency of the developed ontology. The data test consists of
checking the validity of the model after populating it with instance triplets.
Finally, for query tests, informal competency questions must be transformed
into SPARQL queries to ensure that the expected answers are obtained. The
ontology model must be adjusted until all tests are successful. This section
presents our method for conducting these types of tests to ensure the
ontology’s adaptation.

In our work, we evaluated the quality of our ontology by examining its
structure. Specifically, we observed several metrics using the OntoMetrics
framework. Table 2 presents the results from the analysis of OntoMetrics with
base metrics and schema metrics. The base metrics consist of simple metrics
that count the number of axioms, classes, object properties, data properties,
and individual instances. These metrics provide statistics about the ontology
with regard to the quantity of elements present within it. Based on these
statistics, we can conclude that our ontology is a lightweight knowledge graph
that can be adopted in different application architectures. Moreover, the
Description Logic (DL) expressivity of the ontology is $\mathcal{ALH}(D)$,
which indicates the DL variant that the ontology belongs to.
$\mathcal{ALH}(D)$ refers to attribute language $\mathcal{AL}$ with role
hierarchy ($\mathcal{H}$).

Table 2: Results achieved from applying various metrics to UCPO. Base Metrics
---
Class count | 38
Object property count | 27
Data property count | 16
Properties count | 43
Individual count | 159
SubClassOf axioms count | 22
Object property domain axioms count | 28
Object property range axioms count | 33
DL expressivity | $\mathcal{ALH}(D)$
Schema Metrics
Attribute richness (AR) | 0.421053
Inheritance richness (IR) | 0.578947
Relationship richness (RR) | 0.55102
Axiom/class ratio |  30.552632
Class/relation ration | 0.77551

In schema metrics, the evaluation of the ontology focuses on the measure of
attribute, inheritance, and relationship richness [31]. Particularly, the
attribute richness ($AR$) is considered as the average number of attributes
per class and is computed as follows:

$AR=\frac{|NA|}{|C|}$ (1)

where $NA$ is the number attributes for all classes and $C$ denotes the number
of classes. The high score of the attribute richness indicates the high
quality of ontology design and the amount of information belong to instances.
The inheritance richness ($IR$) is defined as the average number of sub-
classes per class and is computed as follows:

$IR=\frac{|H|}{|C|}$ (2)

where $H$ is the sum of the number of inheritance relationships. The
inheritance richness allows describing the distribution of information on
different levels of the ontology. The relationship richness ($RR$) is defined
as the percentage of the relationships between classes and is computed as
follows:

$RR=\frac{|P|}{|H|+|P|}$ (3)

where $P$ the sum of the number of non-inheritance relationships. The
inheritance richness is used to measure the diversity of the relationship’s
types in the ontology. Moreover, the other metrics such as axiom/class ratio
and class/relation ratio demonstrate the ratio between axioms and classes, and
classes-relations. Based on the explanation of metrics, these measured results
in table 2 indicate that our proposed ontology is balanced between a
horizontal (or shallow) ontology and vertical (or deep) ontology.

To conduct a data test for the UCPO, we need to check the validity of the
model after populating it with instance triplets. For example, let’s assume we
have created instances for Henri’s two profiles: his professional profile and
his family profile. We can populate these instances with data such as his
occupation and work location for his professional profile, and his number of
children and preferred family activities for his family profile. This ensures
that the UCPO can capture and express this information in a structured and
consistent manner.

In order to conduct a query test on the UCPO, we can convert the set of
informal competency questions, as shown in Table 1, into SPARQL queries to
ensure that the expected answers are obtained. The queries should be
constructed in a way that enables the retrieval of relevant information from
the ontology, utilizing the appropriate classes, properties, and instances.
For instance, if the competency question is “What is the user’s preferred
vehicle brand?”, a suitable SPARQL query can be formulated as follows:

⬇

PREFIX vo: <http:// vivocaz.fr/vo/ns#>

PREFIX ucpo: <http:// vivocaz.fr/ucpo/ns#>

SELECT ?user ?brand

WHERE {

?user ucpo:hasUserProfile ?userProfile.

?userProfile ucpo:hasVehiclePreference ?userVehiclePreference .

?userVehiclePreference upo:hasFavoriteBrand ?brand .

} ORDER BY ?user LIMIT 10

Listing 1: A SPARQL query expression used to search for the first 10 users and
their favorite brands.

The test and evaluation phase ensures that the UCPO is consistent, valid, and
effective in achieving its intended purpose. By conducting model tests, data
tests, and query tests, we can evaluate the overall structure of the ontology,
the validity of its populated instances, and its ability to accurately answer
competency questions. In the next section, we will explore an application of
the UCPO through a case study focused on generating personalized
recommendations.

##  4 Case study - Personalized Recommendations

To demonstrate the application of the UCPO, we conducted a case study on
personalized recommendations in the vehicle sales domain. Using the ontology,
we were able to capture the user’s contextual information, preferences, and
profiles to provide tailored recommendations for vehicle purchases.
Specifically, scenarios were considered where a user created a profile
specifying their preferred $vehicle$ $type$, $route$ $type$, $mileage$,
$color$, $number$ $o$f $seats$, $location$, $brand$, and $budget$.
Additionally, the user’s profile context included their $activity$, $time$
$of$ $day$, and $device$. This information was used to provide personalized
recommendations for vehicle purchases that catered to the user’s individual
requirements and preferences.

Louis and Pierre are two potential buyers of used vehicles who have registered
on a vehicle sales application and set up their profiles. Louis has specified
his preferences for $vehicle$ $type$, $route$ $type$, $mileage$, $color$,
$number$ $of$ $seats$, $location$, $brand$, and $budget$, while expressing a
particular interest in sedan models. On the other hand, Pierre has emphasized
the importance of safety features as a top priority due to frequent travel
with his family.

The application integrates the UCPO to capture their contextual information
and preferences. By leveraging Louis’s contextual information and preferences,
including his activity or location such as the Peugeot models he has
previously liked, the application generates personalized recommendations for
vehicle purchases that cater to his specific needs and preferences. The
application recommends models such as $Peugeot$ $206$, $Peugeot$ $207$, or
$Peugeot$ $208$ that Louis may like, based on his stated preferences and his
profile context.

On the other hand, the application captures Pierre’s contextual information
and preferences, including his activity or location such as the fact that he
previously owned a $Toyota$ and expressed interest in hybrid vehicles, to
provide tailored recommendations. By leveraging this information, the
application’s recommendation engine suggests several vehicles that fit
Pierre’s criteria, including a $Toyota$ $RAV4$ $Hybrid$, a $Honda$ $CR-V$
$Hybrid$, and a $Lexus$ $UX$ $Hybrid$. The system takes into account factors
such as safety ratings, fuel efficiency, and cost to make the best relevant
recommendations for Pierre.

Consider the example of Henri, who was mentioned in the introduction section,
and has two distinct profiles: one as a professional profile and the other as
a family profile. It is essential to provide personalized recommendations for
Henri that meet his specific needs and preferences in each profile. By
leveraging the UCPO, which captures user profiles and contextual information
of each profile, relevant recommendations can be provided that cater to each
corresponding profile. Specifically, for Henri’s professional profile who has
shown interest in Renault models, vehicles such as $Renault$ $Megane$ and
$Renault$ $Talisman$, which have spacious trunks, advanced safety features,
and good fuel efficiency, could be recommended, as Henri may require frequent
business trips and the recommendations match the contextual information of his
activities on the application. For Henri’s family profile, who has indicated
interest in SUV vehicle types, vehicles such as $Koleos$ $SUV$ and $Renault$
$Scenic$, which have ample space and storage and rear-seat airbags for
passengers, could be recommended as Henri has shown a preference for these
models in the family profile on the applications.

These scenarios illustrate how the UCPO ontology can be utilized to capture
contextual information and preferences to generate personalized
recommendations that cater to the specific needs and preferences of each user.
By analyzing and understanding user information, systems or applications can
provide more accurate and effective recommendations, as demonstrated in
various works [13, 9]. By leveraging the UCPO in this way, it is fully
potential to develop a system or application that provides more personalized
recommendations tailored to the user’s unique needs, preferences, and
contextual information.

##  5 Conclusion and Perspectives

In this paper, we have presented the design and development process of the
User Contextual Profile Ontology with a focus on the vehicle sales domain.
This work established the bridge for the gap in the representation of user
profile and user context by designing a user contextual profile ontology that
considers both user profiles and contextual information on each profile.
Thereby, the UCPO serves as a structural framework for standardizing the
representation of user profiles and contextual information. It is worth
emphasizing that the development of the UCPO follows the main phases of
ontology engineering, ensuring its consistency, validity, and effectiveness in
achieving its intended purpose. The iterative development process of the UCPO
involved testing and evaluation phases, which allowed for assessing the
ontology’s overall structure, validity of populated instances, and ability to
answer competency questions accurately. The case study presented illustrates
the use of the UCPO in generating personalized recommendations in the vehicle
sales domain. By leveraging the ontology’s ability to capture user preferences
and requirements accurately, the system can provide tailored recommendations
that cater to the specific needs, preferences, and context of each user. In
future work, we plan to extend the ontology to include additional information
sources, such as user reviews and social media feeds, to enhance the accuracy
of personalized recommendations. Finally, we aim to collaborate with industry
partners to integrate the UCPO into commercial applications and evaluate its
effectiveness in real-world settings.

## Acknowledgment

This work was funded by the French Research Agency (ANR) and by the company
Vivocaz under the project France Relance - preservation of R&D employment
(ANR-21-PRRD-0072-01).

## References

  * [1] Adewoyin, O., Wesson, J., Vogts, D.: User modelling to support behavioural modelling in smart environments. In: 2022 3rd International Conference on Next Generation Computing Applications (NextComp). pp. 1–6. IEEE (2022)
  * [2] De Nicola, A., Missikoff, M., Navigli, R.: A proposal for a unified process for ontology building: Upon. In: Database and Expert Systems Applications: 16th International Conference, DEXA 2005, Copenhagen, Denmark, August 22-26, 2005. Proceedings 16. pp. 655–664. Springer (2005)
  * [3] Fernández-López, M., Gómez-Pérez, A., Juristo, N.: Methontology: From ontological art towards ontological engineering. In: Proceedings of the Ontological Engineering AAAI-97 Spring Symposium Series. American Asociation for Artificial Intelligence (1997)
  * [4] He, S., Fang, M.: Ontological user profiling on personalized recommendation in e-commerce. In: 2008 IEEE International Conference on e-Business Engineering. pp. 585–589. IEEE (2008)
  * [5] Kourtiche, A., mohamed Benslimane, S., Hacene, S.B.: Oupip: Ontology based user profile for impairment person in dynamic situation aware social networks. International Journal of Knowledge-Based Organizations (IJKBO) 10(2), 12–34 (2020)
  * [6] Kritikou, Y., Demestichas, P., Adamopoulou, E., Demestichas, K., Theologou, M., Paradia, M.: User profile modeling in the context of web-based learning management systems. Journal of Network and Computer Applications 31(4), 603–627 (2008)
  * [7] Lantow, B.: Ontometrics: Putting metrics into use for ontology evaluation. In: Proceedings of the International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management. p. 186–191. IC3K 2016, SCITEPRESS - Science and Technology Publications, Lda, Setubal, PRT (2016), https://doi.org/10.5220/0006084601860191
  * [8] Le, N.L., Abel, M.H., Gouspillou, P.: Apport des ontologies pour le calcul de la similarité sémantique au sein d’un système de recommandation. In: 33ème Journées Francophones d’Ingénierie des Connaissances (IC 2022). pp. 189–198 (2022)
  * [9] Le, N.L., Abel, M.H., Gouspillou, P.: Towards an ontology-based recommender system for the vehicle sales area. In: Progresses in Artificial Intelligence & Robotics: Algorithms & Applications. pp. 126–136. Springer International Publishing, Cham (2022)
  * [10] Le, N.L., Abel, M.H., Gouspillou, P.: A personalized recommender system based-on knowledge graph embeddings. In: The 3rd International Conference on Artificial Intelligence and Computer Vision (AICV2023), March 5–7, 2023. pp. 368–378. Springer (2023)
  * [11] Le, N.L., Abel, M.H., Gouspillou, P.: Improving semantic similarity measure within a recommender system based-on rdf graphs. In: Proceedings of the 6th International Conference on Information Technology & Systems (2023)
  * [12] Le, N.L., Tireau, A., Venkatesan, A., Neveu, P., Larmande, P.: Development of a knowledge system for big data: Case study to plant phenotyping data. In: Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics. pp. 1–9 (2016)
  * [13] Le, N.L., Zhong, J., Negre, E., Abel, M.H.: Constraint-based recommender system for crisis management simulations. In: The 56th Hawaii International Conference on System Sciences (2023)
  * [14] Li, S., Abel, M.H., Negre, E.: Using user contextual profile for recommendation in collaborations. In: Research & Innovation Forum 2019: Technology, Innovation, Education, and their Social Impact 1. pp. 199–209. Springer (2019)
  * [15] Maria, B., Payam, B., Sefki, K.: User profile ontology (2015), http://iot.ee.surrey.ac.uk/citypulse/ontologies/up
  * [16] Maria, G., Akrivi, K., Costas, V., George, L., Constantin, H.: Creating an ontology for the user profile: Method and applications. In: Proceedings AI* AI Workshop RCIS. pp. 407–412 (2007)
  * [17] Muñoz, H.J.M., Cardinale, Y.: Gente: An ontology to represent users in the tourism context. In: 2021 XLVII Latin American Computing Conference (CLEI). pp. 1–10. IEEE (2021)
  * [18] Musen, M.A.: The protégé project: a look back and a look forward. AI matters 1(4), 4–12 (2015)
  * [19] Noy, N.F., McGuinness, D.L., et al.: Ontology development 101: A guide to creating your first ontology (2001)
  * [20] Peroni, S.: A simplified agile methodology for ontology development. In: OWL: Experiences and Directions–Reasoner Evaluation: 13th International Workshop, OWLED 2016, and 5th International Workshop, ORE 2016, Bologna, Italy, November 20, 2016, Revised Selected Papers 13. pp. 55–69. Springer (2017)
  * [21] Poveda-Villalón, M., Gómez-Pérez, A., Suárez-Figueroa, M.C.: Oops!(ontology pitfall scanner!): An on-line tool for ontology evaluation. International Journal on Semantic Web and Information Systems (IJSWIS) 10(2), 7–34 (2014)
  * [22] Rimitha, S., Abburu, V., Kiranmai, A., Marimuthu, C., Chandrasekaran, K.: Improving job recommendation using ontological modeling and user profiles. In: 2019 Fifteenth International Conference on Information Processing (ICINPRO). pp. 1–8. IEEE (2019)
  * [23] Shimizu, C., Hammar, K., Hitzler, P.: Modular ontology modeling. Semantic Web pp. 1–31 (2021)
  * [24] Sieg, A., Mobasher, B., Burke, R.: Web search personalization with ontological user profiles. In: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management. pp. 525–534 (2007)
  * [25] Skillen, K.L., Chen, L., Nugent, C.D., Donnelly, M.P., Burns, W., Solheim, I.: Ontological user profile modeling for context-aware application personalization. In: Ubiquitous Computing and Ambient Intelligence: 6th International Conference, UCAmI 2012, Vitoria-Gasteiz, Spain, December 3-5, 2012\. Proceedings 6. pp. 261–268. Springer (2012)
  * [26] Sosnovsky, S., Dicheva, D.: Ontological technologies for user modelling. International Journal of Metadata, Semantics and Ontologies 5(1), 32–71 (2010)
  * [27] Suárez-Figueroa, M.C., Gómez-Pérez, A., Fernández-López, M.: The neon methodology for ontology engineering. In: Ontology engineering in a networked world, pp. 9–34. Springer (2011)
  * [28] Sutterer, M., Droegehorn, O., David, K.: Upos: User profile ontology with situation-dependent preferences support. In: First International Conference on Advances in Computer-Human Interaction. pp. 230–235. IEEE (2008)
  * [29] Sutterer, M., Droegehorn, O., David, K.: User profile selection by means of ontology reasoning. In: 2008 Fourth Advanced International Conference on Telecommunications. pp. 299–304. IEEE (2008)
  * [30] Tamine-Lechani, L., Boughanem, M., Daoud, M.: Evaluation of contextual information retrieval effectiveness: overview of issues and research. Knowledge and Information Systems 24, 1–34 (2010)
  * [31] Tartir, S., Arpinar, I.B., Sheth, A.P.: Ontological evaluation and validation. In: Theory and applications of ontology: Computer applications, pp. 115–130. Springer (2010)
  * [32] Wang, X.H., Zhang, D.Q., Gu, T., Pung, H.K.: Ontology based context modeling and reasoning using owl. In: IEEE annual conference on pervasive computing and communications workshops, 2004. Proceedings of the second. pp. 18–22. Ieee (2004)


# Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman111Authors contributed equally to this work1, Angelina
Wang111Authors contributed equally to this work2, Morgan Scheuerman3, Su Lin
Blodgett4, Kristen Laird5, Hanna Wallach4, Solon Barocas4

###### Abstract

In this paper, we examine computational approaches for measuring the
“fairness” of image tagging systems, finding that they cluster into five
distinct categories, each with its own analytic foundation. We also identify a
range of normative concerns that are often collapsed under the terms
“unfairness,” “bias,” or even “discrimination” when discussing problematic
cases of image tagging. Specifically, we identify four types of
representational harms that can be caused by image tagging systems, providing
concrete examples of each. We then consider how different computational
measurement approaches map to each of these types, demonstrating that there is
not a one-to-one mapping. Our findings emphasize that no single measurement
approach will be definitive and that it is not possible to infer from the use
of a particular measurement approach which type of harm was intended to be
measured. Lastly, equipped with this more granular understanding of the types
of representational harms that can be caused by image tagging systems, we show
that attempts to mitigate some of these types of harms may be in tension with
one another.

## Introduction

In recent years, computer vision has gone from a largely academic research
topic to a set of technologies that touch the lives of people in a variety of
impactful ways. With this success comes the importance of understanding when
computer vision systems can cause harms. In this paper, we consider some of
the harms that can be caused by a seemingly mundane, yet widespread, computer
vision task: image tagging.

Prior work has already documented several problematic cases of image tagging.
For example, researchers have identified harms caused by gender classification
systems and other systems that sort people into social groups based on their
appearance (Keyes 2018; Barlas et al. 2021), demonstrated that the performance
of image tagging systems can vary according to the identities of the people
depicted in images (Hendricks et al. 2018; Zhao et al. 2017), and revealed a
variety of different ways that such systems can perpetuate stereotypes
(Schwemmer et al. 2020; Bhargava and Forsyth 2019). These findings have been
held up as evidence that image tagging systems can be “unfair,” “biased,” or
even “discriminatory.” However, in many of these cases, it can be difficult to
identify the full range of normative values at stake.

We aim to bring greater structure and precision to discussions of some of the
harms caused by image tagging systems and the many computational approaches
that have been proposed to measure them. We first provide relevant background
by defining image tagging and revisiting the distinction between allocational
and representational harms (Barocas et al. 2017)—emphasizing that problematic
cases of image tagging can affect the understandings, beliefs, and attitudes
that people hold about particular social groups. We then analyze previously
proposed computational approaches for measuring the “fairness” of image
tagging systems, clustering them into five distinct categories, each with its
own analytic foundation.111We restrict our focus to only quantitative
measurement approaches, setting aside a survey of qualitative measurement
approaches (e.g., Barlas et al. 2019b; Hu and Strout 2018; Kay, Matuszek, and
Munson 2015a; Miceli, Schuessler, and Yang 2020; Scheuerman et al. 2020;
Offert and Bell 2020) for future work. Next, we identify four types of
representational harms that can be caused by image tagging systems, moving
beyond generic concerns about “unfairness,” “bias,” or “discrimination.” We
consider how different computational measurement approaches map to each of
these types of harms, finding that each approach can be used to measure all
four types. In light of this, we argue that no single measurement approach is
definitive and that, without explicit reasoning, it is not possible to infer
from the use of a particular measurement approach which type of harm was
intended to be measured. Lastly, we conclude by exploring tensions between
attempts to mitigate different types of representational harms—and what these
tensions might mean for developing appropriate mitigations.

## Background

In this section, we define both image tagging and representational (versus
allocational) harms (Barocas et al. 2017).

### Image Tagging

Image tagging is the task of applying tags to an image so as to describe
salient aspects of its contents. Image tagging differs from the related task
of object recognition, where the goal is to identify all objects that are
depicted in an image. Crucially, although both tasks involve applying tags to
images, object recognition focuses solely on objects and is concerned with
completeness, whereas image tagging focuses on other properties (e.g.,
actions, facial expressions, social groups) as well as objects and is
concerned with salience.

The two tasks also differ in terms of the ways the resulting tags are
typically used. The tags applied by image tagging systems are often intended
for human consumption. For example, image tagging systems are commonly used to
generate alt text descriptions of images for blind and low-vision users (Wu et
al. 2017), provide search or grouping functionality in photo-management
software, and enable users of search engines to find images based on their
contents as well as their accompanying text. In contrast, the tags applied by
object recognition systems are often used to automate tasks, obviating the
need for human involvement. For example, semi-autonomous vehicles rely on
object recognition systems to detect obstacles, but may take autonomous
actions to avoid colliding with those obstacles rather than simply
communicating their presence to drivers.

Image tagging has a unique normative valence because it is concerned with
salience and applies tags that are often intended for human consumption. The
tags applied to images by image tagging systems are expressions of what—and
who—is important and how they deserve to be described. Because the tags are
consumed by people, they affect what people view as salient and how they
understand the world.

### Representational Harms

Much of the foundational work on the “fairness” of machine learning systems
has focused on what Barocas et al. (2017) call allocational harms—that is,
when people belonging to particular social groups are unfairly deprived of
access to important opportunities or resources. Allocational harms often arise
in domains like employment, finance, and housing. However, computer vision
systems, like natural language processing systems, are not typically used to
allocate opportunities or resources and are therefore less prone to causing
allocational harms (Barocas et al. 2017; Crawford 2017; Blodgett et al. 2020).
Instead, these systems produce outputs that can affect the understandings,
beliefs, and attitudes that people hold about particular social groups, and
thus the standings of those groups within society. Barocas et al. (2017) call
these representational harms. Concerns about representational harms have
motivated research examining, for example, the ways that search engines
represent some social groups in less favorable ways than they do others (e.g.,
returning more images of men than women in response to search queries for
“doctor” (Kay, Matuszek, and Munson 2015b) or returning pornographic websites
in response to queries for “Black girls” but not “white girls” (Noble 2018)).

## Computational Measurement Approaches

Equipped with definitions of image tagging and representational harms, we now
analyze previously proposed computational approaches for measuring the
“fairness” of image tagging systems. We focus on prior work that self-
identifies as measuring “unfairness,” “bias,” or “discrimination.” Although
most of the measurement approaches that we describe were proposed for image
tagging, some were proposed for other computer vision tasks but can be applied
to image tagging. We cluster the approaches into five distinct categories
based on their shared analytic foundations.

Incidence-based approaches test whether pre-identified problematic tags, types
of images, or pairings of tags and types of images occur in the inputs and
outputs of image tagging systems or in the datasets used to train them. These
approaches are premised on the belief that there are particular tags, types of
images, and pairings that are inherently problematic: problematic tags should
not be applied to any images, problematic types of images should not have any
tags applied to them, and problematic pairings should not occur.

For example, researchers have used incidence-based approaches to study tags
that refer to properties that cannot be determined from visual appearance
(Yang et al. 2020), such as gender (Keyes 2018; Scheuerman, Paul, and Brubaker
2019; Scheuerman et al. 2020), character traits (Barlas et al. 2019a), emotion
(Bard 2020; Kyriakou et al. 2019; Barlas et al. 2019a; Sedenberg and Chuang
2017), and physical attractiveness (Kyriakou et al. 2019). As another example,
researchers have uncovered the presence of lewd tags and images in several
widely-used computer vision datasets (Crawford and Paglen 2019; Yang et al.
2020; Prabhu and Birhane 2020). Finally, one of the most widely discussed
problematic pairings arose when Google Photos applied the tag _gorillas_ to an
image of Black people (Simonite 2018). Although neither the tag nor the image
would have been considered problematic in isolation, they become so when
paired together.

Because problematic tags, types of images, or pairings must be pre-identified,
using an incidence-based approach means first answering the question of
_which_ tags, types of images, or pairings should be considered problematic—a
task that requires extensive background knowledge. Yet even with such
knowledge, pre-identifying _all_ problematic tags, types of images, or
pairings is likely impossible.

Distribution-based approaches compare differences in the distributions of tags
or types of images in the inputs and outputs of image tagging systems or in
the datasets used to train them. One distribution-based approach compares the
distribution of images that depict different social groups in a training
dataset to some reference distribution, typically the uniform distribution
(Buolamwini and Gebru 2018; Kärkkäinen and Joo 2021; Wang et al. 2021; Yang et
al. 2020). A more ambitious variant of this approach focuses on the
distributions of images that depict attributes (e.g., payot), artifacts (e.g.,
the Torah), or activities (e.g., lighting the menorah) that are bound up with
the identities of different social groups, again comparing them to some
reference distribution (Shankar et al. 2017a; Wang et al. 2021; DeVries et al.
2019). Both variants are premised on the idea that image tagging systems may
under-perform for social groups, attributes, artifacts, or activities that are
not sufficiently well represented in the datasets used to train them.

Another distribution-based approach compares the distributions of tags—either
in a training dataset (Barlas et al. 2019a; Otterbacher et al. 2019; Wang et
al. 2021; Yang et al. 2020) or output by an image tagging system (Alvi,
Zisserman, and Nellaker 2018; Kyriakou et al. 2019; Barlas et al. 2019a;
Schwemmer et al. 2020)—for images that depict different social groups. Other
researchers have instead compared the distributions of social groups depicted
in images for different tags (Kay, Matuszek, and Munson 2015a). Both variants
of this approach are premised on the belief that it is harmful for images of
different social groups to have different distributions of tags, even if these
tags are correctly applied, because this casts the groups in different lights.

A final approach compares the distribution of tags applied to images that
depict different social groups in a image tagging system’s training dataset to
the distribution of tags applied by the system to images that depict these
social groups (Wang et al. 2019, 2020; Zhao et al. 2017; Kay, Matuszek, and
Munson 2015a; Wang and Russakovsky 2021). This approach is premised on the
belief that the outputs of image tagging systems should not exacerbate any
differences between social groups that are present in their training datasets.

Performance-based approaches compare the performance of image tagging systems
for particular tags or types of images. These approaches are premised on the
belief that image tagging systems should perform equally well for different
social groups. One widely discussed performance-based approach tests whether
image tagging systems exhibit comparable performance for images that depict
different social groups when applying tags that indicate membership in those
social groups. For example, researchers have investigated whether binary
gender classification systems perform better for images that depict men than
they do for images that depict women (Bhargava and Forsyth 2019; Buolamwini
and Gebru 2018; Hendricks et al. 2018). Other researchers have conducted
similar analyses focusing on race, ethnicity, and age (Das, Dantcheva, and
Bremond 2018; Kärkkäinen and Joo 2021; Kyriakou et al. 2019). Some researchers
have also taken an intersectional lens by focusing on different combinations
of social groups (Buolamwini and Gebru 2018; Das, Dantcheva, and Bremond
2018).

Another performance-based approach tests whether image tagging systems exhibit
comparable performance for images that depict different social groups when
applying _any_ tags. For example, researchers have investigated whether image
tagging systems perform better when tagging everyday objects in images that
depict some social groups than they do for images that depict others, finding
performance disparities between genders (Bhargava and Forsyth 2019; Wang et
al. 2020), ages (Wang et al. 2020), races (Buolamwini and Gebru 2018; Wilson,
Hoffman, and Morgenstern 2019), socioeconomic statuses (DeVries et al. 2019),
and geographic locations (Shankar et al. 2017a; DeVries et al. 2019)—as well
as different combinations thereof (Buolamwini and Gebru 2018; Khiyari and
Wechsler 2016; Klare et al. 2012).

Perturbation-based approaches involve varying aspects of images provided as
inputs to image tagging systems to see whether different tags are applied,
either correctly or incorrectly. These approaches are premised on two beliefs:
first, that the behaviors of image tagging systems should not reflect spurious
correlations in their training datasets; and second, that particular aspects
of images should not be the basis for particular differences in tagging
behaviors (e.g., the appearance of a person’s face should not affect the
occupation with which they are tagged). As a result, they are often used to
investigate whether particular aspects of images are driving particular
tagging behaviors, providing an an opportunity to reflect on whether these
behaviors are normatively concerning (Wilson, Hoffman, and Morgenstern 2019;
Muthukumar et al. 2018; Klare et al. 2012). For example, if removing occluded
faces from images reduces performance disparities between social groups, these
disparities are likely due to occlusions rather than something more
normatively concerning (Wilson, Hoffman, and Morgenstern 2019).

One perturbation-based approach focuses specifically on invariance. This
approach tests whether the tags applied by image tagging systems are
unaffected by perturbations of particular aspects of images that are known to
be an inappropriate basis for _any_ differences in tagging behaviors. For
example, researchers have re-taken photos using props, such as wigs (Rodden
2017; Algorithmic Justice League 2021), or used GANs to generate faces with
different characteristics (Denton et al. 2021; McDuff et al. 2019), testing
whether there are any differences in the tags applied to these images. Other
researchers have selected pairs of semantically similar images that depict
different social groups (e.g., similarly composed photos of CEOs of different
genders), again testing for any differences in tagging behaviors (Otterbacher
2018; Steed and Caliskan 2021; Stock and Cisse 2018).

Internals-based approaches directly investigate the latent representations
learned by image tagging systems. Like perturbation-based approaches, these
approaches are premised on the belief that the behaviors of image tagging
systems should not reflect spurious correlations in their training datasets
and the belief that particular aspects of images should not be the basis for
particular differences in tagging behaviors. One internals-based approach
relies on saliency maps, which highlight the regions of images that are most
salient when applying particular tags and are typically generated using
gradients. This approach can therefore shed light on the reasoning behind
particular tagging behaviors—for example, whether an image tagging system is
focusing on the people depicted in an image when tagging objects—regardless of
whether the tags in question are correctly applied. Saliency maps have been
used to investigate tagging behaviors from quantitative perspectives (e.g.,
when tagging an object depicted in an image, investigating how much of the
object’s bounding box overlaps with the most salient region of the image) and
qualitative perspectives (e.g., investigating what is depicted in the most
salient region of an image) (Hendricks et al. 2018; Jia, Lansdall-Welfare, and
Cristianini 2018; Muthukumar et al. 2018; Singh et al. 2020).

A second internals-based approach tests whether image tagging systems that are
not intended to apply tags that indicate membership in social groups still
encode notions of social groups in their feature representations. This
suggests that they are inappropriately using these notions as the basis for
tagging behaviors (Alvi, Zisserman, and Nellaker 2018; Roberts 2018; Song and
Shmatikov 2020; Wang et al. 2019).

## Representational Harms

Although the computational measurement approaches that we described in the
previous section have played an important role in raising awareness of and
developing mitigations for some of the harms caused by image tagging systems,
their normative motivations are often unclear. These approaches are all
premised on beliefs that particular properties or behaviors of image tagging
systems or the datasets used to train them are self-evidently problematic,
even though these properties or behaviors are rarely connected to concrete
harms. This lack of clarity is not unique to image tagging systems or even
computer vision systems—Blodgett et al. (2020) found that computational
approaches for measuring the “fairness” of natural language processing systems
often have similarly unclear normative motivations.

In this section, we draw on the work of Blodgett et al. (2020) to argue that
image tagging systems cause representational harms by contributing to the
reproduction of harmful social hierarchies—that is, by casting some social
groups as inferior to others and confining them to subordinate positions
within society. We emphasize that representational harms are not harms to
specific individuals; they relate to the standings of particular social
groups, affecting _all_ members of those groups (Selbst and Barocas 2023). We
identify four types of representational harms that can be caused by image
tagging systems, delineating how each one reproduces harmful social
hierarchies. Having done this, we then introduce an additional normative
concern—that is, when image tagging systems deny people the ability to self-
identify—and discuss how it can give rise to representational harms.

We do not claim to provide an exhaustive list of all of the types of
representational harms that can be caused by image tagging systems. The range
of possible variation for tags and images make them remarkably expressive. As
a result, anticipating every way in which these variations could reproduce
harmful social hierarchies is impossible. Rather, our aim is to bring greater
structure precision to discussions of some of the harms caused by image
tagging systems and the many computational approaches that have been proposed
to measure them. We leave open the possibility that future work will identify
additional types of representational harms as our collective understanding of
image tagging evolves.

### Four Types of Representational Harms

Reifying social groups. Social groups are socially constructed, so their
boundaries are always contested and changing. Despite this inherent fluidity,
social groups are often treated as if they are natural, fixed, or objective
(Hanna et al. 2020), thereby reifying beliefs about their salience and
immutability and beliefs about the boundaries between them. Because image
tagging systems rely on visual characteristics and are intended to apply pre-
identified tags, they are especially likely to reify social groups, as well as
the idea that there are relationships between social groups and visual
appearance (Keyes 2018). For example, binary gender classification systems
rely on the following beliefs: first, that gender can be determined from
visual appearance; and second, that there are only two gender identities.
However, relationships between gender and visual appearance are contested and
unstable, as are gender identities themselves.

Stereotyping social groups. Image tagging systems can also perpetuate
stereotypes—over-generalized beliefs about social groups that reproduce
harmful social hierarchies (Barlas et al. 2021; Bhargava and Forsyth 2019).
For example, researchers have demonstrated that some image tagging systems
systematically apply the tag nurse to images of female doctors. Other
researchers have shown that images of female snowboarders are often
incorrectly tagged as depicting men (Hendricks et al. 2018; Zhao et al. 2017).
In these examples, the latent representations of doctors and snowboarders
learned by the image tagging systems are inextricably linked with gender—that
is, to be a female medical professional is to be a nurse, while to be a
snowboarder is to be a man.

These examples suggest that there are two ways that image tagging systems can
perpetuate stereotypes. First, if an image tagging system is intended to apply
tags that indicate membership in social groups, then it will necessarily
associate particular visual characteristics (e.g., riding a snowboard) with
particular social groups (e.g., men). This means that images that depict
people with those visual characteristics will be more likely to be tagged as
depicting members of those social groups. Second, if an image tagging system
is intended to apply other tags, then it may associate visual characteristics
that are common among members of particular social groups (e.g., long hair
among women) with other properties (e.g., being a nurse). This means that tags
that refer to those properties are more likely to be applied to images that
depict people with those visual characteristics.

Finally, image tagging systems can also perpetuate stereotypes in less overt
ways. For example, some image tagging systems tend to apply tags that refer to
professions to images that depict men, while applying tags that refer to
appearance to images that depict women (Schwemmer et al. 2020). Even if these
tags are correctly applied, they are harmful because they cast men and women
indifferent lights by perpetuating the stereotype that professions are
particularly salient for men, while appearance is particularly salient for
women.

Demeaning social groups. Image tagging systems can also demean social
groups—that is, suggest that members of those groups are less worthy of
respect than members of other groups. This can occur in very overt ways, such
as when image tagging systems apply tags that contain racial epithets—the very
purpose of which is to belittle members of particular racial groups—or tags
that are not inherently problematic but are demeaning when applied to images
that depict particular social groups. As an example of the latter, applying
the tag gorilla to an image of Black people is demeaning because there is a
long history of diminishing and dehumanizing Black people by likening them to
gorillas (Simonite 2018). Image tagging systems can also demean particular
social groups by applying tags that devalue attributes, artifacts, or
activities that are bound up with the identities of those groups. For example,
applying the tag costume to an image that depicts a religious group’s wedding
clothing trivializes the group’s traditions (Shankar et al. 2017b). Evaluative
terms can also be demeaning because they can be applied more (or less) often
to images that depict some social groups than to images that depict others.
For example, image tagging systems that apply the tag beautiful to fewer
images that depict a particular social group than they do to images that
depict others cast that group as being worthy of less esteem (Levin 2016) and
reinforce harmful beliefs about visual characteristics that are considered
beautiful.

Erasing social groups. Erasure occurs when image tagging systems fail to
correctly apply tags that indicate membership in social groups or tags that
refer to attributes, artifacts, or activities that are bound up with the
identities of social groups (Keyes 2018; Buolamwini and Gebru 2018; Benjamin
2019). Erasure of a particular social group implies that members of that group
are not worthy of recognition and contributes to their further marginalization
within society. For example, image tagging systems that do not apply the tag
person to images that depict people wearing hijabs erase Muslims. Similarly,
image tagging systems that fail to apply the tag menorah to images that depict
menorahs—perhaps because the tag is not available, despite the availability of
other tags that refer to Christian artifacts—erase Judaism.

Image tagging systems can also cause harms when they fail to acknowledge the
relevance of people’s membership in particular social groups to what is
depicted in images—for example, by applying the tags _people_ , _walking_ ,
and _street_ to an image depicting women suffragists marching. As well as
down-playing the gravity of what is depicted, these tags fail to acknowledge
the injustices that woman have suffered specifically due to their gender.
Erasure of this sort denies the role that membership in particular social
groups plays in people’s lived experiences, especially experiences relating to
oppression and violence, which are often perpetrated on the basis of people’s
membership in particular social groups.

### Denying People the Ability to Self-Identify

In this section, we introduce an additional normative concern posed by image
tagging systems—denying people the ability to self-identify—and discuss how it
can give rise to representational harms. Image tagging systems are often used
to sort people into social groups based on their appearance (e.g., gender
classification systems), imposing tags on them without their awareness,
involvement, or consent. Given the degree to which membership in social groups
shapes the way that people are treated within society, including others’
expectations of them (Baker, Hanna, and Denton 2020), this imposition can have
profound consequences and may cause people to feel like they have been
deprived of a crucial aspect of their autonomy (Hanna et al. 2020).

We consider denying people the ability to self-identify to be a type of
individual harm—that is, a harm that affects particular individuals as opposed
to the standings of particular social groups. However, patterns in the ways
that image tagging systems deny people the ability to self-identify can cause
each of the four types of representational harms that we described above, as
well as allocational harms. First, it reifies the belief that social groups
can be determined from visual appearance. Second, the visual characteristics
that image tagging systems rely on when tagging people as members of social
groups can perpetuate stereotypes. Third, imposing tags that indicate
membership in social groups onto people can be demeaning—for example by
misgendering them or by accurately applying tags (e.g., the tag women) to some
social groups (e.g., white women) but not others (e.g., Black women). Fourth,
if the available tags exclude particular social groups (e.g., people who are
non-binary), those groups will be erased. Finally, imposing tags that indicate
membership in social groups onto people may increase their visibility and thus
their vulnerability to violence, surveillance, loss of employment, and other
harms.

## Mapping Harms to Measurement Approaches

Figure 1: Examples of how computational measurement approaches belonging to
each of the five categories described earlier might be used to measure each of
the four different types of representational harms that can be caused by image
tagging systems. This mapping is by no means exhaustive.

In this section, we consider how computational measurement approaches
belonging to each of the five categories described earlier map to each of the
four types of representational harms identified in the previous section,
demonstrating that there is not a one-to-one mapping. Due to space
constraints, we only provide an in-depth discussion for stereotyping, but
include examples for the other types in figure 1.

### Stereotyping Social Groups

Stereotyping can be measured using many different computational approaches.
One possibility is to use an incidence-based approach to test, for example,
whether pre-identified tags that contain stereotyping language, such as
_boyish_ or _girly_ , are applied to any images. Another possibility is to use
a distribution-based approach to see whether any differences in the
distributions of tags for images that depict different social groups
correspond to stereotypes—for example, whether tags that refer to professions
are more frequently applied to images that depict men, while tags that refer
to appearance are more frequently applied to women (Schwemmer et al. 2020). A
third possibility is to use a performance-based approach to compare, for
example, the accuracy with which the tag _doctor_ is applied to images that
depict men to the accuracy with which it is applied to images that depict
people of other genders. A fourth possibility is to use a perturbation-based
approach to test for invariance in the tags applied to pairs of images that
are semantically equivalent, such as similarly composed photos of CEOs of
different genders. A final possibility is to use an internals-based approach
to investigate whether the regions of an image that are most salient when, for
example, incorrectly applying the tag _man_ to an image that depicts a woman
correspond to stereotypes about gender presentation.

### Implications of these Mappings

As demonstrated by the exercise of mapping different computational measurement
approaches to different types of representational harms, no single measurement
approach is definitive. Different facets of each type of harm will be revealed
by different approaches (Jacobs and Wallach 2021). As a result, measuring
representational harms does not involve identifying and then using the single
best measurement approach. Rather, many different approaches should be used,
with the resulting measurements interpreted in the context of what those
approaches are capable of revealing. The exercise also demonstrates that
without explicit reasoning, it is not possible to infer from the use of a
particular measurement approach which type of harm was intended to be
measured. We therefore recommend explicitly stating the type of harm of
interest so as to avoid any ambiguity or confusion. In concurrent work, we
provide a concrete demonstration of how to go about this type of analysis,
using a variety of computational approaches to measure different types of
representational harms that can be caused by image captioning systems (Wang et
al. 2022).

## Tensions When Mitigating Harms

Lastly, equipped with a more granular understanding of the types of
representational harms that can be caused by image tagging systems, we explore
tensions between attempts to mitigate some of these types of harms—and what
these tensions might mean for developing appropriate mitigations.

The most obvious way to mitigate reifying social groups is simply to remove
tags that indicate membership in social groups. By removing such tags, image
tagging systems can avoid enforcing boundaries between social groups and allow
members of social groups remain unmarked, decreasing their visibility and thus
their vulnerability to violence, surveillance, loss of employment, and other
harms (Hofmann et al. 2020). In contrast, the most obvious way to mitigate
erasing social groups is make sure that tags that indicate membership in
social groups exist and are correctly applied. In some cases, marking social
groups—that is, calling them out by name—can bring much needed recognition of
the injustices they have suffered. In other cases, marking privileged social
groups that often go unmarked (e.g., men) can provide a way to draw attention
to their privilege.

As another example, removing tags that refer to attributes, artifacts, or
activities that are bound up with the identities of social groups can mitigate
concerns that such tags might be applied incorrectly, thereby demeaning
particular social groups. However, removing tags associated with particular
social groups can instead cause those groups to be erased.

In some scenarios, particular types of images are deemed to be inherently
problematic—for example, pornographic images that stereotype and demean
women—meaning that they should not have any tags applied to them. However,
such images cannot be ignored or removed unless they are first identified,
usually by some computer vision system explicitly trained for that purpose.
Moreover, systems for ignoring or removing particular types of images can
erase social groups by incorrectly labeling images that are associated with
those groups as problematic. For example, when Tumblr banned “adult content,”
they also ended up removing posts from queer communities and sex-positive
communities (Pilipets and Paasonen 2020; Bronstein 2020).

As a final example, differences in the distributions of tags for images that
depict different social groups often correspond to stereotypes. One
straightforward way to mitigate these stereotypes is to ensure that the
distributions of tags for different social groups are similar—for example,
that the distribution of tags applied to images of Black football players is
similar to the distribution of tags applied to white football players (Rhue
2018). However, constraining the distributions of tags for different social
groups can be seen as a denial of the role that membership in particular
social groups plays in people’s lived experiences—a form of erasure.

These tensions suggest that developing appropriate mitigations is a complex
task, with many considerations. However, in many cases, the scenario in which
an image tagging system is intended to be used can (at least partially)
determine the appropriateness of different mitigations. For example, the
developers of an image tagging system for a social media platform might remove
particular tags to avoid stereotyping or demeaning particular social groups.
In contrast, the developers of an image tagging system for blind and low-
vision users might decide that the consequences of erasing social groups are
worse than the consequences of incorrectly applying tags (Bennett et al. 2021;
Hanley et al. 2021).

## Conclusion

In this paper, we argued that generic concerns about “unfairness,” “bias,” or
“discrimination” in the context of image tagging should instead be understood
in terms of the reproduction of harmful social hierarchies—that is,
representational harms. We analyzed previously proposed computational
approaches for measuring the ”fairness” of image tagging systems, finding that
they cluster into five distinct categories, each with its own analytic
foundation. We also identified four types of representational harms that can
be caused by image tagging systems, delineating how each one reproduces
harmful social hierarchies. By considering how different computational
measurement approaches map to each of these types of harms, we demonstrated
that each approach can be used to measure all four types, with different
facets of each type revealed by different approaches. Lastly, we showed that
attempts to mitigate different types of representational harms may be in
tension with one another and explored what these tensions might mean for
developing appropriate mitigations. Moving forward, we hope that this paper
contributes to more precise and actionable discussions of some of the harms
caused by image tagging systems.

## References

  * Algorithmic Justice League (2021) Algorithmic Justice League. 2021.  Drag Vs AI Workshop.
  * Alvi, Zisserman, and Nellaker (2018) Alvi, M.; Zisserman, A.; and Nellaker, C. 2018.  Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings.  In _Workshop on Bias Estimation in Face Analytics at ECCV 2018_.
  * Baker, Hanna, and Denton (2020) Baker, D.; Hanna, A.; and Denton, E. 2020.  Algorithmically Encoded Identities: Reframing Human Classification.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20, 681. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450369367.
  * Bard (2020) Bard, J. 2020.  Developing a Legal Framework for Regulating Emotion AI.  In _University of Florida Levin College of Law Research Paper_.
  * Barlas et al. (2021) Barlas, P.; Kyriakou, K.; Guest, O.; Kleanthous, S.; and Otterbacher, J. 2021.  To ”See” is to Stereotype: Image Tagging Algorithms, Gender Recognition, and the Accuracy-Fairness Trade-off.  _Proceedings of the ACM on Human-Computer Interaction_ , 4(CSCW3): 1–31.
  * Barlas et al. (2019a) Barlas, P.; Kyriakou, K.; Kleanthous, S.; and Otterbacher, J. 2019a.  Social B(eye)as: Human and Machine Descriptions of People Images.  In _Proceedings of the International AAAI Conference on Web and Social Media_.
  * Barlas et al. (2019b) Barlas, P.; Kyriakou, K.; Kleanthous, S.; and Otterbacher, J. 2019b.  What Makes an Image Tagger Fair? Proprietary Auto-tagging and Interpretations on People Images.  In _Proceedings of the 27th ACM Conference On User Modelling, Adaptation And Personalization (UMAP)_.
  * Barocas et al. (2017) Barocas, S.; Crawford, K.; Shapiro, A.; and Wallach, H. 2017.  The Problem With Bias: Allocative Versus Representational Harms in Machine Learning.  In _Proceedings of SIGCIS_. Philadelphia, PA.
  * Benjamin (2019) Benjamin, R. 2019.  _Race After Technology: Abolitionist Tools for the New Jim Code_.  John Wiley & Sons.
  * Bennett et al. (2021) Bennett, C. L.; Gleason, C.; Scheuerman, M. K.; Bigham, J. P.; Guo, A.; and To, A. 2021.  “It’s Complicated”: Negotiating Accessibility and (Mis) Representation in Image Descriptions of Race, Gender, and Disability.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 1–19.
  * Bhargava and Forsyth (2019) Bhargava, S.; and Forsyth, D. 2019.  Exposing and correcting the gender bias in image captioning datasets and models.  _arXiv preprint arXiv:1912.00578_.
  * Blodgett et al. (2020) Blodgett, S. L.; Barocas, S.; Daumé, H., III; and Wallach, H. 2020.  Language (Technology) is Power: A Critical Survey of “Bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , 5454–5476.
  * Bronstein (2020) Bronstein, C. 2020.  Pornography, Trans Visibility, and the Demise of Tumblr.  In _Transgender Studies Quarterly_.
  * Buolamwini and Gebru (2018) Buolamwini, J.; and Gebru, T. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. New York City, NY.
  * Crawford (2017) Crawford, K. 2017.  The Trouble with Bias.  _Conference on Neural Information Processing Systems Keynote_.
  * Crawford and Paglen (2019) Crawford, K.; and Paglen, T. 2019.  Excavating AI: the politics of images in machine learning training sets.  _Excavating AI_.
  * Das, Dantcheva, and Bremond (2018) Das, A.; Dantcheva, A.; and Bremond, F. 2018.  Mitigating Bias in Gender, Age and Ethnicity Classification: a Multi-Task Convolution Neural Network Approach.  In _Workshop at ECCV 2018_.
  * Denton et al. (2021) Denton, E.; Hutchinson, B.; Mitchell, M.; Gebru, T.; and Zaldivar, A. 2021.  Image Counterfactual Sensitivity Analysis for Detecting Unintended Bias.  _arXiv preprint arXiv:1906.06439_.
  * DeVries et al. (2019) DeVries, T.; Misra, I.; Wang, C.; and van der Maaten, L. 2019.  Does Object Recognition Work for Everyone?  arXiv:1906.02659.
  * Hanley et al. (2021) Hanley, M.; Barocas, S.; Levy, K.; Azenkot, S.; and Nissenbaum, H. 2021.  Computer vision and conflicting values: Describing people with automated alt text.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , 543–554.
  * Hanna et al. (2020) Hanna, A.; Denton, E.; Smart, A.; and Smith-Loud, J. 2020.  Towards a Critical Race Methodology in Algorithmic Fairness.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , 501–512.
  * Hendricks et al. (2018) Hendricks, L. A.; Burns, K.; Saenko, K.; Darrell, T.; and Rohrbach, A. 2018.  Women also snowboard: Overcoming bias in captioning models.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 771–787.
  * Hofmann et al. (2020) Hofmann, M.; Kasnitz, D.; Mankoff, J.; and Bennett, C. L. 2020.  Living Disability Theory: Reflections on Access, Research, and Design.  In _The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS)_.
  * Hu and Strout (2018) Hu, Z.; and Strout, J. 2018.  Exploring Stereotypes and Biased Data with the Crowd.  In _arXiv:1801.03261_.
  * Jacobs and Wallach (2021) Jacobs, A. Z.; and Wallach, H. 2021.  Measurement and fairness.  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , 375–385.
  * Jia, Lansdall-Welfare, and Cristianini (2018) Jia, S.; Lansdall-Welfare, T.; and Cristianini, N. 2018.  Right for the Right Reason: Training Agnostic Networks.  In _Lecture Notes in Computer Science_.
  * Kay, Matuszek, and Munson (2015a) Kay, M.; Matuszek, C.; and Munson, S. A. 2015a.  Unequal Representation and Gender Stereotypes in Image Search Results for Occupations.  In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , CHI ’15, 3819–3828. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450331456.
  * Kay, Matuszek, and Munson (2015b) Kay, M.; Matuszek, C.; and Munson, S. A. 2015b.  Unequal Representation and Gender Stereotypes in Image Search Results for Occupations.  In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , 3819–3828.
  * Keyes (2018) Keyes, O. 2018.  The misgendering machines: Trans/HCI implications of automatic gender recognition.  _Proceedings of the ACM on human-computer interaction_ , 2(CSCW): 1–22.
  * Khiyari and Wechsler (2016) Khiyari, H. E.; and Wechsler, H. 2016.  Face Verification Subject to Varying (Age, Ethnicity, and Gender)Demographics Using Deep Learning.  _Journal of biometrics & biostatistics_, 7: 1–5.
  * Klare et al. (2012) Klare, B. F.; Burge, M. J.; Klontz, J. C.; Vorder Bruegge, R. W.; and Jain, A. K. 2012.  Face Recognition Performance: Role of Demographic Information.  _IEEE Transactions on Information Forensics and Security_ , 7(6): 1789–1801.
  * Kyriakou et al. (2019) Kyriakou, K.; Barlas, P.; Kleanthous, S.; and Otterbacher, J. 2019.  Fairness in Proprietary Image Tagging Algorithms: A Cross-Platform Audit on People Images.  In _Proceedings of the International AAAI Conference on Web and Social Media_.
  * Kärkkäinen and Joo (2021) Kärkkäinen, K.; and Joo, J. 2021.  FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_.
  * Levin (2016) Levin, S. 2016.  A beauty contest was judged by AI and the robots didn’t like dark skin.  _The Guardian_.
  * McDuff et al. (2019) McDuff, D.; Ma, S.; Song, Y.; and Kapoor, A. 2019.  Characterizing Bias in Classifiers using Generative Models.  In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Miceli, Schuessler, and Yang (2020) Miceli, M.; Schuessler, M.; and Yang, T. 2020.  Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision.  In _ACM Conference on Computer Supported Cooperative Work (CSCW)_.
  * Muthukumar et al. (2018) Muthukumar, V.; Pedapati, T.; Ratha, N.; Sattigeri, P.; Wu, C.-W.; Kingsbury, B.; Kumar, A.; Thomas, S.; Mojsilovic, A.; and Varshney, K. R. 2018.  Understanding Unequal Gender Classification Accuracy from Face Images.  In _arXiv:1812.00099_.
  * Noble (2018) Noble, S. U. 2018.  _Algorithms of Oppression: How Search Engines Reinforce Racism_.  NYU Press.
  * Offert and Bell (2020) Offert, F.; and Bell, P. 2020.  Perceptual bias and technical metapictures: critical machine vision as a humanities challenge.  In _AI & Society_.
  * Otterbacher (2018) Otterbacher, J. 2018.  Social Cues, Social Biases: Stereotypes in Annotations on People Images.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_.
  * Otterbacher et al. (2019) Otterbacher, J.; Barlas, P.; Kleanthous, S.; and Kyriakou, K. 2019.  How Do We Talk about Other People? Group (Un)Fairness in Natural Language Image Descriptions.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_.
  * Pilipets and Paasonen (2020) Pilipets, E.; and Paasonen, S. 2020.  Nipples, memes, and algorithmic failure: NSFW critique of Tumblr censorship.  In _New Media & Society_.
  * Prabhu and Birhane (2020) Prabhu, V. U.; and Birhane, A. 2020.  Large image datasets: A pyrrhic win for computer vision?  arXiv:2006.16923.
  * Rhue (2018) Rhue, L. 2018.  Racial Influence on Automated Perceptions of Emotions.  _SSRN Electronic Journal_.
  * Roberts (2018) Roberts, C. V. 2018.  Quantifying the Extent to which Popular Pre-Trained Convolutional Neural Networks Implicitly Learn High-Level Protected Attributes.  In _Princeton Masters Thesis_.
  * Rodden (2017) Rodden, K. 2017.  Is that a boy or a girl? Exploring a neural network’s construction of gender.  _Medium_.
  * Scheuerman, Paul, and Brubaker (2019) Scheuerman, M. K.; Paul, J. M.; and Brubaker, J. R. 2019.  How Computers See Gender: An Evaluation of Gender Classification in Commercial Facial Analysis Services.  _Proc. ACM Hum.-Comput. Interact._ , 3(CSCW).
  * Scheuerman et al. (2020) Scheuerman, M. K.; Wade, K.; Lustig, C.; and Brubaker, J. R. 2020.  How We’ve Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis.  _Proc. ACM Hum.-Comput. Interact._ , 4(CSCW1).
  * Schwemmer et al. (2020) Schwemmer, C.; Knight, C.; Bello-Pardo, E. D.; Oklobdzija, S.; Schoonvelde, M.; and Lockhart, J. W. 2020.  Diagnosing gender bias in image recognition systems.  _Socius_ , 6: 2378023120967171.
  * Sedenberg and Chuang (2017) Sedenberg, E.; and Chuang, J. 2017.  Smile for the Camera: Privacy and Policy Implications of Emotion AI.  arXiv:1709.00396.
  * Selbst and Barocas (2023) Selbst, A.; and Barocas, S. 2023.  Unfair Artificial Intelligence: How FTC Intervention Can Overcome the Limitations of Discrimination Law.  _University of Pennsylvania Law Review_ , 171.
  * Shankar et al. (2017a) Shankar, S.; Halpern, Y.; Breck, E.; Atwood, J.; Wilson, J.; and Sculley, D. 2017a.  No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World.  arXiv:1711.08536.
  * Shankar et al. (2017b) Shankar, S.; Halpern, Y.; Breck, E.; Atwood, J.; Wilson, J.; and Sculley, D. 2017b.  No classification without representation: Assessing geodiversity issues in open data sets for the developing world.  In _Proceedings of the NIPS 2017 Workshop on Machine Learning for the Developing World_.
  * Simonite (2018) Simonite, T. 2018.  When It Comes to Gorillas, Google Photos Remains Blind.  _Wired, January_.
  * Singh et al. (2020) Singh, K. K.; Mahajan, D.; Grauman, K.; Lee, Y. J.; Feiszli, M.; and Ghadiyaram, D. 2020.  Don’t Judge an Object by Its Context: Learning to Overcome Contextual Bias.  arXiv:2001.03152.
  * Song and Shmatikov (2020) Song, C.; and Shmatikov, V. 2020.  Overlearning Reveals Sensitive Attributes.  arXiv:1905.11742.
  * Steed and Caliskan (2021) Steed, R.; and Caliskan, A. 2021.  Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, 701–713. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450383097.
  * Stock and Cisse (2018) Stock, P.; and Cisse, M. 2018.  ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases.  arXiv:1711.11443.
  * Wang et al. (2022) Wang, A.; Barocas, S.; Laird, K.; and Wallach, H. 2022.  Measuring Representational Harms in Image Captioning.  _ACM Conference on Fairness, Accountability, and Transparency (FAccT)_.
  * Wang et al. (2021) Wang, A.; Liu, A.; Zhang, R.; Kleiman, A.; Kim, L.; Zhao, D.; Shirai, I.; Narayanan, A.; and Russakovsky, O. 2021.  REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets.  arXiv:2004.07999.
  * Wang and Russakovsky (2021) Wang, A.; and Russakovsky, O. 2021.  Directional Bias Amplification.  arXiv:2102.12594.
  * Wang et al. (2019) Wang, T.; Zhao, J.; Yatskar, M.; Chang, K.-W.; and Ordonez, V. 2019.  Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations.  arXiv:1811.08489.
  * Wang et al. (2020) Wang, Z.; Qinami, K.; Karakozis, I. C.; Genova, K.; Nair, P.; Hata, K.; and Russakovsky, O. 2020.  Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation.  arXiv:1911.11834.
  * Wilson, Hoffman, and Morgenstern (2019) Wilson, B.; Hoffman, J.; and Morgenstern, J. 2019.  Predictive Inequity in Object Detection.  arXiv:1902.11097.
  * Wu et al. (2017) Wu, S.; Wieland, J.; Farivar, O.; and Schiller, J. 2017.  Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Network Service.  _ACM Conference on Computer-Supported Cooperative Work And Social Computing (CSCW)_.
  * Yang et al. (2020) Yang, K.; Qinami, K.; Fei-Fei, L.; Deng, J.; and Russakovsky, O. 2020.  Towards fairer datasets.  _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_.
  * Zhao et al. (2017) Zhao, J.; Wang, T.; Yatskar, M.; Ordonez, V.; and Chang, K.-W. 2017.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2979–2989.