# “If I Had All the Time in the World”: Ophthalmologists’ Perceptions of
Anchoring Bias Mitigation in Clinical AI Support

Anne Kathrine Petersen Bach<EMAIL_ADDRESS>Aalborg
UniversityAalborgDenmark ,  Trine Munch Nørgaard<EMAIL_ADDRESS>Aalborg UniversityAalborgDenmark ,  Jens Christian Brok
<EMAIL_ADDRESS>Aalborg UniversityAalborgDenmark and  Niels van Berkel
<EMAIL_ADDRESS>0000-0001-5106-7692 Aalborg UniversityAalborgDenmark

(2023)

###### Abstract.

Clinical needs and technological advances have resulted in increased use of
Artificial Intelligence (AI) in clinical decision support. However, such
support can introduce new and amplify existing cognitive biases. Through
contextual inquiry and interviews, we set out to understand the use of an
existing AI support system by ophthalmologists. We identified concerns
regarding anchoring bias and a misunderstanding of the AI’s capabilities.
Following, we evaluated clinicians’ perceptions of three bias mitigation
strategies as integrated into their existing decision support system. While
clinicians recognised the danger of anchoring bias, we identified a concern
around the impact of bias mitigation on procedure time. Our participants were
divided in their expectations of any positive impact on diagnostic accuracy,
stemming from varying reliance on the decision support. Our results provide
insights into the challenges of integrating bias mitigation into AI decision
support.

Cognitive bias, anchoring bias, decision support, artificial intelligence, AI
support, bias mitigation, CDSS, DSS, ophthalmology

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
2023 CHI Conference on Human Factors in Computing Systems; April 23–28, 2023;
Hamburg, Germany††booktitle: Proceedings of the 2023 CHI Conference on Human
Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg,
Germany††price: 15.00††doi: 10.1145/3544548.3581513††isbn:
978-1-4503-9421-5/23/04††ccs: Human-centered computing Human computer
interaction (HCI)††ccs: Human-centered computing Collaborative and social
computing††ccs: Applied computing Health care information systems

##  1\. Introduction

Healthcare specialists increasingly rely on Clinical Decision Support Systems
(CDSS) to interpret patient symptoms and determine treatment. In this work, we
focus on Artificial Intelligence (AI)-powered CDSS in the context of
ophthalmology (i.e., clinical care specialising in eye and vision care). In
Denmark, where our study took place, ophthalmologists experience a burdensome
workload (Albinus, 2021). Data from 2018 showed that the country’s 383
ophthalmologists were responsible for close to 700.000 patients (of Health and
the Elderly), 2020; Albinus, 2021). To help alleviate the ophthalmologist’s
workload and decrease the impact of human error, AI decision support is
increasingly integrated into the decision-making process (Kapoor et al.,
2019). Specifically, the Department of Ophthalmology at Aalborg University
Hospital uses an AI system to detect diabetic retinopathy (DR). DR is an eye
disease caused by high blood sugar damaging the eye’s retina and can result in
blindness if not diagnosed and treated timely (Gregori, 2021). The detection
of DR and many other eye diseases is done through visual inspection of the
retinal fundus and optical coherence tomography (OCT) images. Fundus images
show the eye’s interior surface, whereas OCT provides a cross-sectional image
of the retina (see Figure 1).

Figure 1. Imaging device in use at an ophthalmology clinic (left) and the
resulting OCT and fundus images (right).

Two photos. On the left, a person puts their chin on a rest while looking into
a machine for scanning their eyes. On the right, a computer screen shows the
resulting OCT and fundus image for analysis, a joystick is visible in front of
the screen.

With the increasing use of AI support systems, prior work in HCI and beyond
has begun to assess the impact of such systems in real-world use. In this
study, we set out to qualitatively investigate the role of anchoring bias in
clinical practice as introduced by AI support systems. Anchoring bias is a
widely established cognitive bias where a person’s decisions are affected by a
previously observed or experienced reference point. Tversky & Kahneman
describe how “people make estimates by starting from an initial value that is
adjusted to yield the final answer” (Tversky and Kahneman, 1974), with any
adjustments to this initial value typically being insufficient to compensate
for the error of the initial reference point (Tversky and Kahneman, 1974). In
clinical decision-making, anchoring bias can result in severe consequences.
While it is challenging to quantify the effect of anchoring bias on clinical
outcomes precisely, numerous case studies have highlighted the real-world
implications of this bias (Horowitz et al., 2021; Ogdie et al., 2012; Vickrey
et al., 2010). As a timely example, Horowitz et al. describe how two febrile
patients, seen amidst the COVID-19 pandemic, were misdiagnosed as having
COVID-19 instead of their tick-borne infections due to anchoring bias induced
by the many encounters with patients (Horowitz et al., 2021). Ogdie et al.
surveyed 41 medical residents on their prior diagnostic errors, with 88% of
respondents pointing to anchoring bias as the most common cognitive bias
(Ogdie et al., 2012). While AI-based image recognition has shown remarkable
performance in diagnosis (Kruse and Ehrbar, 2020; Tao et al., 2020), adopting
these systems in clinical practice often results in numerous challenges
(Cabitza and Zeitoun, 2019). Prior work has sought to address concerns
regarding the equity and fairness of recommendations (Wang et al., 2020), the
presentation of relevant and timely information (Cai et al., 2019a; van Berkel
et al., 2022), and alignment with clinical workflows (Cai et al., 2019b;
Piorkowski et al., 2021). While these are all critical aspects of designing
and using AI systems, the role of cognitive biases—and in particular end-
users’ perceptions of techniques to combat these biases—has remained
underexplored.

To better understand clinicians’ use of AI-powered CDSS, we conducted an
initial study of ophthalmologists’ screening processes totalling seven
interviews and three contextual inquiry sessions. Specifically, we aimed to
investigate how our participants use the CDSS, how it affects their screening
process, and understand their motivations for (not) using the CDSS. While the
presentation of the AI’s assessment (annotation of patient images) increases
clinicians’ efficiency, it is also perceived as deceiving as the image
analysis focuses solely on DR, thereby excluding conditions such as glaucoma
or tumours. Using our newfound understanding of the clinicians’ use of the
CDSS, we designed an interactive prototype which integrates three bias
mitigation strategies. These established bias mitigation strategies—‘hear the
story first’ (Lighthall and Vazquez-Guillamet, 2015), ‘decision justification’
(Isler et al., 2020; Lambe et al., 2016), and ‘consider the opposite’
(Mussweiler et al., 2000; Adame, 2016)—were designed to reduce clinicians’
anchoring bias and to boost accuracy. We evaluated our prototype with six
ophthalmologists to understand their perceptions of our proposed ‘debiasing
workflow’.

Our findings emphasise conflicting perspectives on the potential effect of
bias mitigation on diagnostic accuracy, with some participants believing that
their diagnostic accuracy could not be further enhanced. Participants
generally highlighted the potential of bias mitigation techniques to keep them
focused throughout the workday. However, concerns about decreasing efficiency
due to these bias mitigation efforts were also common. Participants perceived
the value of not only the CDSS, but also of the bias mitigation strategies to
increase with more complex cases. This highlights not only clinicians’
increased need for support in complicated cases, but also their desire to
double-check their intuition.

Our results provide an understanding of clinicians’ perceptions towards
incorporating bias mitigation in AI-based CDSS. We note that best practices
from the bias mitigation literature can be opposite to clinicians’ real-world
challenges. In particular, the time pressure clinicians face limits the appeal
of mitigation strategies which require user input. As AI systems are
increasingly deployed into the real world, critically assessing the cognitive
biases introduced by their use is vital.

##  2\. Related work

Amidst an increase in the interest and use of AI-driven decision support
tools, cognitive biases due to these systems have become an area of concern
within HCI (Echterhoff et al., 2022; Nourani et al., 2021; Wang et al., 2019;
Zhang et al., 2015; van Berkel et al., 2023). Here, we discuss related work
across three key topics. First, we assess the literature on cognitive biases
in clinical work, focusing on anchoring bias due to its relevance for decision
support tools. Second, we outline prior work that has aimed to mitigate either
the onset or effect of cognitive biases. Third, we present prior work on AI-
supported CDSS, highlighting the challenges in deploying these systems in the
real world and recent technological advances within CDSS for ophthalmology.

###  2.1. Cognitive Biases in Clinical Work

A cognitive bias is typically defined as a systematic error in our thinking
(Lilienfeld and Lynn, 2014), i.e., an error that is often unconsciously
ingrained in our believes. These biases are common in everyday life, and
research shows no clear relation between intelligence and propensity to
cognitive biases (Teovanović et al., 2015). In the clinical world, decisions
are often made under time pressure and with a large degree of uncertainty,
increasing the chance for cognitive biases to emerge. As cognitive biases can
result in flawed decision making, they can have a severe impact on patient
health. According to Croskerry, over a hundred different biases have been
described which affect clinical decision making (Croskerry, 2013).

In this paper, we focus on anchoring bias due to the contingency of this
cognitive bias arising when faced with system-driven recommendations
(Echterhoff et al., 2022; Nourani et al., 2021). Anchoring bias is described
as “the disproportionate influence on decision makers to make judgements that
are biased toward an initially presented value” (Tversky and Kahneman, 1974).
This initial reference point subsequently skews our final decision outcome.
Lieder et al. argue that anchoring bias is a reflection of people’s rational
use of their limited time and cognitive resources (Lieder et al., 2018). In a
literature review of over 40 years of research on anchoring bias, Furnham and
Boo find that this bias occurs in a wide range of domains, both as studied
under controlled conditions in the laboratory and in real-world decision-
making (Furnham and Boo, 2011). Within the medical domain, anchoring bias has
been identified as the most common cognitive bias in a survey among 41 medical
residents, with a large majority of 88% respondents having encountered this
bias (Ogdie et al., 2012). Royce et al. summarise case studies of misdiagnosis
and state that “errors arising from cognitive bias play a role in over 50% of
identified cases of diagnostic error in ambulatory clinics and in up to 83% of
cases involving physician-reported diagnostic errors”. Despite this, Croskerry
highlights that decision makers are often unaware of their biases, with many
clinicians under-appreciating the role of cognitive biases in diagnostic error
(Croskerry, 2013). This furthermore complicates the use of debiasing
strategies.

Anchoring bias is particularly relevant in the context of clinical AI support
systems, as the AI recommendations act as anchors in decision-making processes
(Rastogi et al., 2022). Hence, in this study, we investigate ophthalmologists’
perceptions towards the inclusion of bias mitigation strategies into their AI-
supported CDSS.

###  2.2. Mitigating Cognitive Biases

Given the widespread occurrence of cognitive biases (Tversky and Kahneman,
1974) and the effect of task context on the suitability of specific bias
mitigation strategies (van Berkel et al., 2023), the literature has explored a
variety of methods and techniques for overcoming these biases. The use of
educational strategies for recognising and subsequently addressing cognitive
biases has often been suggested, although the effect of these strategies has
been inconclusive (Norman et al., 2017). For example, Sherbino et al.
conducted a controlled trial to investigate the effect of cognitive forcing
strategies training on medical students (Sherbino et al., 2014). Their results
showed no reduction in diagnostic error of those having completed the training
as compared to the control group. Royce et al. argue that studies on the
effect of teaching clinical reasoning skills to reduce cognitive biases are
affected by methodological problems which affect their effectiveness (Royce et
al., 2019). In their critique, Royce et al. point to limitations in terms of
cognitive biases studied, low levels of applicability to real-world clinical
reasoning and decision making, and low levels of ecological validity (Royce et
al., 2019).

Within HCI, different works have explored how changes to the interface and
overall interaction may serve to mitigate cognitive biases. Highly relevant to
our work is a recent study by Fogliato et al. (Fogliato et al., 2022). In a
study involving 19 radiologists, Fogliato et al. presented participants with
an image analysis task. Participants were shown X-ray images either
simultaneously with an AI inference or were shown the AI inference after
submitting their provisional diagnosis of the X-ray image. Their results show
that the radiologists who registered their provisional responses before being
shown the AI inference were less likely to agree with the AI and perceived its
assessments as less helpful. Task completion time did not differ between the
two conditions. Zhang et al. studied two cognitive biases in the context of a
decision support system (Zhang et al., 2015), ‘conservatism’ and ‘loss
aversion’. Conservatism describes the behaviour in which we fail to adjust our
beliefs when presented with new information, instead choosing to embrace the
original information presented to us. Loss aversion describes our
disproportionate preference for avoiding losses over acquiring comparable
gains. With the aim of minimising these biases, Zhang et al. introduce two
debiasing techniques; a model bootstrapped on prior suggestions and presenting
the expected return for the various decision options. Their results showed
that the two studied biases have a large impact on participants’ decision
making, with the evaluated debiasing techniques being only partly effective.
Solomon introduces and studies ‘customisation bias’ in the context of decision
support systems (Solomon, 2014). Using a fantasy baseball game as an
experimental design, participants were led to believe that they could
customise the recommendations provided by the decision support system to
favour specific baseball-related aspects (e.g., batting average). The results
of this study subsequently showed that participants who were under the
impression that they had customised the recommendation system were more likely
to follow its recommendations, regardless of the accuracy of these
recommendations (Solomon, 2014).

Cognitive biases have also been found and studied outside the context of
decision support systems. In a study on information search, Rieger et al.
sought to mitigate confirmation bias by obfuscating attitude-confirming search
results (i.e., results which are in line with a user’s existing attitude)
(Rieger et al., 2021). Their results showed that by targeted obfuscation of
moderate and extreme attitude-confirming search results through means of a
warning message, participants were less likely to engage with this type of
result. However, the authors warn that this effect might also be impacted by
participants’ desire to avoid additional effort. In a study on conversational
agents, Santhanam et al. found that participants who were presented with a
textual or numerical anchor prior to rating the conversation agent were more
likely to provide a rating that is closer to the value of the anchor presented
(Santhanam et al., 2020). This highlights that cognitive biases, such as
anchoring bias, may also affect the outcomes of evaluation studies in HCI.
Recent work by Dingler et al. explores a method for surfacing implicit
cognitive biases, in particular implicit associations (Dingler et al., 2022).
Measuring participants’ responses to (visual) stimuli, known as the implicit
association test (Sriram and Greenwald, 2009), enables researchers to measure
people’s leanings on polarising topics.

These collective results highlight the challenges in mitigating cognitive
biases, with no ‘silver bullet’ solution in sight. In light of the long-term
goal of mitigating cognitive biases in real-world clinical contexts, we seek
to understand end-users’ perceptions of bias mitigation strategies.

###  2.3. AI-supported CDSS

Recent years have seen a rise of AI-powered decision support in the medical
domain (Jiang et al., 2017). Spanning a variety of computational techniques
(e.g., image recognition) and application areas (e.g., medical imaging,
surgery), these applications typically aim to assist clinicians in diagnosis
or treatment. Despite promising results in terms of these system’s technical
capabilities (Jiang et al., 2017), real-world deployments often reveal and
introduce a wide range of challenges that inhibit or limit the system’s
capabilities (Wang et al., 2021; Molin et al., 2016; Lee et al., 2021). For
example, Wang et al. studied the deployment of an AI-based decision support
system in a rural context (Wang et al., 2021). Their findings highlight how
the high workload of clinical staff makes it practically impossible to enter
the required information into the CDSS. Wang et al. conclude that while the AI
system might be technically competent, it is designed to support a ‘textbook’
workflow rather than clinical reality. Similarly, in a study on CDSS in the
heart pump implant decision process, Yang et al. found that current decision
support systems lack both clinician’s trust and perceived need (Yang et al.,
2016). The authors point to the CDSS’s sole focus on the final decision making
point, whereas decision support could prove more beneficial when presented
across the patients’ entire healthcare trajectory.

In a study on the integration of automation in digital pathology, Molin et al.
study the reflections of pathologists following the introduction of a
partially automated digital workstation (Molin et al., 2016). One of their
findings highlights how a lack of transparency in the inner workings of
algorithms can result in end-users making incorrect assumptions about either
the algorithm or the source material analysed (Molin et al., 2016). Cai et al.
presented an investigation in the same domain, in which they focus on the user
interface used by pathologists (Cai et al., 2019a). By providing pathologists
with the ability to refine AI suggestions based on standard medical concepts
rather than merely the visual similarity of the image, Cai et al. were able to
increase both the perceived diagnostic utility of the tool and end-user trust.
Jo et al. similarly sought to improve the alignment between clinicians and a
CDSS tool, focusing on the planning of discontinuation of psychiatric drugs
(Jo et al., 2022). Following a user-centred approach, their study identified a
need for incorporating interpersonal (e.g., health literacy) and
infrastructural constraints (e.g., insurance coverage of prescriptions),
aspects not previously considered in the CDSS. These examples highlight the
value of end-users’ agency in collaborating with CDSSs for technology
acceptance.

Finally, we highlight prior work focused on ophthalmology, in particular
diabetic retinopathy. Diagnosis in ophthalmology is based mainly on the
analysis of medical images, a task that is time-consuming, strenuous, and
error-prone (Kapoor et al., 2019; Nguyen et al., 2020). The image-centred
nature of ophthalmology aligns well with the pattern recognition abilities of
AI technology, with hopes of increased efficiency and quality when introduced
into clinical practice (Kapoor et al., 2019). Consequently, the use of AI for
the analysis of patients’ medical images has been explored for a variety of
eye-related conditions. Nguyen et al. present an automated classification
system for diabetic retinopathy (Nguyen et al., 2020), with other work
focusing on retinal anomalies such as macular edema (excessive fluids in the
centre of the retina) (Roy et al., 2017), exudates (fluid leaking out of blood
vessels) (Akram et al., 2014), and cotton-wool (localised reduced blood flow)
(Niemeijer et al., 2007), which are typically the result of a specific
condition. Despite these technological advances, a recent review on the
current state of AI in ophthalmology warns of existing limitations (Kapoor et
al., 2019), citing risks of clinicians’ deskilling, AI’s lack of understanding
of underlying clinical contexts, and a lack of transparency of algorithms’
underlying mechanisms. Beede et al. evaluate the use of an AI-support CDSS for
the detection of diabetic retinopathy in clinical practice (Beede et al.,
2020). Through interviews and observations across clinics in Thailand, the
authors identified high variability in clinical practice between the clinics
and a continuous organisational strain to deal with the extensive volume of
patients. Beede et al. subsequently studied the challenges introduced by the
deployment of the AI-supported CDSS. These focused on obtaining patient
consent, working around the study protocol to better align with patients’
needs, and clinical factors negatively affecting system performance. For
example, the authors point to the CDSS having stringent image quality
guidelines, disqualifying images as blurry which are still interpretable by
clinical staff (Beede et al., 2020).

Integrating AI support into clinical practice has remained a challenge, with a
broad call for further research on how CDSSs can be successfully implemented
into the clinical context (Wang et al., 2021; Gulshan et al., 2016). We build
upon this prior work by investigating ophthalmologists’ perceptions of bias
mitigation as integrated into their existing AI-supported CDSS.

##  3\. Detecting Diabetic Retinopathy

The Department of Ophthalmology at Aalborg University Hospital deals with all
kinds of eye-related diseases and ailments. The focus of our study is the
screening of DR. DR poses a severe threat to diabetes patients, and spans four
stages of severity. First, the illness begins with the emergence of
microaneurysms in the retina. Second, more severe changes occur in the retina,
such as haemorrhages, which can affect the patient’s vision. Third, scar
tissue forms, which can lead to retinal detachment—requiring immediate
surgery. Fourth, the disease evolves into diabetic maculopathy, where repeated
treatment is needed and severe vision loss may be inevitable (NHS, 2021).

Figure 2. Assessment of fundus images as presented in the AI-powered CDSS.
Yellow bars indicate that one to three lesions have been detected. Two black
outlines towards the centre and right-hand side of fundus image indicate
lesions.

Screenshot of AI-powered CDSS system. Centrally visible on the screen is a
fundus image of a patient’s eye. Thick yellow lines are visible on the left-
and right-hand sides of the fundus image. The rest of the interface shows
additional fundus images of the same patient.

Diabetes patients in Denmark are offered regular screenings to monitor DR-
related changes in their retinas. As these patients typically attend
screenings related to other potential complications, the hospital offers
patients to bundle all these appointments in one day. During this day,
patients go through various tests, including having images of their retina
taken. Instead of waiting several days or weeks to receive all test results,
patients receive all results by the end of the day. Specifically, using the
equipment shown in Figure 1, a nurse or bio-analyst creates an OCT scan and
takes five fundus images per eye. The retinal images and the results from the
various tests are collected and stored electronically. The ophthalmologist
assigned to screening duty for the day assesses each of the images from the
same-day screening as they appear in the system. As the patients wait at the
hospital to receive their screening results, the doctor finalises each
screening within one hour of receiving the images. Here, they look through the
patient’s medical record, their OCT and fundus images, and add their
assessment to the medical record. Depending on the result, the patient is
either called in for a consultation or scheduled for a subsequent screening.

In 2018, the hospital introduced an AI system to assist in fundus image
assessment, aiming to ease the doctors’ workload and increase the efficiency
of same-day screenings. This AI system is part of the screening software
currently used by the doctors, and is provided by a commercial company
operating across Europe. The system analyses fundus images and can detect
microaneurysms and haemorrhages, which are signs of DR. The use of such
automated systems for detecting DR lesions is increasingly common due to
rising patient numbers and healthcare costs, both in clinical settings (as
studied here) by opticians, and in primary care (Abràmoff et al., 2018). Goh
et al. provide an overview of commercially available systems (Goh et al.,
2016), which differ in functionality between detecting DR (as studied here) or
classifying cases as either ‘high’ or ‘low’ risk for DR and provide a
subsequent referral recommendation. The system we investigate integrates with
the hospital’s electronic patient record system. When an ophthalmologist opens
a patient’s fundus images, they are presented with ten unprocessed images
followed by the same ten images, but processed and labelled by the AI system.
The output presented to the doctors is shown in Figure 2. The left and right
sides of each image’s thumbnail are coloured either green, yellow, or red,
depending on the number of lesions detected. When an image is viewed in full
size, the colour label of that image lines its left and right edge. The system
generates a black outline that follows the shape of any detected lesions.

##  4\. Study 1 - Contextual understanding

Next, we elaborate on the process of our interviews and contextual inquiry, as
well as our use of reflexive thematic analysis. Our process of data gathering
was guided by the following questions:

  * •

How do ophthalmologists use the AI system?

  * •

Does the AI system affect ophthalmologists in their DR screening process? In
which ways?

  * •

To what degree do the ophthalmologists trust the output of the system?

  * •

What are the ophthalmologists’ motivations for using or not using the AI
system?

Our sample consisted of seven ophthalmologists, all between the ages of 35–50,
with their experience ranging from less than one year to 16 years (see Table
1). We prepared an initial interview guide, which we refined between
interviews to obtain the most valuable insights. Each interview lasted
approximately one hour, was audio recorded, and subsequently transcribed for
analysis.

Three of the seven ophthalmologists also participated in our contextual
inquiry (P1, P2, and P3). Contextual inquiry enabled us to observe the
doctors’ use of the CDSS in their natural surroundings. We thereby obtained a
rich understanding of the factors that influence their use of the CDSS.
Throughout the three contextual inquiry sessions, we continuously wrote notes,
paying attention to the participants’ comments and actions. Each session
lasted approximately four hours, during which the ophthalmologists carried out
their typical tasks and one of the paper’s authors occasionally asked
questions. In addition, all three ophthalmologists explained their work
process and showed us the tools with which they worked.

Participant | Primary job title | Sex | Age | Years of experience | Study 1 | Study 2
---|---|---|---|---|---|---
P1 | Chief physician | F | 47 | 14 |  |
P2 | Chief physician and surgeon | F | 50 | 16 |  |
P3 | Specialist | M | 37 | 4 |  |
P4 | Surgeon | F | 44 | 13 |  |
P5 | Specialist | M | 35 | 3.5 |  |
P6 | Senior registrar | M | 35 | 0.5 |  |
P7 | Senior registrar | M | 44 | 3.5 |  |
P8 | Chief physician and surgeon | M | 40 | 12 |  |
Table 1. Overview of the participating ophthalmologists across Study 1 and
Study 2.

We analysed our transcriptions and observation notes following Braun &
Clarke’s approach to reflexive thematic analysis (Braun et al., 2019).
Reflexive thematic analysis allows researchers to build on their existing
knowledge and cultural understanding of a domain. Precisely, we follow
reflexive thematic analysis to examine which factors affect the use of an AI-
enabled CDSS, focusing on the work processes and challenges of
opthalmologists. Two of the paper’s authors started by separately conducting
an initial analysis of two of the same interviews. All authors reconvened to
discuss and compare the identified codes and themes. This enabled us to build
a shared dictionary of knowledge and agree on the same language, creating
alignment before analysing the remaining data separately. Once the coding of
all interviews and observation notes was concluded, we again discussed our
results collectively to arrive at a final set of themes.

###  4.1. Findings

We next present the most relevant findings from our contextual understanding
study, focusing on the following two themes: ‘trust’ and ‘system
expectations’.

####  4.1.1. Trust

A recurring theme in our findings relates to the degree of trust the
ophthalmologists expressed towards the AI system and the factors influencing
this trust. While P2 noted that the AI has a better track record than
ophthalmologists in detecting microaneurysms and haemorrhages, others often
pointed out mistakes by the AI. The most common error was the marking of
harmless retinal pigments as though they were microaneurysms or haemorrhages
(i.e., false positives). Another common error was the AI failing to detect
microaneurysms or haemorrhages (i.e., false negatives). Such instances
prompted P3 to say that “The AI system does not perform well enough for me to
ignore the green images”. P1 described an occurrence in which the system was
unable to detect a large tumour, with the image labelled as green. While the
system is not trained to detect tumours, this participant stated that this
obstructs her from relying entirely on the system’s report, as severe cases
may slip past. During observations, both P1 and P3 experienced instances of
low image quality, an aspect that the AI system does not consider but which
does impact its ability to analyse those images successfully. The
ophthalmologists noted these errors as a reason for distrust in the system
and, to some, a reason to refrain from using the system.

As apparent in the observations and interviews, most ophthalmologists were
influenced by the AI system’s assessment colours. In validating the system’s
assessment, ophthalmologists repeatedly referred to the colour labels of the
images. For instance, when the AI reported only one yellow image among all
green images, both P1 and P2 would check whether the yellow-labelled image was
assessed correctly. In cases where all images were labelled as green,
ophthalmologists spent substantially less time analysing a patient—dropping in
analysis time from roughly 15 seconds to 5 seconds per image. One participant
noted that “you can be pretty sure that nothing is there if all the images are
green” (P5), and “I look through all of the images, and if it [the AI system]
says they are all green, well then I go through the images slightly faster”
(P5). Both P1 and P4 expressed a similar sentiment, with P4 specifically
pointing to an increased sense of confidence when she agreed with the AI
system: “[the green labels] just give me a feeling of security”. P3, who had
otherwise been critical of the AI system, stated that “the colours do not
matter, unless it’s all green, in which case I go through them quickly”.
Interestingly, when asked about the meaning of each of the three colours, this
participant was unsure what determined each label. Similarly, P7 was unsure
about the meaning of the yellow label. When it came to images labelled as
yellow or red, both P4 and P6 stated that they paid equally more attention to
those compared to the ones labelled green. P4 stated that “If I have some that
are yellow or red—and it really doesn’t matter whether they are one or the
other—then I look at them very carefully”.

####  4.1.2. System Expectations

Our participants had varying ideas of what to expect from the system. While
some saw value in the system’s abilities, others were more negative and
compared it with more advanced AI systems. Four participants (P1, P2, P4, P6)
valued the system enough to regularly use it in their assessment, whereas the
remaining three ophthalmologists (P3, P5, P7) found its value proposition to
be too elementary. When assessing fundus images, the doctors look for several
abnormalities. These abnormalities include cotton-wool, neovascularization,
macular edema, exudates, and microaneurysms. The AI system in use is limited
to detecting microaneurysms and haemorrhages, prompting one of our
participants to state “it only finds red dots, and I can’t really use that for
anything” (P7). Although the overall consensus was that the system should
ideally detect all abnormalities in the retina, some ophthalmologists
highlighted that they also valued the system’s current functionality. P1, P4,
and P6 stated that the system provides a quick overview of a patient’s
progression of DR, as they could see the colour-based labelling on the
thumbnails. Additionally, P2, P4, and P5 explained that they make use of the
colour labelling as a way of self-checking, obtaining a sense of comfort in
their analysis when the system’s output corresponded to their own assessment.

The AI system as used by our participants in the clinic is also used by
optometrists in non-clinical settings. The optometrists typically refer
patients to the hospital in case any lesions or deviations are detected. Given
the aforementioned concerns regarding the accuracy of the system, some of our
participants expressed concern regarding the use of this system by those
without extensive ophthalmological training. As optometrists may not have the
necessary experience in assessing the validity of the AI’s assessments, our
participants expressed the perceived risk of both optometrists and patients
obtaining a false sense of security if the system did not mark any lesions.
Our participants did not express a similar concern for their own assessments,
even though our interviews did identify a clear pattern in the handling of
‘green’ cases in comparison with ‘yellow’ and ‘red’-labelled images.

Throughout our interviews and observations, some ophthalmologists compared the
system to other AI systems and their capabilities. For instance, P5 mentioned
that it would be valuable if they could train the system by giving it feedback
on its classifications, making it “evolve over time and not be stuck”. P3 and
P7 furthermore brought up the idea of connecting the AI system to the
patient’s medical record, enabling it to provide a more precise diagnosis
based on a wider range of patient health details. Besides fundus images, the
ophthalmologists also assess OCT scans, which they would also want the AI
system to be able to process.

##  5\. Study 2 - Bias mitigation

As observed in Study 1, the AI-generated colour labels act as an anchor in the
clinical decision-making process. Our study participants expressed a change in
their inspection as dependent on the label presented, a behaviour which we
were able to confirm through our observations. Further, our participants did
not point to any measures taken to counter this potential bias in their
clinical assessment. Our observation of anchoring bias emerging due to the
introduction of decision support aligns with recent literature (Rastogi et
al., 2022; Fogliato et al., 2022). Prior work furthermore indicates that
clinicians are unaware of the scale of cognitive biases and their effect on
clinical decision-making (Croskerry, 2013), which explains why no effort had
been taken to address the impact of anchoring bias in the presented case.
Given the importance of minimising anchoring bias in clinical decision making
(Fogliato et al., 2022; Furnham and Boo, 2011; Croskerry, 2013), we set out to
design and assess bias mitigation strategies in the context of an existing
ophthalmology CDSS.

Figure 3. Overview of the presented interaction flows. Interaction flow
depends both on the clinician’s and the AI’s assessment.

Overview of the workflows. The top of the image shows the current workflow,
minus the AI annotation. These images are labelled as ‘Hear the story first’.
The remainder of the image is divided into four ’flows’, the presentation of
which is dependent on the combination of the AI’s assessment
(positive/negative) and the clinician’s assessment (positive/negative). In
case of a positive assessment by the clinician, a popup is shown to ask the
clinician for further motivation (labelled: decision justification). In case
of a negative assessment by the clinician but a positive assessment by the AI,
the AI’s classification is shown (labelled: consider the opposite).

###  5.1. Prototyping bias mitigation strategies

Anchoring bias is the result of the cognitive tendency to make assessments
based on the initially observed information rather than considering all
subsequently received information, with the final decision largely affected by
the initial impression (Lighthall and Vazquez-Guillamet, 2015). We explore
three strategies for mitigating anchoring bias in the context of CDSS: ‘hear
the story first’, ‘decision justification’, and ‘consider the opposite’.

####  5.1.1. Hear the story first

One approach to mitigating anchoring bias is to assess the facts before being
presented with the assessment of others (Lighthall and Vazquez-Guillamet,
2015). In Groopman’s investigation on the thought processes that underlie
medical decisions, an interviewed clinician stressed the value of avoiding
anchors; “When a case first arrives, I don’t want to hear anyone else’s
diagnosis. I look at the primary data” (Groopman, 2007). ‘Hear the story
first’ directly addresses the observation from Study 1 that opthalmologists
spent less time on the images labelled as ‘green’ by the CDSS, even though our
participants were aware that the AI’s assessment is based on a limited number
of clinical symptoms. This mitigation strategy was implemented by Fogliato et
al., who refer to this as a ‘two-step workflow’, in which the AI assessment
only shows after the participant has made an initial decision (Fogliato et
al., 2022).

In implementing this bias mitigation strategy, the clinicians can only view
the original (i.e., not processed by the AI system) fundus images. After
looking at all the relevant data, the clinicians return to the patient journal
to fill in the diagnosis, as depicted in Figure 3. Depending on the assessment
of the clinician and the behind-the-scene AI assessment, the application
initiates one of four pathways. ‘Hear the story first’ is implemented by
removing any AI assessments from the initial view, as shown in the
initialisation stage in Figure 3.

####  5.1.2. Decision justification

This strategy prompts decision-makers to explain their reasoning to activate
reflective thinking. Doing so increases the quality of diagnosis and mitigates
cognitive biases (Schmidt et al., 2014). Further, ‘decision justification’ was
identified as a consistently successful strategy in improving diagnostic
accuracy (Isler et al., 2020; Lambe et al., 2016).

Through our observations in Study 1, we learned that ophthalmologists already
describe any positive diagnosis in the hospital’s electronic patient record
following their assessment. However, the current workflow does not support
active reflection, as the goal of the logging activity is documentation rather
than reflection. We, therefore, implement the ‘decision justification’
strategy to occur prior to the presentation of the AI’s assessment and
whenever the ophthalmologist has identified an abnormality in the patient’s
fundus images (Figure 3, Flow 1 and 3). The interface presents the user with a
pop-up window in which they write their motivation for this assessment,
thereby activating their analytical mindset. If the AI system has also
identified an abnormality, it displays its assessment to allow the user to
compare their diagnosis (Figure 3, Flow 1). This is particularly relevant if
the ophthalmologist and the AI system have identified different abnormalities.
If the AI system has not identified an abnormality, this information is shared
with the user following their decision justification (Figure 3, Flow 3).

####  5.1.3. Consider the opposite

Considering possible alternatives before deciding is a bias mitigation
strategy that has been proven effective at mitigating anchoring bias
(Mussweiler et al., 2000; Adame, 2016). This strategy prompts the decision-
maker to consider information that contradicts their current beliefs
(Mussweiler et al., 2000). Simply presenting decision-makers with multiple
alternative options to consider has been shown to reduce cognitive biases in
general (Adame, 2016).

This bias mitigation strategy is activated whenever the ophthalmologist finds
no abnormalities, but the AI system does. In this case, the user is presented
with an opposing answer from the AI system once they have entered their
diagnosis into the patient journal (Figure 3, Flow 2). This lets the
ophthalmologist consider whether or not they have missed anything in their
diagnosis.

In case neither the ophthalmologist nor the AI system found DR-related
abnormalities, the ophthalmologist is presented with a banner at the bottom of
the screen stating that the AI system did not find any microaneurysms or
haemorrhages (Figure 3, Flow 4). This design choice was based on Study 1, in
which we discovered that the ophthalmologists experienced a sense of comfort
when their negative assessment matched that of the AI system.

####  5.1.4. Integrating bias mitigation strategies

We chose to integrate all three bias mitigation strategies in one interactive
prototype. While this prevents us from contrasting the strategies’ individual
effects (e.g., each workflow starts with the ‘hear the story first’ strategy,
possibly followed by an additional strategy—see Figure 3), it ensures that we
can discuss all strategies with our participant sample and consider all
possible combinations of assessment outcomes. The prototype’s user interface
closely resembles the application currently used in the clinic. This
guaranteed participants’ understanding of the system and ensured that
participants’ focus was not diverted toward the visual elements of the system.

|  Hear the story first |  Decision justification |  Consider the opposite
---|---|---|---
Positive reflections |  Allows clinicians to provide their own assessment without external influence. |  Possible value in organising thoughts and sharing these with other team members. |  The short interruption to the workflow allows the clinician to reconsider their initial assessment.
Negative reflections |  Additional time spent analysing each image, efficiency brought in by AI system is lost. |  Additional effort required by clinicians to write down their motivation for assessment. Perceived as repetitive to the assessment already provided in the patient journal. |  The AI suggestions can be perceived as correcting the clinician rather than supporting them in their task.
Table 2. Summary overview of participants’ reflections on the three bias
mitigation strategies.

###  5.2. Method

To investigate clinicians’ perceptions of the bias mitigation techniques, we
conducted evaluations with members of Aalborg University Hospital’s Department
of Ophthalmology. Through our liaison within the department, we were able to
recruit six ophthalmologists, five of which also participated in Study 1. We
report participant details in Table 1. We conducted a pilot evaluation with
our liaison at the department before the evaluations to identify potential
errors. All six evaluations were audio-recorded and transcribed. Each of these
evaluations, including the interview, lasted for approximately one hour. The
evaluations took place in the participant’s place of work, where they have
access to and regularly interact with the AI system outlined in Section 4.
Three of the paper’s authors were present at all evaluations. This allowed us
to distribute tasks, including the introduction of the prototype, the
interview, and taking notes. During the evaluations, participants were
presented with the prototype and walked through each of the four flows. We
subsequently asked for participants’ reflections on the debiasing techniques
and cognitive biases.

Following the evaluations, all transcripts were analysed using reflexive
thematic analysis, following the same procedure as in Study 1. The same
authors present at the evaluations took part in analysing the transcripts.
Initially, one transcript was chosen to be analysed individually by all. Then,
all authors compared the resulting codes and themes, sharing our
understandings to establish alignment. The remaining transcripts were then
divided to be analysed individually. When all six transcripts had been
analysed, we collaboratively assessed all themes and codes.

###  5.3. Findings

We identified four main themes from our evaluation results. These themes focus
on the envisioned effect on diagnostic accuracy, operational efficiency,
openness to AI-generated insights, and the role of bias in daily
ophthalmological practice. We summarise participants reflections on the three
bias mitigation strategies in Table 2.

####  5.3.1. Envisioned Effect on Diagnostic Accuracy

Diagnostic accuracy was a core discussion point in evaluating the bias
mitigation techniques. Several ophthalmologists (P1, P3, P7) could not imagine
that the debiasing techniques would heighten their diagnostic accuracy, as
they generally did not think their diagnoses could get any more accurate. The
rest (P2, P6, P8) firmly believed that debiasing would improve diagnostic
accuracy, but at the cost of efficiency. For instance, P6 stated: “I think it
would be really great to have periodically […] to get back into it and to
start thinking for yourself, and not fall asleep completely”. This points to
the fact that screening for several hours can exhaust clinicians’ ability to
stay focused. When asked to imagine that they had as much time as they wanted
to conduct a screening, disregarding overall efficiency, P2 stated: “If I had
all the time in the world, this would be the optimal way to do it”.

In relation to the implementation of ‘hear the story first’, P2 stated that
removing the AI output from the fundus image would likely increase diagnostic
accuracy. Specifically, she stated: “The bias has been removed, I arrive at my
own conclusion, and then I compare my conclusion with the AI’s conclusion”
(P2). Four of the ophthalmologists (P2, P6, P7, P8) stated that ‘consider the
opposite’ would be the most helpful in ensuring that they do not miss any
critical details. For instance, P2 stated: “Sometimes you get tired, so if the
program said ‘Something is there’, and I scale up [the image] and then I think
‘Yes, I didn’t notice that myself’”. Though more sceptical, P8 stated, when
referring to ‘consider the opposite’: “It is absolutely a more useful
solution. But it wouldn’t make our work any faster, but from a quality
perspective, then yes”. Concerning the strategy of ‘decision justification’,
P6 hypothesised that by sharing one’s justifications with the team, the
diagnostic accuracy could be heightened collectively, as they would gain
insight into each other’s thought processes.

All ophthalmologists expressed that the debiasing techniques could be a
learning tool for inexperienced screeners to heighten their diagnostic
accuracy. As inexperienced ophthalmologists can be less confident of their
decisions, the system could help them develop their diagnostic reasoning. For
instance, P7 commented that it could “act as a kind of safety measure”. In
that regard, P2 stated: “When you are training, you have a need for someone
else to check what you are doing, ‘Is this right or is it wrong?’”. Similarly,
P8 stated: “It would work well as a learning tool since you have to reflect on
what you have found—but to those of us that are as obdurate as we are, I don’t
think it will change very much”.

####  5.3.2. The Restraining Role of Efficiency

The clinicians’ focus on efficiency in their work plays as large a role as
their diagnostic accuracy. As P1 stated, “in our everyday work, we have to
focus on efficiency mostly, we have to be quick and view many images because
the patient numbers keep growing. So we need a solution that helps us become
quick”. Similarly, P2 reflected; “efficiency is king today, and that is what
has become problematic. Often you consciously compromise”, indicating that the
desired efficiency comes at the expense of diagnostic accuracy. P2 added that
the number of tests performed on each patient has increased, resulting in more
data for each patient. Consequently, the ophthalmologists were critical of the
impact on efficiency that bias mitigation would have. One of the aspects that
exemplified this standpoint was the implementation of ‘hear the story first’,
in which the AI-generated colour labels were removed. Specifically, two of the
ophthalmologists (P1, P2) commented that they would have to spend more time
examining each fundus image themselves, as opposed to getting a quick overview
of all fundus images as provided by the AI colour labels. In this regard, P2
stated that “if I already trust what the AI system says and I have its answer
from the start then I can shortcut looking through all the images. I can’t do
that if I get them [the processed images] afterwards. Then I have to analyse
every image as new”. P1 elaborated: “There are very many patients that have
nothing on their retinas”, referring to how she can quickly identify the cases
with no DR due to the AI system labelling all of these patients’ fundus images
green.

Other ophthalmologists (P3, P6, P7, P8) expressed indifference to removing the
colour labels. This was primarily because they did not use the AI system’s
output at all or did not look at it until they had gone through the original
images themselves. P7 saw potential in using ‘hear the story first’ to avoid
being biased at the beginning of screenings and elaborated that he hopes his
colleagues using the colour labels do not take them too literally. When
presented with the first and third flow (Figure 3), in which ‘decision
justification’ is implemented, participants again expressed concern regarding
efficiency. For instance, P8 considered it counter-productive to justify
whenever he found any retinal abnormalities: “from an efficiency point of
view—no thanks. It would just prolong the process completely”.

####  5.3.3. Openness to AI-generated Insights

Our incorporation of the bias mitigation strategies sparked reflections on
participants’ role as domain experts. As stated by P2: “When you’re an expert,
you have many years of experience, but you also think you do it better than
others. And then it is up to me how much doubt I allow in myself – whether I
accept other information that contradicts me”. This participant then suggested
that some experts are more open to input than others due to their belief that
they outperform automated solutions. Most of the ophthalmologists (P2, P3, P6,
P7, P8) brought up that the presentation made it seem like a precautionary
measure to fact-check their answer, which pertains specifically to the
strategy of ‘consider the opposite’. As P2 commented: “At the moment, it [the
AI system] suggests ‘this could be a haemorrhage’ and then I go onto the image
and see if that is correct—so in that way, I fact-check the program. In this
way, the program is the one fact-checking me”.

Some ophthalmologists (P2, P6, P7) found the prospect of being fact-checked by
the AI system appealing and an excellent way to take advantage of its
capabilities. In addition, P6 elaborated that he would use the AI system’s
output to go “back to see if what I found is just nonsense” and that “even
though we might agree, there could still be anomalies that you would notice
after that you have missed”. On the contrary, other participants (P3, P8) were
more sceptical of the debiasing techniques due to their preexisting distrust
of the AI system due to it mistaking harmless retinal pigments for DR-related
abnormalities. However, had the AI system been more capable (i.e., detecting
more types of abnormalities and higher precision), they would be more open to
using the debiasing techniques. As P3 stated, “if the AI was very skilled and
functioning, then I could easily imagine it as some sort of mentor that would
hit people over the hand and say, ‘Let’s go through 20 images, is anything
there or not?’”. He elaborated: “It does not work if it’s the janitor that
comes in and lectures the executive. That is kind of what is happening here.
It has to be another executive, capacity-wise, someone who is capable and who
knows what can be done, that comes in and lectures the executive”, alluding to
the fact that he does not recognise the AI system as able to correct his
diagnosis. This point of view is supported by P3: “it is very rare that it
[the AI system] shows me something that will make me change my mind”.
Contrarily, as is evident in the following statement, P2 thinks more highly of
the AI system: “I think this [decision justification] works well because there
is a disagreement between two systems, where I believe in myself, but I also
believe in the system, and then it is important to document why I disagree”.

####  5.3.4. Bias in Current Practice

The ophthalmologists recalled bias as being only a minor part of their
education. Bias was not perceived as an aspect they put a lot of thought into
during their daily practice. When asked about the role of bias in their work,
P1, P3, and P6 stated that bias does not affect their work much. Others (P2,
P7, P8) thought it played a more prominent and unavoidable role in their work.
P2 explained how she sometimes sees bias as playing in her favour: “If I want
to be able to use 15 minutes looking at the patient, then I need to have a
preliminary answer. That means that I want to have bias because it helps me
with my own decision-making”. P7 explained how they work based on a working
diagnosis (i.e., the most probable diagnosis that the clinicians believe to be
the cause of a patient’s symptoms), guiding the choice of diagnostic tests to
confirm or deny the working diagnosis. As new information is discovered, the
clinicians adjust by choosing among other potential causes, or differential
diagnoses, before ultimately arriving at a final diagnosis. P2 added: “you
have to create a working diagnosis: ‘What do I think it could be?’, ‘How can I
treat it?’, ‘What if it’s wrong?’, ‘What kind of mistakes might I produce from
the treatment?’”. P7 stated: “You receive a diagnosis that you work from, and
at this point, it is important to ‘reset’ and instead create your own
diagnosis, as it [the symptoms] could be caused by something else. We have to
provide evidence for the diagnoses we make”. Thus, to avoid premature closure
of cases, ophthalmologists use differential and working diagnoses to ensure
that alternative diagnoses are considered.

##  6\. Discussion

Having collected domain experts’ perceptions on the integration of bias
mitigation strategies, we discuss clinicians’ perspectives on the role of bias
and bias mitigation in decision support systems.

###  6.1. Designing for Bias Mitigation

We have evaluated three bias mitigation strategies as integrated into an
interactive CDSS prototype. Although mitigation strategies are frequently
discussed in the literature (Isler et al., 2020; Lambe et al., 2016; Adame,
2016; Mussweiler et al., 2000), relatively few studies have explored how bias
mitigation can be incorporated into AI decision-support systems. We evaluated
three bias mitigation strategies, namely ‘hear the story first’, ‘decision
justification’, and ‘consider the opposite’, which we selected based on prior
work’s suggestion that these are well suited for combatting anchoring bias. We
first summarise participants’ perceptions of these strategies and subsequently
outline three recommendations and opportunities for future bias mitigation
tools in the context of Human-AI interaction.

Our findings indicate that the ophthalmologists using AI-generated colour
labels to obtain an overview of a patient’s status did not favour the ‘hear
the story first’ strategy. Those who did not use this overview expressed
indifference to removing the colour labels. All ophthalmologists expressed
efficiency concerns concerning the ‘decision justification’ strategy,
reflecting the additional effort it requires by clinicians. The
ophthalmologists largely agreed that ‘consider the opposite’ would be the most
helpful strategy to avoid missing important details. Despite this, several
participants commented that they would experience such a system as if being
‘fact-checked’ by the AI system. These results highlight the inherent
challenges in incorporating bias mitigation strategies in AI-supported CDSS.

Recommendation 1: Introduce subtle design patterns to overcome aversion to
bias mitigation interfaces. Given these complexities and trade-offs, we call
on the HCI community to consider and evaluate alternative interface designs
that could support bias mitigation strategies. In particular, more subtle
designs could help to make end-users feel less fact-checked. How to achieve
subtleness in interactions is a relatively unexplored area within HCI (Pohl et
al., 2019), although a number of works have evaluated how to ‘nudge’ people’s
attention or behaviour through interface adjustments (Yang et al., 2019). For
example, Echterhoff et al. show that manipulating the order in which decisions
are presented can reduce sequential dependencies to prior decisions, thereby
reducing anchoring bias (Echterhoff et al., 2022). Another example of subtle
interaction is presented by Sridharan et al., who show that the gaze of novice
radiologists can be manipulated to follow the scanpath of an expert
radiologist, resulting in increased diagnostic accuracy (Sridharan et al.,
2012).

Recommendation 2: Activate bias mitigation techniques periodically rather than
constantly. We encountered several participants that questioned the need for
bias mitigation in their work. Prior work has highlighted that decision-makers
can become overconfident in their abilities (Mussweiler et al., 2000;
Lighthall and Vazquez-Guillamet, 2015). Those questioning the need for bias
mitigation described the AI system as a junior colleague presenting their
opinion. This exacerbated the notion of feeling fact-checked by the
recommendation system. To reduce the feeling of being constantly evaluated, a
possible solution is to periodically activate the bias mitigation strategies.
This follows from participants’ comments that bias is merely a minor part of
their education and is not of serious concern to them throughout the working
day. While evidence of long-term retention effects of bias mitigation
techniques is scarce (Korteling et al., 2021), several studies have found
positive effects of bias mitigation in studies lasting up to eight weeks
(Korteling et al., 2021; Veinott et al., 2013; Morewedge et al., 2015). Such a
period would suffice in the clinical context studied in this paper, thereby
supporting the real-world feasibility of this recommendation.

Recommendation 3: Increase user awareness of potential cognitive biases
through the decision-support interface. Cognitive biases are ubiquitous, and
clinical work is no exception (Croskerry, 2013). An essential step towards
addressing these biases is by providing clinicians with the tools to increase
and subsequently maintain their awareness of these biases. While the
clinicians in our study sample expressed concern towards the potentially
detrimental effects of cognitive biases, they also pointed to a lack of
coverage of cognitive biases in their formal training. Although the need for
adequate bias mitigation is increasingly recognised in both the academic
literature and national guidelines for clinical training (O’Sullivan and
Schofield, 2018), reflection on the role of decision-support systems in these
efforts remains rudimentary. The interfaces with which clinicians interact
throughout the day could, therefore, provide a gateway to increase awareness
of cognitive biases. Prior work has, for example, augmented the interface of a
social media platform with visual hints to indicate a wider diversity of
opinions to the user (Gao et al., 2018). Awareness of cognitive biases is
especially critical in light of a recent literature review which indicates
that cognitive biases can aggravate with the introduction of explainable AI
(Bertrand et al., 2022).

###  6.2. Expectations and Realities of Real-World AI Adoption

The efficiency of clinicians is of critical importance to the Department of
Ophthalmology. As the number of tests performed per patient has gone up
(Albinus, 2021), so has the amount of data that needs to be interpreted by
doctors. In addition, a political decision was made in Denmark to screen all
diabetes patients regularly to monitor progressing DR symptoms. This has
increased the pressure on the public health care system, with some of our
participants highlighting how other matters are at times neglected to ensure
efficiency. The promise of an AI system which supports clinicians in dealing
with an increasing number of cases was therefore well received by the
department.

Our results challenge some of the perceived benefits of deploying an AI
decision-support system into ophthalmology practice. When discussing the
implementation of bias mitigation strategies with the ophthalmologists,
several mentioned that they would be more likely to favour the proposed
debiasing techniques if the AI system was more capable. Specifically, they
wished for the AI system to produce fewer false positive alerts and be able to
detect more types of abnormalities. This points to a mismatch between
clinicians’ mental model and expectations of the AI’s capabilities (van Berkel
et al., 2023). Similar observations were made by Yang et al. in their
investigation of decision-making practices in the context of heart pump
implants (Yang et al., 2016). Here, the AI’s quantitative predictions did not
align with the information needs of the clinicians. In our study, several
participants pointed to past experiences in which the AI system missed a large
tumour. While the deployed AI system was not designed to recognise and alert
to such occurrences, these ‘failures’ to detect obvious abnormalities do
negatively impact clinicians’ experiences. As detection technology advances,
such problems may eventually be overcome. However, the expected timeline for
this remains uncertain. A recent literature review on AI for DR screening
states that “most [commercial] systems are still being actively developed with
changes and improvements to detection algorithms, user interface, scalability,
better detection of DMO, etc. in progress.” (Grzybowski et al., 2020). As
such, HCI research must do more to align end-users’ expectations with AI
systems’ capabilities and ensure integration in existing work practices, as
highlighted in prior work (van Berkel et al., 2021; Yang et al., 2019).
Grzybowski et al. state that even if AI assessment systems increase in
quality, clinical experts will still be required to assess atypical situations
and low-quality images, as well as for quality monitoring (Grzybowski et al.,
2020).

In addition to the technological limitations highlighted by our participants,
our results further point to the tension that may arise between the need for
respecting clinicians’ expertise and the need for mitigating cognitive biases
in CDSS. Prior work on AI decision support has shown that clinical experts may
choose not to use such support systems due to a lack of perceived need (Yang
et al., 2016; Wang et al., 2021), especially among senior clinicians (Yang et
al., 2016; van Berkel et al., 2021). In contrast, enabling the clinicians to
direct the CDSS rather than merely receiving its recommendations has improved
clinicians’ acceptance and efficiency (Jo et al., 2022; Cai et al., 2019a; Jo
et al., 2022). Bias mitigation may risk compromising clinicians’ agency by
intervening in their workflow, a critique also reflected in our findings.
Given the impact of clinicians’ agency on the eventual adoption of CDSS, this
topic warrants critical consideration in future work on bias mitigation.

Lastly, while the additional screening of people is desirable to improve
individual health outcomes, extensive screening increases the overall pressure
on the healthcare system as the number of positive cases increases. As
observed in our case study, this additional workload is currently not
accounted for in the resources available within the public health care system.
Despite advances in DR detection systems, the involvement of ophthalmologists
remains essential (Chandakkar et al., 2012). In order to assess the benefit of
AI-enabled CDSS, it is therefore critical that their impact is assessed not
only on their image classification performance but rather their effectiveness
in real-world practice.

###  6.3. Limitations and Future Work

We acknowledge several limitations in our work. First, we merged three
debiasing techniques into a single prototype to ensure a more realistic
workflow. This limits our ability to assess the suitability of individual
techniques, as participants’ perception was inevitably affected by their
encounters with the other strategies. Second, the bias mitigation strategies
we presented can be integrated into the user interface in myriad ways. As
such, the ophthalmologists’ statements regarding our strategies’
implementation may not directly transfer to individual bias mitigation
strategies. However, to integrate bias mitigation strategies into CDSS, we
must suggest and evaluate concrete ways of doing so. Third, a quantitative
assessment of the impact of the bias mitigation strategies was out of the
scope of our study as we focused on clinicians’ perceptions of bias
mitigation. While we are therefore unable to report on the impact of bias
mitigation during actual real-world usage, we consider this as a critical step
towards aligning with stakeholder needs. This aligns with Bertrand et al.’s
recent call for an increased emphasis on cognitive biases in the context of
explainable AI (Bertrand et al., 2022).

Future work may explore the effect of specific debiasing strategies and
eventually study its effect in clinical trials to understand its impact in
practice. Reflecting on our role as researchers, we chose to point to the
literature when discussing cases of cognitive bias in the clinic—rather than
potentially upsetting participants by pointing to their own biases. Although
several participants actively recognised the impact of their own cognitive
biases on their work, not all expressed a similar sentiment. This points to
the need to increase participant awareness of potential cognitive biases, and
the challenges researchers may face when involving end-users in designing
cognitive bias mitigation strategies. This study focused exclusively on
ophthalmological examination, DR in particular. Although tedious and requiring
a thorough investigation, the assessment of DR is not representative of all
ophthalmological tasks or clinical image assessment more broadly. In
particular, different tasks may evoke other notions of urgency, both in terms
of time and clinical need. It would therefore be valuable to study perceptions
towards bias mitigation in other clinical contexts.

##  7\. Conclusion

In this paper, we investigated ophthalmologists’ perceptions of bias
mitigation strategies as part of their AI-supported clinical workflow. We
first conducted a contextual understanding study in which we interviewed and
observed domain experts to understand their use of an AI support system.
Following, we designed and evaluated an interactive prototype that embodies
three bias mitigation strategies, ‘hear the story first’, ‘decision
justification’, and ‘consider the opposite’, as part of their existing
workflow in screening diabetes patients for DR. We find that clinicians were
largely aware of the cognitive biases that may emerge when faced with AI
suggestions. However, we encountered extensive aversion towards any bias
mitigation strategy that may negatively impact overall efficiency. Based on
our findings, we point to the potential of subtle design patterns to overcome
aversion to bias mitigation techniques, to examine biases periodically rather
than constantly, and to increase awareness of potential cognitive biases
through the DSS’s user interface. Given the growing role of AI support in
clinical care, we urge the HCI research community to critically assess and
work to minimise any cognitive biases that may arise due to the increased
reliance on AI-driven decision-support systems.

###### Acknowledgements.

The authors would like to thank Tobias Nissen for his valuable support and
express our gratitude to the clinicians who took part in the studies. This
work is supported by the ‘Algorithmic Explainability for Everyday Citizens’
project of the Carlsberg Foundation and the ‘Explain-Me’ project of Digital
Research Centre Denmark (DIREC) under Innovation Fund Denmark.

## References

  * (1)
  * Abràmoff et al. (2018) Michael D. Abràmoff, Philip T. Lavin, Michele Birch, Nilay Shah, and James C. Folk. 2018\.  Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices.  _npj Digital Medicine_ 1, 1 (2018), 39.  https://doi.org/10.1038/s41746-018-0040-6
  * Adame (2016) Bradley J Adame. 2016\.  Training in the mitigation of anchoring bias: A test of the consider-the-opposite strategy.  _Learning and Motivation_ 53 (2016), 36–48.  https://doi.org/10.1016/j.lmot.2015.11.002
  * Akram et al. (2014) M. Usman Akram, Anam Tariq, Shoab A. Khan, and M. Younus Javed. 2014.  Automated detection of exudates and macula for grading of diabetic macular edema.  _Computer Methods and Programs in Biomedicine_ 114, 2 (2014), 141–152.  https://doi.org/10.1016/j.cmpb.2014.01.010
  * Albinus (2021) Niels-Bjørn Albinus. 2021\.  _Antallet af øjenpatienter er eksploderet på 15 år_.  https://dagensmedicin.dk/antallet-af-oejenpatienter-er-eksploderet-paa-15-aar/
  * Beede et al. (2020) Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M. Vardoulakis. 2020\.  A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3313831.3376718
  * Bertrand et al. (2022) Astrid Bertrand, Rafik Belloum, James R. Eagan, and Winston Maxwell. 2022.  How Cognitive Biases Affect XAI-Assisted Decision-Making: A Systematic Review. In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_ (Oxford, United Kingdom) _(AIES ’22)_. Association for Computing Machinery, New York, NY, USA, 78–91.  https://doi.org/10.1145/3514094.3534164
  * Braun et al. (2019) Virginia Braun, Victoria Clarke, Nikki Hayfield, and Gareth Terry. 2019.  _Thematic Analysis_.  Springer Singapore, Singapore, 843–860.  https://doi.org/10.1007/978-981-10-5251-4_103
  * Cabitza and Zeitoun (2019) F. Cabitza and J. D. Zeitoun. 2019.  The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence.  _Annals of Translational Medicine_ 7, 8 (2019), 161\.  https://doi.org/10.21037/atm.2019.04.07
  * Cai et al. (2019a) Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, and Michael Terry. 2019a.  Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3290605.3300234
  * Cai et al. (2019b) Carrie J. Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019b.  ”Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 104 (2019), 24 pages.  https://doi.org/10.1145/3359206
  * Chandakkar et al. (2012) Parag S. Chandakkar, Ragav Venkatesan, Baoxin Li, and Helen K. Li. 2012.  A Machine-Learning Approach to Retrieving Diabetic Retinopathy Images. In _Proceedings of the ACM Conference on Bioinformatics, Computational Biology and Biomedicine_ _(BCB ’12)_. Association for Computing Machinery, New York, NY, USA, 588–589.  https://doi.org/10.1145/2382936.2383030
  * Croskerry (2013) P. Croskerry. 2013\.  From mindless to mindful practice–cognitive bias and clinical decision making.  _The New England Journal of Medicine_ 368, 26 (2013), 2445–2448.  https://doi.org/10.1056/NEJMp1303712
  * Dingler et al. (2022) Tilman Dingler, Benjamin Tag, David A. Eccles, Niels van Berkel, and Vassilis Kostakos. 2022.  Method for Appropriating the Brief Implicit Association Test to Elicit Biases in Users. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 243, 16 pages.  https://doi.org/10.1145/3491102.3517570
  * Echterhoff et al. (2022) Jessica Maria Echterhoff, Matin Yarmand, and Julian McAuley. 2022\.  AI-Moderated Decision-Making: Capturing and Balancing Anchoring Bias in Sequential Decision Tasks. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 161, 9 pages.  https://doi.org/10.1145/3491102.3517443
  * Fogliato et al. (2022) Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Paul Fisher, Diane Wilson, Michael Fitzke, Mark Parkinson, Eric Horvitz, Kori Inkpen, and Besmira Nushi. 2022\.  Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 1362–1374.  https://doi.org/10.1145/3531146.3533193
  * Furnham and Boo (2011) Adrian Furnham and Hua Chu Boo. 2011.  A literature review of the anchoring effect.  _The Journal of Socio-Economics_ 40, 1 (2011), 35–42.  https://doi.org/10.1016/j.socec.2010.10.008
  * Gao et al. (2018) Mingkun Gao, Hyo Jin Do, and Wai-Tat Fu. 2018.  Burst Your Bubble! An Intelligent System for Improving Awareness of Diverse Social Opinions. In _23rd International Conference on Intelligent User Interfaces_ (Tokyo, Japan) _(IUI ’18)_. Association for Computing Machinery, New York, NY, USA, 371–383.  https://doi.org/10.1145/3172944.3172970
  * Goh et al. (2016) James Kang Hao Goh, Carol Y. Cheung, Shaun Sebastian Sim, Pok Chien Tan, Gavin Siew Wei Tan, and Tien Yin Wong. 2016.  Retinal Imaging Techniques for Diabetic Retinopathy Screening.  _Journal of Diabetes Science and Technology_ 10, 2 (2016), 282–294.  https://doi.org/10.1177/1932296816629491
  * Gregori (2021) Ninel Z Gregori. 2021\.  _Diabetic Retinopathy: Causes, Symptoms, Treatment_.  https://www.aao.org/eye-health/diseases/what-is-diabetic-retinopathy
  * Groopman (2007) Jerome E Groopman. 2007\.  _How doctors think_. Vol. 82.  Houghton Mifflin Boston.
  * Grzybowski et al. (2020) Andrzej Grzybowski, Piotr Brona, Gilbert Lim, Paisan Ruamviboonsuk, Gavin S. W. Tan, Michael Abramoff, and Daniel S. W. Ting. 2020\.  Artificial intelligence for diabetic retinopathy screening: a review.  _Eye_ 34, 3 (01 Mar 2020), 451–460.  https://doi.org/10.1038/s41433-019-0566-0
  * Gulshan et al. (2016) Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. 2016.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.  _JAMA_ 316, 22 (2016), 2402–2410.  https://doi.org/10.1001/jama.2016.17216
  * Horowitz et al. (2021) H. W. Horowitz, C. Behar, and J. Greene. 2021.  COVID-19 and the Consequences of Anchoring Bias.  _Emerging Infectious Diseases_ 27, 8 (2021), 2235–2236.  https://doi.org/10.3201/eid2708.211107
  * Isler et al. (2020) Ozan Isler, Onurcan Yilmaz, and Burak Dogruyol. 2020\.  Activating reflective thinking with decision justification and debiasing training.  _Judgment and Decision Making_ 15, 6 (2020), 926–938.
  * Jiang et al. (2017) Fei Jiang, Yong Jiang, Hui Zhi, Yi Dong, Hao Li, Sufeng Ma, Yilong Wang, Qiang Dong, Haipeng Shen, and Yongjun Wang. 2017.  Artificial intelligence in healthcare: past, present and future.  _Stroke and Vascular Neurology_ 2, 4 (2017), 230–243.  https://doi.org/10.1136/svn-2017-000101
  * Jo et al. (2022) Eunkyung Jo, Myeonghan Ryu, Georgia Kenderova, Samuel So, Bryan Shapiro, Alexandra Papoutsaki, and Daniel A. Epstein. 2022.  Designing Flexible Longitudinal Regimens: Supporting Clinician Planning for Discontinuation of Psychiatric Drugs. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 352, 17 pages.  https://doi.org/10.1145/3491102.3502206
  * Kapoor et al. (2019) Rahul Kapoor, Stephen P. Walters, and Lama A. Al-Aswad. 2019\.  The current state of artificial intelligence in ophthalmology.  _Survey of Ophthalmology_ 64, 2 (2019), 233–240.  https://doi.org/10.1016/j.survophthal.2018.09.002
  * Korteling et al. (2021) J. E. H. Korteling, J. Y. J. Gerritsma, and A. Toet. 2021\.  Retention and Transfer of Cognitive Bias Mitigation Interventions: A Systematic Literature Study.  _Front Psychol_ 12 (2021), 629354.
  * Kruse and Ehrbar (2020) Clemens Scott Kruse and Nolan Ehrbar. 2020.  Effects of computerized decision support systems on practitioner performance and patient outcomes: systematic review.  _JMIR Medical Informatics_ 8, 8 (2020), e17283.
  * Lambe et al. (2016) Kathryn Ann Lambe, Gary O’Reilly, Brendan D Kelly, and Sarah Curristan. 2016.  Dual-process cognitive interventions to enhance diagnostic reasoning: a systematic review.  _BMJ Quality & Safety_ 25, 10 (2016), 808–820.  https://doi.org/10.1136/bmjqs-2015-004417
  * Lee et al. (2021) Min Hun Lee, Daniel P. Siewiorek, Asim Smailagic, Alexandre Bernardino, and Sergi Bermúdez i Badia. 2021\.  A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 392, 14 pages.  https://doi.org/10.1145/3411764.3445472
  * Lieder et al. (2018) Falk Lieder, Thomas L. Griffiths, Quentin J. M. Huys, and Noah D. Goodman. 2018.  The anchoring bias reflects rational use of cognitive resources.  _Psychonomic Bulletin & Review_ 25, 1 (2018), 322–349.  https://doi.org/10.3758/s13423-017-1286-8
  * Lighthall and Vazquez-Guillamet (2015) Geoffrey K Lighthall and Cristina Vazquez-Guillamet. 2015\.  Understanding decision making in critical care.  _Clinical Medicine & Research_ 13, 3-4 (2015), 156–168.  https://doi.org/10.3121/cmr.2015.1289
  * Lilienfeld and Lynn (2014) Scott O. Lilienfeld and Steven Jay Lynn. 2014.  _Errors/Biases in Clinical Decision Making_.  John Wiley & Sons, Ltd, 1–9.  https://doi.org/10.1002/9781118625392.wbecp567
  * Molin et al. (2016) Jesper Molin, Paweł W. Woźniak, Claes Lundström, Darren Treanor, and Morten Fjeld. 2016\.  Understanding Design for Automated Image Analysis in Digital Pathology. In _Proceedings of the 9th Nordic Conference on Human-Computer Interaction_ _(NordiCHI ’16)_. Association for Computing Machinery, New York, NY, USA, Article 58, 10 pages.  https://doi.org/10.1145/2971485.2971561
  * Morewedge et al. (2015) Carey K. Morewedge, Haewon Yoon, Irene Scopelliti, Carl W. Symborski, James H. Korris, and Karim S. Kassam. 2015.  Debiasing Decisions: Improved Decision Making With a Single Training Intervention.  _Policy Insights from the Behavioral and Brain Sciences_ 2, 1 (2015), 129–140.  https://doi.org/10.1177/2372732215600886
  * Mussweiler et al. (2000) Thomas Mussweiler, Fritz Strack, and Tim Pfeiffer. 2000\.  Overcoming the inevitable anchoring effect: Considering the opposite compensates for selective accessibility.  _Personality and Social Psychology Bulletin_ 26, 9 (2000), 1142–1150.
  * Nguyen et al. (2020) Quang H. Nguyen, Ramasamy Muthuraman, Laxman Singh, Gopa Sen, Anh Cuong Tran, Binh P. Nguyen, and Matthew Chua. 2020.  Diabetic Retinopathy Detection Using Deep Learning. In _Proceedings of the 4th International Conference on Machine Learning and Soft Computing_ _(ICMLSC 2020)_. Association for Computing Machinery, New York, NY, USA, 103–107.  https://doi.org/10.1145/3380688.3380709
  * NHS (2021) NHS. 2021.  _Stages, Diabetic Retinopathy_.  https://www.nhs.uk/conditions/diabetic-retinopathy/stages/
  * Niemeijer et al. (2007) Meindert Niemeijer, Bram van Ginneken, Stephen R. Russell, Maria S. A. Suttorp-Schulten, and Michael D. Abràmoff. 2007\.  Automated Detection and Differentiation of Drusen, Exudates, and Cotton-Wool Spots in Digital Color Fundus Photographs for Diabetic Retinopathy Diagnosis.  _Investigative Ophthalmology & Visual Science_ 48, 5 (2007), 2260–2267.  https://doi.org/10.1167/iovs.06-0996
  * Norman et al. (2017) G. R. Norman, S. D. Monteiro, J. Sherbino, J. S. Ilgen, H. G. Schmidt, and S. Mamede. 2017\.  The Causes of Errors in Clinical Reasoning: Cognitive Biases, Knowledge Deficits, and Dual Process Thinking.  _Academic Medicine_ 92, 1 (2017), 23–30.
  * Nourani et al. (2021) Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Donald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021.  Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. In _26th International Conference on Intelligent User Interfaces_ _(IUI ’21)_. Association for Computing Machinery, New York, NY, USA, 340–350.  https://doi.org/10.1145/3397481.3450639
  * of Health and the Elderly) (2020) Magnus Heunicke (Minister of Health and the Elderly). 2020.  Question 54.  _The Danish Parliament_ (2020).  https://www.ft.dk/samling/20201/almdel/SUU/spm/54/svar/1706109/index.htm
  * Ogdie et al. (2012) Alexis R. Ogdie, James B. Reilly, Wyki G. Pang, Shimrit Keddem, Frances K. Barg, Joan M. Von Feldt, and Jennifer S. Myers. 2012.  Seen Through Their Eyes: Residents’ Reflections on the Cognitive and Contextual Components of Diagnostic Errors in Medicine.  _Academic Medicine_ 87, 10 (2012).  https://doi.org/10.1097/ACM.0b013e31826742c9
  * O’Sullivan and Schofield (2018) ED O’Sullivan and SJ Schofield. 2018.  Cognitive Bias in Clinical Medicine.  _Journal of the Royal College of Physicians of Edinburgh_ 48, 3 (2018), 225–232.  https://doi.org/10.4997/jrcpe.2018.306
  * Piorkowski et al. (2021) David Piorkowski, Soya Park, April Yi Wang, Dakuo Wang, Michael Muller, and Felix Portnoy. 2021\.  How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1, Article 131 (2021), 25 pages.  https://doi.org/10.1145/3449205
  * Pohl et al. (2019) Henning Pohl, Andreea Muresan, and Kasper Hornbæk. 2019\.  Charting Subtle Interaction in the HCI Literature. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3290605.3300648
  * Rastogi et al. (2022) Charvi Rastogi, Yunfeng Zhang, Dennis Wei, Kush R. Varshney, Amit Dhurandhar, and Richard Tomsett. 2022\.  Deciding Fast and Slow: The Role of Cognitive Biases in AI-Assisted Decision-Making.  _Proc. ACM Hum.-Comput. Interact._ 6, CSCW1, Article 83 (2022), 22 pages.  https://doi.org/10.1145/3512930
  * Rieger et al. (2021) Alisa Rieger, Tim Draws, Mariët Theune, and Nava Tintarev. 2021\.  This Item Might Reinforce Your Opinion: Obfuscation and Labeling of Search Results to Mitigate Confirmation Bias. In _Proceedings of the 32nd ACM Conference on Hypertext and Social Media_ _(HT ’21)_. Association for Computing Machinery, New York, NY, USA, 189–199.  https://doi.org/10.1145/3465336.3475101
  * Roy et al. (2017) Abhijit Guha Roy, Sailesh Conjeti, Sri Phani Krishna Karri, Debdoot Sheet, Amin Katouzian, Christian Wachinger, and Nassir Navab. 2017\.  ReLayNet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional networks.  _Biomedical Optics Express_ 8, 8 (2017), 3627–3642.  https://doi.org/10.1364/BOE.8.003627
  * Royce et al. (2019) Celeste S. Royce, Margaret M. Hayes, and Richard M. Schwartzstein. 2019.  Teaching Critical Thinking: A Case for Instruction in Cognitive Biases to Reduce Diagnostic Errors and Improve Patient Safety.  _Academic Medicine_ 94, 2 (2019).
  * Santhanam et al. (2020) Sashank Santhanam, Alireza Karduni, and Samira Shaikh. 2020\.  Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376318
  * Schmidt et al. (2014) Henk G Schmidt, Sílvia Mamede, Kees Van Den Berge, Tamara Van Gog, Jan LCM Van Saase, and Remy MJP Rikers. 2014.  Exposure to media information about a disease can cause doctors to misdiagnose similar-looking clinical cases.  _Academic Medicine_ 89, 2 (2014), 285–291.
  * Sherbino et al. (2014) J. Sherbino, K. Kulasegaram, E. Howey, and G. Norman. 2014\.  Ineffectiveness of cognitive forcing strategies to reduce biases in diagnostic reasoning: a controlled trial.  _Canadian Journal of Emergency Medicine_ 16, 1 (2014), 34–40.
  * Solomon (2014) Jacob Solomon. 2014\.  Customization Bias in Decision Support Systems. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’14)_. Association for Computing Machinery, New York, NY, USA, 3065–3074.  https://doi.org/10.1145/2556288.2557211
  * Sridharan et al. (2012) Srinivas Sridharan, Reynold Bailey, Ann McNamara, and Cindy Grimm. 2012.  Subtle Gaze Manipulation for Improved Mammography Training. In _Proceedings of the Symposium on Eye Tracking Research and Applications_ _(ETRA ’12)_. Association for Computing Machinery, New York, NY, USA, 75–82.  https://doi.org/10.1145/2168556.2168568
  * Sriram and Greenwald (2009) N. Sriram and Anthony G. Greenwald. 2009.  The Brief Implicit Association Test.  _Experimental Psychology_ 56, 4 (2009), 283–294.  https://doi.org/10.1027/1618-3169.56.4.283
  * Tao et al. (2020) Liyuan Tao, Chen Zhang, Lin Zeng, Shengrong Zhu, Nan Li, Wei Li, Hua Zhang, Yiming Zhao, Siyan Zhan, and Hong Ji. 2020.  Accuracy and effects of clinical decision support systems integrated with BMJ best practice–aided diagnosis: interrupted time series study.  _JMIR medical informatics_ 8, 1 (2020), e16912.
  * Teovanović et al. (2015) Predrag Teovanović, Goran Knežević, and Lazar Stankov. 2015\.  Individual differences in cognitive biases: Evidence against one-factor theory of rationality.  _Intelligence_ 50 (2015), 75–86.  https://doi.org/10.1016/j.intell.2015.02.008
  * Tversky and Kahneman (1974) Amos Tversky and Daniel Kahneman. 1974.  Judgment under Uncertainty: Heuristics and Biases.  _Science_ 185, 4157 (1974), 1124–1131.  https://doi.org/10.1126/science.185.4157.1124
  * van Berkel et al. (2021) Niels van Berkel, Omer F. Ahmad, Danail Stoyanov, Laurence Lovat, and Ann Blandford. 2021.  Designing Visual Markers for Continuous Artificial Intelligence Support: A Colonoscopy Case Study.  _ACM Trans. Comput. Healthcare_ 2, 1, Article 7 (dec 2021), 24 pages.  https://doi.org/10.1145/3422156
  * van Berkel et al. (2023) Niels van Berkel, Maura Bellio, Mikael B. Skov, and Ann Blandford. 2023.  Measurements, Algorithms, and Presentations of Reality: Framing Interactions with AI-Enabled Decision Support.  _ACM Trans. Comput.-Hum. Interact._ (2023).  https://doi.org/10.1145/3571815
  * van Berkel et al. (2022) Niels van Berkel, Jeremy Opie, Omer F. Ahmad, Laurence Lovat, Danail Stoyanov, and Ann Blandford. 2022\.  Initial Responses to False Positives in AI-Supported Continuous Interactions: A Colonoscopy Case Study.  _ACM Trans. Interact. Intell. Syst._ 12, 1, Article 2 (2022), 18 pages.  https://doi.org/10.1145/3480247
  * Veinott et al. (2013) Elizabeth S. Veinott, James Leonard, Elizabeth Lerner Papautsky, Brandon Perelman, Aleksandra Stankovic, Jared Lorince, Jared Hotaling, Travis Ross, Peter Todd, Edward Castronova, Jerome Busemeyer, Christoper Hale, Richard Catrambone, Elizabeth Whitaker, Olivia Fox, John Flach, and Robert R. Hoffman. 2013\.  The effect of camera perspective and session duration on training decision making in a serious video game. In _2013 IEEE International Games Innovation Conference (IGIC)_. 256–262.  https://doi.org/10.1109/IGIC.2013.6659170
  * Vickrey et al. (2010) Barbara G. Vickrey, Martin A. Samuels, and Allan H. Ropper. 2010.  How neurologists think: A cognitive psychology perspective on missed diagnoses.  _Annals of Neurology_ 67, 4 (2010), 425–433.  https://doi.org/10.1002/ana.21907
  * Wang et al. (2021) Dakuo Wang, Liuping Wang, Zhan Zhang, Ding Wang, Haiyi Zhu, Yvonne Gao, Xiangmin Fan, and Feng Tian. 2021\.  “Brilliant AI Doctor” in Rural Clinics: Challenges in AI-Powered Clinical Decision Support System Deployment. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 697, 18 pages.  https://doi.org/10.1145/3411764.3445432
  * Wang et al. (2019) Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019\.  Designing Theory-Driven User-Centric Explainable AI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3290605.3300831
  * Wang et al. (2020) Ruotong Wang, F. Maxwell Harper, and Haiyi Zhu. 2020.  Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differences. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3313831.3376813
  * Yang et al. (2019) Qian Yang, Aaron Steinfeld, and John Zimmerman. 2019\.  Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–11.  https://doi.org/10.1145/3290605.3300468
  * Yang et al. (2016) Qian Yang, John Zimmerman, Aaron Steinfeld, Lisa Carey, and James F. Antaki. 2016.  Investigating the Heart Pump Implant Decision Process: Opportunities for Decision Support Tools to Help. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 4477–4488.  https://doi.org/10.1145/2858036.2858373
  * Zhang et al. (2015) Yunfeng Zhang, Rachel K.E. Bellamy, and Wendy A. Kellogg. 2015\.  Designing Information for Remediating Cognitive Biases in Decision-Making. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 2211–2220.  https://doi.org/10.1145/2702123.2702239