# Thought Bubbles: A Proxy into Players’ Mental Model Development

Omid Mohaddesi  0000-0002-6245-4003 Northeastern University360 Huntington
Ave.BostonMassachusettsUSA02115<EMAIL_ADDRESS>,  Noah Chicoine
0000-0002-0602-1086 Northeastern University360 Huntington
Ave.BostonMassachusettsUSA02115<EMAIL_ADDRESS>,  Min Gong
0000-0002-8686-3442 Northeastern University360 Huntington
Ave.BostonMassachusettsUSA02115<EMAIL_ADDRESS>,  Ozlem Ergun
Northeastern University360 Huntington Ave.BostonMassachusettsUSA02115
<EMAIL_ADDRESS>,  Jacqueline Griffin  Northeastern University360
Huntington Ave.BostonMassachusettsUSA02115<EMAIL_ADDRESS>,
David Kaeli  Northeastern University360 Huntington
Ave.BostonMassachusettsUSA02115<EMAIL_ADDRESS>,  Stacy Marsella
Northeastern University360 Huntington Ave.BostonMassachusettsUSA02115
<EMAIL_ADDRESS>and  Casper Harteveld  Northeastern University360
Huntington Ave.BostonMassachusettsUSA02115<EMAIL_ADDRESS>
(2023)

###### Abstract.

Studying mental models has recently received more attention, aiming to
understand the cognitive aspects of human-computer interaction. However, there
is not enough research on the elicitation of mental models in complex dynamic
systems. We present _Thought Bubbles_ as an approach for eliciting mental
models and an avenue for understanding players’ mental model development in
interactive virtual environments. We demonstrate the use of Thought Bubbles in
two experimental studies involving 250 participants playing a supply chain
game. In our analyses, we rely on Situation Awareness (SA) levels, including
perception, comprehension, and projection, and show how experimental
manipulations such as disruptions and information sharing shape players’
mental models and drive their decisions depending on their behavioral profile.
Our results provide evidence for the use of thought bubbles in uncovering
cognitive aspects of behavior by indicating how disruption location and
availability of information affect people’s mental model development and
influence their decisions.

mental model development, mental model elicitation, dynamic decision-making,
supply chain, thought bubble

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
2023 CHI Conference on Human Factors in Computing Systems; April 23–28, 2023;
Hamburg, Germany††booktitle: Proceedings of the 2023 CHI Conference on Human
Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg,
Germany††price: 15.00††doi: 10.1145/3544548.3581570††isbn:
978-1-4503-9421-5/23/04††ccs: Human-centered computing Empirical studies in
HCI††ccs: Human-centered computing User studies

##  1\. Introduction

Complex dynamic systems, such as supply chains, comprise a network of
interdependent actors that constantly interact with each other (Ridolfi et
al., 2012). The behavior of these actors results in emergent dynamics and
influences how they interact with the system over time (Bekebrede et al.,
2015; Lukosch et al., 2018). Some actors show distinct decision patterns
(Mohaddesi et al., 2022), and some even exhibit irrational behavior, such as
panic-buying and hoarding, in the context of supply chains (Sterman and Dogan,
2015). Such behaviors and varying decisions have been shown to be the outcome
of different mental models about the underlying system (Gary and Wood, 2011;
Brown et al., 2009). Therefore, it is vital to investigate how decision-makers
form mental models of such complex dynamic systems over time and undisturbed
in the decision context. Such investigation, however, calls for methods for
eliciting mental models and evaluating mental model development over time (Van
Daalen et al., 2014).

In this paper, we present such an elicitation method that we refer to as
thought bubbles. This method evokes thought processes in situ and diegetically
through multiple open-ended prompts over time as part of an interactive
virtual environment, and thus seeks to elicit mental model development in the
context of complex dynamic systems. We use the term mental model development
to refer to a temporal process that happens during the interaction with
complex dynamic systems and over time. The need for a process view stems from
mental models not being static constructs (Landriscina, 2013; Crandall et al.,
2006). They dynamically change depending on new information or perceptions of
the system. Previous studies provide evidence that the interaction forms
behavior (Mohaddesi et al., 2022), and that behavior is the outcome of
cognitive constructs (Klein et al., 1993; Endsley, 2015). Hence, it is not
unreasonable to think that the interaction affects mental model development.
In addition, we leverage interactive virtual environments because they have
been demonstrated to be useful for studying and learning about complex dynamic
systems (Lukosch et al., 2018; Mohaddesi et al., 2020; Bekebrede and Mayer,
2006) and allow for eliciting mental models diegetically, which mitigates the
effect of any out-of-context effects such as pre/post-interaction information
or experimenter bias (Doyle and Ford, 1998).

Eliciting mental models is only the first step in gaining insight into the
cognitive aspect of human actions (Grenier and Dudzinska-Przesmitzki, 2015).
We also need reliable methods for analyzing the elicited mental model
concepts. As our elicited mental models result from textual verbalization of
thought processes, we can leverage qualitative analysis to make sense of our
elicitation (Gero et al., 2020; Jones et al., 2011; Grenier and Dudzinska-
Przesmitzki, 2015). In grounding our qualitative analysis, we must align our
outcomes with existing theories to gain a theoretical understanding of mental
model development. We rely on the Situation Awareness (SA) model, introduced
by Endsley (Endsley, 1995), as a guiding framework for our qualitative
analysis. SA framework is a well-established model that describes people’s
perception of the elements of the environment, the comprehension of their
meaning, and the projection of future states (Endsley, 1988); hence providing
a window into their mental model development (Endsley, 2000).

We test thought bubbles and our qualitative analysis framework using an
interactive virtual environment called gamettes (Mohaddesi et al., 2020) and
conduct two experimental studies (Study 1: n=$115$ and Study 2: n=$135$) in a
supply chain context. These studies are exploratory in nature and serve two
purposes. First, we test whether elicitation via thought bubbles can offer us
a meaningful outlook on the cognitive aspects of decision-making. In both
studies, we examine how experimental manipulations such as disruption location
in the supply chain or the level of information sharing affect mental model
development. Study 2 also investigates mental model development to help
explain cognitive aspects of decisions made by players with different
behavioral profiles. Second, we use the results of our qualitative coding in
Study 1 as a codebook for the qualitative analysis of Study 2, aiming to test
the reliability of SA for analyzing elicited mental model concepts over time.
The following are the contributions of this work:

  * •

We present thought bubbles as a method for collecting qualitative data on
human thought processes diegetically and temporally to elicit mental model
development;

  * •

We show a mixed-method approach for analyzing elicited mental models to gain a
theoretical understanding of cognitive aspects of human decisions and their
mental model formation; and

  * •

We provide evidence on the effect of disruption location and information
sharing on mental model development (Study 1), as well as the effect of
information sharing on the mental model development of players with distinct
behavioral profiles (Study 2).

##  2\. Related Work

In our review of the related literature, we first study the notion of mental
models as a cognitive construct and methods for eliciting mental models. We
then look into the study of mental models in the context of dynamic decision-
making.

###  2.1. Mental Models

Mental models are internal and small-scale representations of the real world
that people rely on in their interaction with external reality (Jones et al.,
2011; Gentner and Stevens, 2014). The notion of mental models, which was first
put forth in 1943 by Craik (Craik, 1943), has been extensively researched by
cognitive psychologists and tested in various contexts. As a cognitive
construct, mental models are compared with the notion of “schemata”, which are
sets of expectations or long-term knowledge structures based on prior
experiences (Brewer and Treyens, 1981; Wilson and Rutherford, 1989; Johnson-
Laird, 1983; Rumelhart, 1984). Psychologists typically refer to both concepts
to better contrast their definitions (Holland et al., 1989; Brewer, 1987;
Rutherford and Wilson, 1991). The difference is that schemata are inflexible
and generic knowledge structures that provide predictive knowledge for routine
situations (Holland et al., 1989; Rutherford and Wilson, 1991). Mental models,
on the other hand, are dynamically formed each time a new data or situation is
perceived, and people form richer and more consistent mental models as they
gain more experience (Crandall et al., 2006). When faced with unfamiliar
domains, people may rely on analogies to extend knowledge from prior
experiences that are well-understood and perceived as similar (Gentner and
Stevens, 2014; Collins and Gentner, 1987).

Prior research also describes how mental models can be used to run mental
simulations (i.e., mentally project into the future) to envision and explore
alternative future states before acting (Crandall et al., 2006; Landriscina,
2013). This is where cognition meets behavior; the mental model construct
directs actions when people rely on mental simulations in interacting with the
environment (Klein et al., 1993). From this perspective, understanding
behavior requires gaining insight into mental models. However, we cannot
directly observe cognitive constructs such as mental models. We need
instruments and methodologies to allow people to externalize their thought
process (Ifenthaler, 2010). With thought bubbles, we aim to gain insight into
the cognitive aspect of human decision-making by eliciting mental models.

Prior research points to the lack of consistent methodologies for mental model
elicitation (Grenier and Dudzinska-Przesmitzki, 2015). The literature refers
to elicitation approaches in two dimensions: (1) techniques used for
extracting thought processes (i.e., verbal, visual, or hybrid); and (2)
elicitation context (i.e., situated vs. non-situated) (Jones et al., 2014).
Verbal elicitation refers to using dialogue or discussion in an interview
procedure. In contrast, visual elicitation is related to using graphics to
represent an individual’s mental model. The hybrid approach concerns a
combination of verbal and visual elicitation (Grenier and Dudzinska-
Przesmitzki, 2015; Cooke, 1994; LaMere et al., 2020). Situated practices
involve eliciting mental models in the context (i.e., diegetic). The non-
situated approach, on the other hand, involves elicitation in a location
removed from the topic of study (Jones et al., 2014). Thought bubbles are a
form of verbal elicitation that we utilize in situ. By allowing players to
provide open-response comments, we leverage the verbalization of players’
thought processes to investigate their mental model development in the context
of dynamic decision-making and without introducing framing bias (Tversky and
Kahneman, 1985; Memon et al., 2013).

###  2.2. Dynamic Decision Making in Interactive Systems

Dynamic decision-making tasks involve taking a series of actions in a dynamic
system over time to achieve some goal (e.g., maximizing total profit)
(Edwards, 1962). Actions are interdependent, which means later actions depend
on the earlier ones, and the environment will change spontaneously or as the
result of earlier actions (Hotaling et al., 2015; Klein et al., 1993). The
study of mental models is fundamental in dynamic decision-making environments.
Most studies in this area are motivated by trying to develop a better
understanding of mental models to improve decision-making (Doyle and Ford,
1998). For example, researchers have studied mental model elicitation from
experts to improve system dynamics models for mental model improvement (Ford
and Sterman, 1998). Others have studied individuals’ understanding of the
decision context and showed how mental models affect human rationality in
dynamic decisions (Rouwette et al., 2004), how different mental models lead to
different decisions and performances (Gary and Wood, 2011), and how higher
similarity between mental models and the decision context results in higher
performance (Ritchie-Dunham, 2001). Most of these studies rely on a simulation
for the decision context and a form of elicitation technique, as described in
the previous section, to capture mental models.

Researchers have proposed various approaches for analyzing elicitation results
and studying mental models. From our research perspective, these approaches
fall into one of three categories: (1) not applicable to dynamic decision-
making, (2) applicable but not practical (e.g., demonstrated by lack of
empirical use), and (3) applicable and practical. As an example of the first,
Johnson-Laird (Johnson-Laird, 1980) studied mental models in connection with
human reasoning though propositional reasoning, deduction, and syllogistic
inference (Johnson-Laird and Bara, 1984; Johnson-Laird et al., 1992; Johnson-
Laird, 1983). Despite its prominence, Johnson-Laird’s view is not immediately
applicable to dynamic decision-making environments because of its lack of
support for studying mental models formed through interaction with the task or
the environment (Staggers and Norcio, 1993; Brown et al., 2009). As for the
second, Richardson et al. (Richardson et al., 1994) developed a theory of
perception, planning, action, and learning specifically for studying mental
models in dynamic decision-making. They followed the idea that mental models
are multifaceted (i.e., comprising three main sub-models: ends model, means
model, and ends-means model (Andersen and Rohrbaugh, 1992)) and linked system
dynamics feedback theory with models of judgment. Unfortunately, there is not
enough evidence in the literature on the empirical use of the theory proposed
by Richardson et al. (Richardson et al., 1994).

Finally, as for the third, Endsley (Endsley, 1995) introduced the theory of
Situation Awareness (SA) in dynamic decision-making and proposed its use for
studying mental models (Endsley, 2000). SA model has been extensively and
empirically researched through numerous studies by Endsley (Endsley, 2015,
2018, 2017, 2021) and others (Chen et al., 2021; Teichmann and Motus, 2021;
Andrews et al., 2022). Therefore, we rely on the SA model for approximating
players’ mental model development over time. This idea of leveraging the use
of SA to characterize mental models has also been examined by other
researchers in inductive reasoning tasks (Zhang et al., 2022). Endsley
describes situation awareness in dynamic decision-making as:

> “[…] the perception of the elements in the environment within a volume of
> time and space, the comprehension of their meaning, and the projection of
> their status in the near future.” (Endsley, 1988)

According to this model, situation awareness is achieved through: (1)
perceiving the status, attributes, and dynamics of elements of the environment
(perception); (2) understanding the situation based on a synthesis of
perceived elements and determining their relevance to decision goals
(comprehension); and (3) the ability to project the future events and dynamics
of the elements of the system (projection). According to Endsley (Endsley,
2000), a situation model 111Here we use Situation Awareness and Situation
Model interchangeably. provides a useful window into people’s mental models.
While people may have a mental model of how a dynamic system (here a supply
chain) works, their interaction with the environment directs how they update
their situation model, which eventually directs the selection and revision of
their mental model (Endsley, 2015). More specifically, mental models direct
attention to key features of the environment and the comprehension,
projections, and expectations. Therefore, a situation model is a key to
understanding the mental model (Endsley, 2000).

Endsley (Endsley, 1988, 1995) also introduced the Situation Awareness Global
Assessment Technique (SAGAT) to measure individuals’ situation awareness.
SAGAT involves stopping the simulation in random intervals and querying the
subjects with specific questions to determine their SA at that particular time
(Endsley, 1988). While we do not use SAGAT, with thought bubbles, we follow
the same idea by prompting players in intervals. Our goal is not to measure SA
using thought bubbles. Instead, we use the SA framework for making sense of
players’ comments during our qualitative coding process resulting from
responding to the thought bubbles deployed in gamettes.

##  3\. Gamettes

We introduced gamettes in our previous study (Mohaddesi et al., 2020), as a
serious game approach for collecting data on behavioral aspects of human
decision-making in supply chain experiments. A gamette is a short game-based
scenario that immerses human decision-makers into a specific situation,
requiring them to make decisions by responding to a dialog or taking actions.
The term gamette is a contraction of “game” with “vignette”, and similar to a
vignette, a gamette aims to provide a brief description of a situation, as
well as to portray someone. Here, we utilize gamettes to understand an
individual’s mental model supporting their decision-making in a drug delivery
supply chain. For this, we use the integrated simulation framework proposed by
Doroudi et al. (Doroudi et al., 2018). This framework comprises a Flow
Simulator for simulating the supply chain dynamics and a gamette environment
for engaging human decision-makers with the simulation by immersing them in a
specific role and particular state of the supply chain (see Figure 1).

Figure 1. The integrated simulation framework.

[Simulation architecture, including gamette and Flow Simulator]The
architecture of the integrated simulation framework shows how the gamette
environment is connected to the Flow simulator. Each gamette environment
comprises a gamette server that communicates with the Flow Simulator. Multiple
gamette clients can communicate with the gamette server. Gamette server stores
all game data in gamette decision storage.

Within this framework, the Flow Simulator is a multi-agent simulation and the
central hub that controls the dynamics of a drug delivery supply chain,
including the flow of information and physical products over time. These
information and physical flows are driven by the decisions and actions taken
by the agents of the system (i.e., manufacturers, wholesalers, and health
centers). The Flow Simulator can run in a standalone mode and without any
human agents, which means that the Flow Simulator simulates the evolution of
the supply chain system and also controls the decisions and actions of the
agents through predefined policies. Alternatively, the Flow Simulator can
simulate the evolution of the supply chain by fetching information from a
gamette client that captures the decisions of human players.

Following the approach in our prior work (Mohaddesi et al., 2020), we created
a gamette with StudyCrafter222StudyCrafter is a platform where users can
easily create, play, and share gamified projects. It can be accessed at
https://studycrafter.com, where players take the role of a wholesaler in a
drug delivery supply chain. Details of the gamette design are described in
Section 5. The same gamette is used for Study 1 and Study 2; the difference is
in their experimental design (i.e., Study 1 considers the disruption location
and various forms of information sharing; Study 2 considers different
behavioral profiles for decision-makers and how they respond to information
sharing).

Figure 2. Thought bubble design in the gamette as part of the meeting scene .
Players experience the meeting scene every four weeks during the gameplay (8
times in total), starting on Week 24. Each time, players are first provided
with a performance review  and then face the thought bubbles . An NPC delivers
the thought bubble prompt  in an open-ended format to evoke players to reflect
on their performance by asking “How do you think we are doing Kate?” . Players
provide their open-ended responses in the input area . Note that the gameplay
phase starts on Week 21 after the tutorial ends.

[Thought Bubble design]Design of thought bubbles and its characteristics where
players review performance and face the thought bubbles prompt. The prompt is
delivered by NPC that asks, ”How do you think we are doing Kate?” to evoke
players to reflect on their performance. Players can provide their response in
an open-ended format in an input area.

##  4\. Thought Bubbles

We designed thought bubbles with three main characteristics: (1) diegetic, (2)
verbal and open-ended input, and (3) over time. Figure 2 demonstrates the
design of thought bubbles and its characteristics. To ensure diegetic data
collection, we included thought bubbles (see  in Figure 2) as part of a
recurrent meeting scene in the gamette where players would first review their
performance (see  in Figure 2). This feedback is provided to players through
interaction with other Non-Player Characters (NPCs) during the gameplay phase
and includes factual information (i.e., historical graphs and the current
state of supply chain parameters), without introducing any form of bias into
the players’ performance. After performance review, the Boss NPC would deliver
the thought bubble prompt (see  in Figure 2). In terms of aesthetics, we did
not design a thought bubble in literal terms. Instead, we chose to request
players’ thoughts via NPC dialogue as it felt natural to request this prompt
as part of the meeting scenario, where the results are discussed, helping keep
players in the context of the game.

To evoke players to reflect on their experience, we prompted them by asking,
“How do you think we are doing Kate?” (see  in Figure 2) and allowed them to
provide open-ended input (see  in Figure 2). We chose an open-ended question
and response format to mitigate framing bias (Tversky and Kahneman, 1985) and
help players to articulate their thought process by engaging associative
memory (Memon et al., 2013). The analyses we present in this paper center
around players’ responses to this question. Finally, players experienced the
meeting scene once every four weeks (8 times total) throughout the entire game
(see  in Figure 2), allowing us to collect data on their thought process over
time. We chose a four-week interval because, in doing so, we could frame the
experience as a monthly meeting scene which felt more natural and resulted in
the minimum distraction of players from their decision-making task. Moreover,
our supply chain experiment includes a lead time of two weeks for ordering
decisions (one week for orders to be processed by the manufacturer and one
week for players to receive shipments). By considering a four-week interval,
we allow players to have sufficient experience and observe the short-term
outcome of their decisions before each thought bubble. A video preview of the
gamette and the thought bubbles is available in the OSF repository
(https://osf.io/btfzx/?view_only=8211d2334d5440a0b75ae947811cb845).

##  5\. Study 1

The motivation behind Study 1 is to understand the role of human behavior in
pharmaceutical supply chains experiencing drug shortages by studying the
mental model development of decision makers. Specifically, we aim to
elicit—through the thought bubbles—an individual decision maker’s mental model
and how this changes over time as they interact with the environment. To this
end, we simulated a pharmaceutical supply chain network using the Flow
Simulator and used a gamette to immerse human participants into a specific
role within this network. We considered a supply chain network that includes
two manufacturers, two distributors, and two health centers. Figure 3
illustrates the network structure and the direction of shipments between each
entity in this supply chain. We use this network because it permits the
analysis of the supply chain network in conjunction with the behavioral
dynamics that ensue from having multiple agents within each echelon (Doroudi
et al., 2020).

###  5.1. Methods

####  5.1.1. Hypotheses

Drug shortages can occur for different reasons. However, most shortages can be
traced back to supply chain disruptions (Tucker et al., 2020b). Many
researchers have attempted to mitigate the impact of drug shortages through
increasing supply chain resilience (Tucker et al., 2020a), implementing
decision support systems (Chihaoui et al., 2019), and developing optimal
inventory management and ordering policies (Azghandi et al., 2018). Another
key, and often understudied, element is the behavior of human decision-makers,
which can prolong or aggravate the effect of shortages (Doroudi et al., 2020).
Such behaviors are exemplified in our previous research where we showed people
tend to deviate from optimal order suggestions (Mohaddesi et al., 2020), and
that these behaviors are attributed to hoarding and panic buying when facing
shortages (Mohaddesi et al., 2022). This is due to people’s tendency to ignore
the temporal dynamics of the system when forming mental models of the supply
chains (Fu and Gonzalez, 2006). Additionally, different disruption locations
can create different local dynamics for each role within the supply chain
(Rong et al., 2008; Sarkar and Kumar, 2016). These local dynamics present
unique experiences for players, ultimately affecting their mental models and
influencing their inventory management decisions, which contribute to the
propagation of shortage effects throughout the rest of the supply chain. So
our aim is to explore how players’ mental models are affected when they are in
different positions relative to the supply chain disruption, and hypothesize
that:

H1. People show differences in their mental model development of a disrupted
supply chain, depending on the disruption location.

Previous research on drug shortage management also points to insufficient
information sharing between stakeholders in pharmaceutical supply chains as a
source for disruption propagation (Yang et al., 2016), and that many of the
costs associated with managing shortages in such supply chains can be
mitigated by increasing collaboration and sharing information (Pauwels et al.,
2015). Hence, information sharing is considered as one of the essential
strategies to improve the resilience of supply chains (Iyengar et al., 2016).
Information that is shared between supply chain stakeholders adds to the
agents’ management experiences in shaping their mental models of the
surrounding supply chain dynamics. Thus, it is important to investigate the
influence of information sharing on mental model development and the behavior
of decision-makers. We expect that sharing various levels of information
results in the development of different mental models. Hence:

H2. People exhibit differences in mental model development of a disrupted
supply chain, depending on the level of information being shared with them.

Figure 3. Supply chain network structure that included two manufacturers, two
wholesalers and two health centers. Players play the role of Wholesaler 1
using the gamette.

The structure of supply chain network. [Supply chain network
structure]Structure of the supply chain network, including two manufacturers,
two wholesalers, and two health centers. Players play the wholesaler role in
the gamette.

####  5.1.2. Experimental Design

The experimental setting for our study is summarized in Table 1. Players in
all conditions play the role of Wholesaler 1 (WS1; see Figure 3), and all
other supply chain roles are assigned to agents that the Flow Simulator
controls. The simulation agents make decisions based on order-up-to-level
policy. According to this policy, each agent orders or produces enough product
to bring their inventory position to a predefined level based on a periodic
review policy with zero fixed costs (Snyder and Shen, 2011). We considered
different scenarios for testing the effect of disruption location on players’
mental models (H1), where disruption can happen in either Manufacturer 1 (MN1)
or Manufacturer 2 (MN2). We created these disruptions in the form of a
manufacturing shutdown which reduces the production capacity of the disrupted
manufacturer by 95%.

Table 1. Summary of the experiment settings for Study 1.

|  Condition 1
(MN1/No-Info) |  Condition 2
(MN2/No-Info) |  Condition 3
(MN1/Partial-Info) |  Condition 4
(MN2/Partial-Info) |  Condition 5
(MN1/Complete-Info) |  Condition 6
(MN2/Complete-Info)
---|---|---|---|---|---|---
Player Role |  WS1 |  WS1 |  WS1 |  WS1 |  WS1 |  WS1
Disrupted Manufacturer |  MN1 |  MN2 |  MN1 |  MN2 |  MN1 |  MN2
Information Sharing |  No |  No |  MN1 inventory |  MN1 inventory |  MN1 inventory + HCs behavior and delivery rates |  MN1 inventory + HCs behavior and delivery rates
No. of Participants |  17 |  19 |  20 |  21 |  18 |  20

We also designed the two health-center agents (HC1 and HC2) in the Flow
Simulator with different ordering behaviors to generate different local
dynamics for the WS1 role based on disruption location. The difference in
health-center behavior is related to how each health-center agent splits
orders between wholesalers. While both health-centers receive a constant
demand, the HC1 agent splits its orders to its upstream agents by means of a
trustworthiness measure, meaning that HC1 orders less from the upstream
wholesaler that fails to deliver drugs consistently. HC2, on the other hand,
splits its orders always equally regardless of its wholesalers’
trustworthiness. This trust-based behavior of HC1 particularly affects players
during the shortage period (weeks 32-36) but differently depending on
disruption location. When MN1 is disrupted, in addition to incurring stockout
cost for not satisfying demand, players will also experience a decrease in
HC1’s demand. When MN2 is disrupted, players experience an increase in HC1’s
demand. According to these dynamics, we considered different options for
testing the effect of information sharing on mental model development (H2):
(1) without; (2) with information sharing on MN1 inventory (Partial Info); and
(3) with information sharing on MN1 inventory, health-centers’ ordering
behavior, and delivery rates to each health-center (Complete Info). Figure 4
and Figure 5 illustrate our designs for different levels of information
sharing.

####  5.1.3. Participants

We recruited 115 participants ($87$ males, $26$ females, and $2$ not stated).
Participants were full-time students enrolled in undergraduate and graduate
level courses of Logistics and Supply Chain Management at Northeastern
University. The age range is 18 to 32 years (M=22.72, SD=2.01).

####  5.1.4. Incentive Design

Previous research points to the importance of incentives in conducting
experimental research (Katok, 2018). To motivate participants to engage with
the task and perform well, we offered them a monetary incentive ($50) which
was gifted through a raffle. Players who performed better had a higher chance
in the raffle. Each participant received one ticket for completing the game
plus one ticket for every $1000 in-game profit that they made, more than the
average profit of all other players.

(a) No information sharing on MN1 inventory

(b) Sharing MN1 inventory

Figure 4. The ordering scene as part of the decision task in (a) No-Info vs.
(b) Partial and Complete Info conditions. Note that in (b) “Manufacturer’s
Inventory” is mentioned at top while in (a) this information is not displayed.
Players in both groups receive an order suggestion.

[Difference between ordering scenes in conditions with and without information
sharing in MN1 inventory]Ordering scenes in different conditions look
identical except for the manufacturer’s inventory that is provided in the
information sharing conditions.

(a) No information sharing on delivery rates to HCs

(b) Sharing delivery rates to HCs

(c) Information sharing on HC1 behavior

(d) Information sharing on HC2 behavior

Figure 5. The allocating scene (as part of the decision task) and tutorial
scenes in (a) No-Info and Partial Info vs. (b-d) Complete Info conditions.
Note that in (b) “Delivery Rate to each HC” is mentioned in the allocating
scene while in (a) this information is not displayed. (c) and (d) are also not
presented to players in No-Info and Partial Info conditions.

[Difference between allocating and tutorial scenes in different
conditions]Allocating scenes in different conditions look identical except for
the Delivery Rate to each HC that is provided in the complete information
sharing conditions. In addition, the ordering behavior of health-centers are
only displayed to players in Complete Info conditions during the tutorial
phase.

####  5.1.5. Procedure

All participants first visited the study website, where they were formally
briefed about the experiment and its purpose. By starting the game, each
participant was randomly assigned to one of the six conditions. Table 1 shows
the number of participants assigned to each condition. The gamette in all
conditions looked the same in all aspects except for the ordering scene, the
allocating scene, and part of the tutorial that provided information on
health-centers’ ordering behavior. Participants played the role of a character
named Kate who was hired as a supply chain director in a wholesaler company.
At the beginning of the game, an NPC (Kate’s boss) expresses that the game’s
goal is maximizing the company’s profit by minimizing the inventory and
stockout costs and maximizing sales revenue. The NPC also informs players,
through dialogue, about the sales revenue and cost breakdown, and the lead
time of two weeks for orders (one week for orders to be processed by the
manufacturer and one week for players to receive shipments).

Each participant first played four weeks of tutorial to familiarize themselves
with the game (i.e., how to gather information) and was given instructions
about ordering from the manufacturer and allocating to health-centers.
Participants were asked to make an ordering decision at each period, but only
make allocating decisions when their inventory level was lower than their
total demand. In cases where they had enough inventory, the game automatically
allocated drugs to each health-center. After finishing the tutorial, all sales
and cost data was reset, and they played the game for 35 game weeks, starting
at Week 21 of the simulation. After placing the order on Week 24, rather than
moving to Week 25, the game transited players to the monthly meeting scene
(Figure 2) where they were queried via the thought bubble after receiving
information on their performance via NPC dialogue. Players transited to this
meeting scene every four weeks, with Week 52 being their last meeting. The
disruption started on Week 28 and ended on Week 33. We framed the disruption
as a manufacturing shutdown due to COVID-19.

At each period, participants received a shipment from their upstream
manufacturer and could review the inventory, demand, and backlog information.
Next, if they had limited inventory compared to demand, they were asked to
select one of the presented allocation policies: (1) allocate to HC1 first,
(2) allocate to HC2 first, or (3) allocate proportionally to each health
center’s demand. Finally, they received an order suggestion according to the
order-up-to-level policy for making an ordering decision. They had the option
to order the suggested amount or modify it. The gamette sends player decisions
to the Flow Simulator, which moves the simulation to the next period and sends
back the updated parameters for the next round to the gamette. After playing
for 35 weeks, through dialogue, an NPC debriefed participants about the study,
experimental setting, and their performance.

####  5.1.6. Data Analysis

To analyze players’ responses to the thought bubbles prompt (“How do you think
we are doing Kate?”), we first applied Initial Coding (Saldaña, 2021), where
we qualitatively coded the open-response comments from players. We followed an
inductive approach in our coding practice, where we used a combination of in
vivo and constructed codes. Three researchers coded all players’ comments
across the eight prompts, leading to 94 initial codes. In further making sense
of this data, we considered how these initial codes can lend insight into
mental models by looking into existing theories and frameworks. We leveraged
the Situation Awareness (SA) model introduced by Endsley (Endsley, 1995) (see
Section 2.2) to categorize our initial codes. We used three levels of SA to
determine whether our generated codes reflect players’ perception of the
supply chain (in the game), their comprehension of the state relating to the
game’s goals, or their ability to project the future state of the supply
chain.

After assigning each code to one of the three SA levels, we determined what
aspect of the supply chain (i.e., topic) is described in players’ responses
(e.g., profit, cost, inventory, demand, order, general, etc.) and how it is
described through either a verb or adjective (i.e., description/action). The
descriptions represented the rate of change (e.g., increase, decrease,
constant), connotation (e.g., positive, negative, neutral) or specific actions
(e.g., over-order, improve, anticipate, etc.) related to the supply chain
topic. For example, we determined the comment “inventory cost goes high” to
reflect a player’s perception (SA level) of an increase (description) in
inventory cost (topic). The process took place by examining the codes through
several group meetings. We did not seek to calculate an inter-rater
reliability (IRR) as our generated codes were the process and not the product
(McDonald et al., 2019). We use the codes as a passage to get to the aspects
of situation awareness and, from there, to get insight into mental model
development. Table 2 summarizes the result of our qualitative analysis.

To understand players’ mental model development, we performed quantitative
analyses in three steps: (1) counting the number of codes representing each
aspect of SA in player comments and performing hypothesis testing to
investigate associations; (2) studying patterns by examining the counts of
codes temporally and across the eight measurements; and (3) explaining the
patterns and associations by examining the topics/descriptions of SA aspects
as represented in players’ comments. All comments, codes, and generated
categories (i.e., topics and descriptions/actions) through our qualitative
analysis are available in the OSF repository
(https://osf.io/btfzx/?view_only=8211d2334d5440a0b75ae947811cb845). In Section
5.2 and in Table 2, we refer to participant quotes as “(Player ID, gender,
age).” For example, a female player aged 23 with player ID 85 who participated
in Study 1 would be displayed as “(PL1-85, female, 23).”

Table 2. Aspects of mental models identified by applying SA to qualitative
responses of players to the thought bubbles.

SA Level |  Topic |  Description/Action |  Example
---|---|---|---
Perception |  inventory cost, backlog cost, costs, profit, inventory, demand, backlog, order |  increase, decrease, consistent, zero , over-order, under-order |  “there is uncertainity [sic] in supply, thats [sic] why our inventory cost goes high, we need to talk with supplier.”–(PL1-18, male, 23)
“I think we did pretty well. The profit was constant and the cost was minimal.
All orders were satisfied.”–(PL1-112, female, 23)”
Comprehension |  general, inventory, demand, backlog, supply line, order |  positive, negative, neutral, uncertain |  “Awful. Need to introspect and make the necessary changes.”–(PL1-3, male, 24)
“We are doing fantastic and meeting demands regularly with enough saftey [sic]
stock in hand to satisfy sudden hike in the demand.”–(PL1-2, male, 24)”
Projection |  general, profit, inventory, demand, backlog, order, allocation |  improve, anticipate problem/uncertainty, increase, decrease, constant, uncertain, proportionally, HC with higher delivery rate, HC2 |  “We have more inventory since the manufacturer 2 is closed due to COVID-19 so we have to prepare for the incresing [sic] of Healthcare 2 demand.”–(PL1-47, female, 25)
“Profit is going down and performing poorly due to high backorder costs. I
need to order more.”–(PL1-98, male, 23)”

###  5.2. Results

Participants wrote on average six words per prompt (Mdn=$3$, IQR=$7$). The
thought bubble prompt was left unanswered in 5.5% of the instances across the
eight measures, spreading across 19 participants. The mean time players spent
in the meeting scene was 55.1 seconds (SD=69.7), which included the time they
reviewed their performance and responded to the prompt. We report these
statistics to illustrate how thought bubbles are utilized. The following
sections describe what we learned from players’ responses in Study 1.

We used the outcome of our qualitative analysis and mental model development
to compute the frequency of players’ comments that reflected each aspect of SA
depending on the disruption location (see Table 3). Chi-square test of
independence revealed significant association between disruption location and
players’ mental models, as captured by SA via thought bubbles
(${\chi}^{2}_{2,1011}=12.047$, $p=.002$), and indicated a small effect
(Cramer’s $V=.109$) (Cohen, 2013). More specifically, players who experienced
a disruption in their own supplier (i.e., MN1) wrote significantly more
comments reflecting their comprehension of the environment ($p<.001$). These
comments mostly indicate general negative expressions, such as:

> “Awful. Need to introspect and make the necessary changes.”–(PL1-3, male,
> 24)
> “very bad.”–(PL1-66, female, NA)
> “We’re doing terribly since our supplier is unable to provide more than 20
> shipments per weeky [sic] order.”–(PL1-74, male, 21)
> “We are not recieving [sic] orders, that’s big problem.”–(PL1-13, male, 24)

Higher comprehension frequency in the comments of this group (i.e., MN1
disruption) does not necessarily reflect a better understanding of the
environment and instead shows the level of uncertainty or inability to make
sense of the environment towards goals. We also investigated the progression
of mental models over time (see Figure 6-A2). We realized that most negative
expressions of players with MN1 disruption are mainly observed at the end of
the shortage period (i.e., Week 36). This can indicate that the generated
dynamics by disruption in MN1 create more uncertainty for players to the
extent that they primarily comprehend their environment as negative.
Therefore, the source of disruption affects each player’s mental model
development relative to their comprehension.

Table 3. Contingency table of frequency of players’ comments representing each
aspect of Situation Awareness (SA), separately for implemented manipulations
and across the eight thought bubbles prompts in Study 1.

|  | SA Level |
---|---|---|---
Manipulation |  | Perception | Comprehension | Projection | Total
Disrupted Location |  |  |  |  |
MN1 ($n=55$) | Count | 69.0 | 358.0$\ast\ast$ | 38.0$\ast$ | 465.0
| Expected | 79.1 | 334.3 | 51.5 | 465.0
MN2 ($n=60$) | Count | 103.0 | 369.0$\ast\ast$ | 74.0$\ast$ | 546.0
| Expected | 92.9 | 392.6 | 60.4 | 546.0
Information Sharing |  |  |  |  |
Complete ($n=38$) | Count | 60.0 | 244.0 | 34.0 | 338.0
| Expected | 57.5 | 243.0 | 37.4 | 338.0
Partial ($n=41$) | Count | 80.0$\ast$ | 246.0$\ast\ast$ | 51.0 | 377.0
| Expected | 64.13 | 271.1 | 41.7 | 377.0
No-Info ($n=36$) | Count | 32.0$\ast\ast$ | 237.0$\ast\ast$ | 27.0 | 296.0
| Expected | 50.3 | 212.8 | 32.8 | 296.0
Total | Count | 172.0 | 727.0 | 112.0 | 1011.0
| Expected | 172.0 | 727.0 | 112.0 | 1011.0
Counts show the presence of each SA level in players’ responses. Expected
represents the expected counts for the Chi-square test under the null
hypothesis (i.e., no association between manipulations and SA levels.)
$\ast$Chi-square Post-hoc test shows significant association between
experimental manipulations and SA level at $\alpha=0.05$ with Bonferroni
correction.
$\ast\ast$Chi-square Post-hoc test shows significant association between
experimental manipulations and SA level at $\alpha=0.01$ with Bonferroni
correction.

Our analysis also showed that a disruption in MN1 is associated with
significantly lower projection, as evidenced by players’ comments ($p=.003$).
Such discrepancy can be mainly attributed to the comments at the end of the
shortage period (see Figure 6-A3), where comments from players with a MN1
disruption in Week 36 reflect significantly fewer projections ($p=.002$). As
players with a MN2 disruption experience an increase in their demand, their
projections represent topics such as “anticipating uncertain demand”,
“anticipating demand to increase,” or “planning to over-order”:

> “Doing well so far. I expect it will be difficult once the demand becomes
> uncertain due to covid19.”–(PL1-12, male, 25)
> “demand from HC1 is increasing and we have to supply accordingly.”–(PL1-56,
> male, 24)

Finally, the chi-square post-hoc test did not show a significant difference in
the frequency of players’ perceptions depending on disruption location when
considering all thought bubble responses. However, we found that players with
MN1 disruption represent significantly lower perception in their responses
during Week 36 ($p=.005$) (see Figure 6-A1). Players experiencing a MN1
disruption seem to only perceive the “increase in their backlog”, “increase in
backlog cost”, and “decrease in profit”:

> “So bad, first I wanted to keep the demand of HC1, but then I found more
> demand increase more backlogs which cost more money.”–(PL1-7, male, 22)
> “Net profit is negative because of backlog cost.”–(PL1-113, male, NA)

The disruption location affects the state of the supply chain for a given
point in time such that players in different groups experience different
dynamics. Our results show that these emergent dynamics caused by different
disruption locations drive the development of different mental models in how
players comprehend the environment and project into the future in general, and
how they perceive the system when facing a shortage—thus providing support for
H1.

####  5.2.1. Mental Model Development vs. Level of Information Sharing (H2)

Chi-square test of independence also showed a significant association between
the level of information and mental model development
(${\chi}^{2}_{4,1011}=19.172$, $p<.001$). The effect size for this finding was
small (Cramer’s $V=.097$). The results of the chi-square post-hoc test (see
Table 3) did not reveal a significant association between complete information
sharing and the development of mental models. However, our results indicated
that the responses of players who only received information on their supplier
inventory (i.e., partial information) represented significantly higher
perception ($p=.003$) and significantly lower comprehension ($p<.001$). As for
the perception, players in the Partial-Info group seem to speak to almost all
topics more frequently than players in No-Info group, as exemplified in:

> “We are in very high loss but may fulfil their requierments [sic] as
> manufacturing unit have more units readily available.”–(PL1-44, male, 22)
> “we have done really well over the past 2-3 weeks. we’ve been able to keep
> up with demand while maintaining low costs. I will look to keep the
> inventory low for the time being.”–(PL1-73, male, 21)
>

The higher perception of players in this group is also reflected in the
average count of comments over time. Players in the Partial-Info group show an
increase in the average number of comments related to perception when the
shortage starts in Week 32 (see Figure 6-B1). On the other hand, the lower
frequency of comments related to comprehension in the Partial-Info group is
mainly reflective of general positive comments:

> “the past several weeks have been good, although manufacturer 2’s accident
> in Puerto Rico may change things up…”–(PL1-73, male, 21)
> “Its [sic] look market is working good but still we keep high inventory due
> to uncertainity”–(PL1-44, male, 22)

These players seem to be more cautious in interpreting the environment as
positive because of having access to partial information. Finally, chi-square
post-hoc test indicated that not having access to information is associated
with significantly lower perception ($p<.001$) and significantly higher
comprehension ($p<.001$). Higher comprehension of players in the No-Info group
reflects codes “general-positive” and “general-negative”. As the results
indicate, not having access to information makes it challenging for players to
make sense of the environment. On the other hand, partial information directs
players’ attention to perceive more, and complete information has no effect,
perhaps due to information overload (Endsley, 1995). These results show
partial evidence for H2, where information sharing on supplier inventory
affects mental model development of disrupted supply chains.

Figure 6. Progression of different aspects of situation model over time
depending on disruption location and level of information sharing in Study 1.
The vertical dashed line in each graph shows the time players were notified
about the disruption. The highlighted area illustrates the shortage period due
to manufacturing disruption. For each experimental group, count ratio was
calculated by dividing the number of comments that reflect each SA aspect by
total number of players in that group.

[lineplot showing average count of SA concepts in Study 1]six lineplots
showing the average count of SA concepts in Study 1 for different disruption
locations and information sharing levels

###  5.3. Summary

Study 1 served as the foundation for exploring the use of thought bubbles for
eliciting mental model development. We obtained a codebook (Table 2) by
examining players’ comments through qualitative analysis and grounding it in
SA theory. This codebook represented the aspect of players’ situation models
in our decision-making task, which has shown to be directly related to mental
models (Endsley, 2015). It is crucial to test the reliability of this approach
for eliciting mental model concepts. Hence, we will use our codebook to
qualitatively code responses to the thought bubbles prompt for new players in
Study 2. Study 1 also demonstrated how experimental manipulations affect
mental model development over time. More specifically,

  * •

a disruption in players’ suppliers affects their mental model development by
directing them to comprehend the environment as unfavorable and projecting
less about how the future unfolds.

  * •

receiving information on supplier inventory directs players’ attention to key
features of the environment (i.e., more perception) and causes them to be more
cautious in how they interpret the situation.

  * •

sharing no information causes players to perceive the elements of the system
less and face difficulty in making sense of the environment.

In examining mental model development in Study 1, however, we did not consider
the observed behavior of players. Therefore, in the subsequent study, we
examine if the observed decisions of players—as reflected in their behavioral
profiles—are the outcome of different mental model development processes.

##  6\. Study 2

To get insight into how mental models drive human decisions, we need to
investigate another key aspect influencing mental model development:
individual differences. Prior research provides evidence of different decision
patterns among supply chain decision makers (Mohaddesi et al., 2022; Sun et
al., 2016). Therefore, we conducted Study 2 to explain the cognitive aspect of
decisions made by players with different behavioral profiles, while testing
the reliability of our generated codebook in Study 1. We also made some
changes to the experimental setting of Study 1. First, since we did not find
evidence for the effect of complete information sharing, we decided to drop
that as a manipulation for Study 2. Second, we decided to limit the disruption
location to MN1 to isolate the generated dynamics for easier comparison of
mental model developments across players with distinct behavioral profiles.

###  6.1. Methods

####  6.1.1. Hypotheses

Individuals show different behaviors which might be affected by various
factors (Appelt et al., 2011). In the context of dynamic decision-making, one
can attribute these behavioral differences to the changes in mental model
development (Salvendy, 2012). The literature around SA also points to
individual characteristics affecting SA (Endsley, 2015; Endsley and Garland,
2000), which in turn can affect mental model development. Therefore, we expect
to see associations between observable behavioral profiles and mental model
development as captured via thought bubbles. Hence, we hypothesize:

H3. People show differences in mental model development of a disrupted supply
chain dynamics, depending on their behavioral profile.

We retrieve behavioral profiles following the methods we proposed in our
previous work (Mohaddesi et al., 2022) (see 6.1.4) to test this hypothesis.
Furthermore, in continuation of Study 1, we investigate how information
sharing affects mental model formation for different behavioral profiles.
While we did not find significant association between complete information
sharing and SA levels, we found that access to partial information (i.e.,
supplier’s inventory level) is associated with significantly higher perception
and lower comprehension. In our previous study, we provided empirical evidence
on how information sharing affects the behavior of players with different
decision patterns (Mohaddesi et al., 2022). Consequently, we expect the effect
of information sharing on observable actions to be reflected in players’
mental model development as captured via thought bubbles. Hence:

H4. People with different behavioral profiles show differences in mental model
development of a disrupted supply chain, depending on the level of information
being shared with them.

We test these hypotheses using the same approach as in Study 1.

####  6.1.2. Experimental Design

Table 4 summarizes the experiment settings for Study 2, where we considered a
disruption in MN1 and two options for information sharing: (1) without, and
(2) with information sharing on MN1 inventory. This dataset contains players’
responses to the thought bubbles prompt collected following the same
procedures described in Section 5.1.5. All other aspects, including supply
chain roles and disruption settings, are identical to Study 1 (see Section
5.1.2).

####  6.1.3. Participants

Through Prolific 333www.prolific.co, we recruited 135 (57 males, 74 females, 2
non-binary, 2 not stated) online participants. We only allowed participation
for working professionals who had an undergraduate degree or higher and
reported English as their first language or were fluent in it. Each
participant spent, on average, 58 minutes (SD=19.35) playing and received a
$7.5 reward for their participation. The age range is 21 to 71 years (M=34.23,
SD=9.86). We provided participants with the same incentive as in Study 1 (see
Section 5.1.4) and followed the same procedures (see Section 5.1.5).

Table 4. Summary of the experiment settings in Study 2.

| Player Role | Disrupted Manufacturer | Information Sharing | No. of Participants
---|---|---|---|---
| Hoarders | Reactors | Followers | Total
Condition 1 (No-Info) | WS1 | MN1 | No | 31 | 21 | 9 | 61
Condition 2 (Info) | WS1 | MN1 | Yes | 25 | 27 | 8 | 60
No. of Participants |  |  |  | 56 | 48 | 17 | 121

####  6.1.4. Data Analysis

The three researchers involved in data analysis of Study 1 qualitatively coded
players’ open-response comments (a total of 1093 comments) by using the
previously generated categories as a codebook (see Table 2). Each researcher
used a tuple in the form ¡SA Level, Topic, Discription/Action¿ to code each
comment within the dataset. In case a player’s comment reflected multiple
topics or SA aspects simultaneously, we used multiple tuples to represent
them. Our goal was to evaluate whether data are being interpreted in the same
way relative to the previous analysis in Study 1. Therefore, per the
recommendations of McDonald et al. (McDonald et al., 2019), we attempted to
measure inter-rater reliability (IRR) by calculating Fleiss’ kappa ($\kappa$)
(Fleiss, 1971). After the first round of coding, we obtained $\kappa$=$0.75$.
While this showed a relatively high agreement rate, we examined the codes once
more to address sources of disagreement. After our second attempt, we obtained
$\kappa$=$0.84$, which we considered sufficient for our hypothesis testing.

The steps for our quantitative analysis were similar to those mentioned in
Study 1 (see 5.1.6). However, to decide which set of codes to use for the rest
of our analysis, we considered two heuristics: (1) using the majority vote
(i.e., codes in which two out of three coders agreed upon), and (2) using
codes from the master coder (i.e., first author) in addition to the majority
vote. During our analysis, we realized that both heuristics show similar
results. Therefore, for simplicity, we chose to move forward with the first
heuristic. Players’ comments and qualitative codes for Study 2 can be accessed
in the OSF repository
(https://osf.io/btfzx/?view_only=8211d2334d5440a0b75ae947811cb845).

To obtain the behavioral profile of players, we followed the methods from our
prior study (Mohaddesi et al., 2022). We first filtered outlier players
(n=$14$) and then used a Hidden Markov Model (HMM) to infer players’ response
modes in their deviation from order suggestions (see Figure 4 for an example
of order suggestion). Finally, through cluster analysis of sequences of
response modes, we found three behavioral profiles: (1) Hoarders who deviate
from order suggestions and over-order more frequently, (2) Reactors who follow
order suggestions but react to the disruption news and deviate from
recommendation only after disruption, and (3) Followers who almost always
follow order suggestions. For more detailed description of the methods for
retrieving behavioral profiles we refer to (Mohaddesi et al., 2022). Table 4
shows the count of players characterized by each behavioral profile across the
two conditions. Moving forward, we investigate the mental model development
(as captured with thought bubbles) of these three profiles in testing our
hypotheses.

###  6.2. Results

Similar to Study 1, we investigated how thought bubbles were utilized among
Study 2 participants. On average, players wrote five words in each of their
comments (Mdn=$2.00$, IQR=$5.25$). The thought bubble prompt was left
unanswered in 3.8% of the instances across the eight measures, which were
spread across 12 players. Players spent, on average, 56.4 seconds (SD=47.3),
where they reviewed their performance and responded to the prompt.

Table 5. Contingency table of frequency of players’ comments representing each
aspect of Situation Awareness (SA), separately for implemented manipulations
and across the eight thought bubbles prompts in Study 2.

|  |  | SA Level |
---|---|---|---|---
| Profile/Manipulation |  | Perception | Comprehension | Projection | Total
Behavior Profile | Hoarder | Count | 88.0 | 387.0$\ast$ | 56.0 | 531.0
| Expected | 75.5 | 404.8 | 50.6 | 531.0
Reactor | Count | 60.0 | 333.0 | 42.0 | 435.0
| Expected | 61.8 | 331.6 | 41.5 | 435.0
Follower | Count | 4.0$\ast\ast$ | 95.0$\ast\ast$ | 4.0 | 103.0
| Expected | 14.6 | 78.5 | 9.8 | 103.0
Information Sharing per Behavior Profile | Hoarder |  |  |  |  |
Info ($n=25$) | Count | 57.0$\ast\ast$ | 163.0$\ast\ast$ | 36.0$\ast$ | 256.0
| Expected | 42.4 | 186.5 | 26.9 | 256.0
No-Info ($n=31$) | Count | 31.0$\ast\ast$ | 224.0$\ast\ast$ | 20.0$\ast$ | 275.0
| Expected | 45.5 | 200.4 | 29.0 | 275.0
Reactor |  |  |  |  |
Info ($n=27$) | Count | 21.0$\ast$ | 185.0$\ast\ast$ | 14.0 | 220.0
| Expected | 30.3 | 168.4 | 21.2 | 220.0
No-Info ($n=21$) | Count | 39.0$\ast$ | 148.0$\ast\ast$ | 28.0 | 215.0
| Expected | 29.6 | 164.5 | 20.7 | 215.0
Follower |  |  |  |  |
Info ($n=8$) | Count | 3.0 | 46.0 | 2.0 | 51.0
| Expected | 1.9 | 47.03 | 1.9 | 51.0
No-Info ($n=9$) | Count | 1.0 | 49.0 | 2.0 | 52.0
| Expected | 2.0 | 47.9 | 2.01 | 52.0
Total | Count | 152.0 | 815.0 | 102.0 | 1069.0
| Expected | 152.0 | 815.0 | 102.0 | 1069.0
Counts show the presence of each SA level in players’ responses. Expected
represents the expected counts for the Chi-square test under the null
hypothesis (i.e., no association between manipulations and SA levels.)
$\ast$Chi-square Post-hoc test shows significant association between
behavioral profiles or experimental manipulations and SA level at
$\alpha=0.05$ with Bonferroni correction.
$\ast\ast$Chi-square Post-hoc test shows significant association between
behavioral profiles or experimental manipulations and SA level at
$\alpha=0.01$ with Bonferroni correction.

####  6.2.1. Mental Model Development vs. Behavioral Profiles (H3)

We first investigated the mental model development of players with different
behavioral profiles. Table 5 shows the frequency of players’ comments that
reflect each aspect of SA and separated for Hoarders, Reactors, and Followers.
Chi-square test of independence showed a significant association between
behavioral profiles and players’ situation model
(${\chi}^{2}_{4,1069}=18.132$, $p=.001$), and indicated a weak dependence
(Cramer’s $V=.092$). Specifically, Hoarders’ comments demonstrate
significantly lower comprehension ($p=.005$). According to our prior study
(Mohaddesi et al., 2022), Hoarders over-order more frequently compared to
other players. Such behavior is associated with a poor understanding of the
system’s dynamics, especially when facing a complex decision-making
environment (Sterman and Dogan, 2015):

> “Terrible, the ordering system has made it too confusing to track where the
> saline is in the system, so I don’t know how I can be expected to turn a
> profit like this”–(PL2-30, male, 27)
> “Bad. The slow supply of saline has messed things up. I’ll have to keep
> ordering more.”–(PL2-56, female, 32)
>

Even if they comprehend their situation as positive, they seem to give more
weight to future uncertainties and, as a result, behave proactively.
Followers, on the other hand, show significantly higher comprehension
($p<.001$) and lower perception ($p<.001$) in their comments (See Table 5).
The comprehension aspect of the Followers’ situation model is mainly reflected
in the code “general-positive” as exemplified in comments such as “good”,
“excellent”, “great”, “very good”. This could explain why Followers trust the
order suggestions and do not deviate: in their mind everything is going well
and there is no reason to change. In addition, because they comprehend the
state as positive more frequently, they may have less incentive to pay
attention to the elements of the system, which can explain their lower
perception.

Next, we examined the progression of mental model development as evidenced by
SA levels (Figure 7). The disruption seems to have triggered an increase in
perception, especially for Hoarders and Reactors, which is later reduced
during the shortage period (see Figure 7-A). The majority of the comments
reflecting perceptions during the disruption period center around “backlog-
increase” or “profit-decrease” codes (see Figure 9-A1):

> “The reduced supply from our manufacturers is affecting our supply chain. We
> could not meet the demands in the past month.”–(PL2-26, male, 32)
> “We need to get more order to deliver to out customer as we have a backlog
> we need to supply.”–(PL2-90, male, not stated)
>

Figure 7. Progression of different aspects of situation model over time
depending on behavioral profiles in Study 2. The vertical dashed line in each
graph shows the time players were notified about the disruption. The
highlighted area illustrates the shortage period due to manufacturing
disruption. For each behavioral profile group, count ratio was calculated by
dividing the number of comments that reflect each SA aspect by total number of
players in that group.

[lineplots showing the average count of SA concepts in Study 2]three lineplots
showing the average count of SA concepts in Study 2 for different behavioral
profiles

Hoarders also show a peak in their projection level right after being notified
about the disruption on Week 28 (see Figure 7-C), which is mainly related to
“anticipating problem/uncertainty” (see Figure 9-C1):

> “So far everything has been smooth, but I anticipate a shortage of inventory
> due to Covid at our supplier, so I have upped the most recent order to be
> prepared.”–(PL2-105, female, 54)
> “Good, ordered more inventory this past week incase [sic] supply chain runs
> into issues over the next few weeks due to covid.”–(PL2-39, female, 22)

As it appears again, Hoarders are more proactive when facing uncertainty.
Reactors fall behind in their projections, although showing more projections
toward the end (see Figure 7-C). Their projections predominantly concern
improvement as captured by codes “general-improve” and “profit-improve” (see
Figure 9-C2):

> “Badly but hopefully the situation is stabilising”–(PL2-32, female, 62)
> “The problem will be solved very soon”–(PL2-68, male, 48)

Such comments, however, illustrate a lack of clear strategy in Reactors’
projections. While they show similar trends in perception levels as Hoarders
(Figure 7-A), Reactors’ mental model development over time does not seem to
enable them to be strategic about the future. In sum, these results support
H3: we observe that behavioral profiles result in a different mental model
development.

####  6.2.2. Mental Model Development vs. Information Sharing per Behavioral
Profile (H4)

To study how information sharing affects the mental model development of
players with different behavioral profiles, we followed the same steps as in
previous analyses. First, we counted the presence of each SA aspect in
Hoarders’ comments, where we found a significant association between
information sharing and SA levels (${\chi}^{2}_{2,531}=21.216$, $p<.001$),
indicated by a small effect (Cramer’s $V=.200$). According to chi-square post-
hoc test (Table 5), being a Hoarder and having access to information is
associated with significantly higher perception ($p<.001$), significantly less
comprehension ($p<.001$), and significantly higher projection ($p=.005$). In
addition, comments from Hoarders in the Info group show fluctuations of SA
aspects over time (see Figure 8-A1-3). We can see an increase in the
perception aspect of these Hoarders after the disruption notification (i.e.,
Week 28 - see Figure 8-A1). We noticed that comments from Hoarders in the Info
group do not directly reflect the perception of the supplier inventory.
However, their comments do show signs of becoming more proactive and perhaps
using the information on supplier inventory to hoard before shortage affects
their supply:

> “Good, ordered more inventory this past week incase [sic] supply chain runs
> into issues over the next few weeks due to covid.”–(PL2-39, female, 22)
> “we have as much available as possible heading into uncertain
> times.”–(PL2-50, female, 35)

This pattern is also represented in the projection aspect of Hoarders in the
Info group (Figure 8-A3). It seems access to information affects their mental
model development by adding to their uncertainties:

> “The shutdown of the factory was a suprise [sic]. I’ll have to make sure we
> keep a good supply.”–(PL2-56, female, 32)
> “Okay, but concerned about future supply issues.”–(PL2-60, female, 35)

As mentioned before, poor understanding of the system’s dynamics makes
hoarding behavior more likely. Hoarders in Study 2 also represent this poor
understanding through generic descriptions/adjectives such as “general-
positive”, “general-negative”, “general-neutral” (see Figure 9-B2). Providing
information to Hoarders does not necessarily increase their understanding of
the system. It does, however, seem to reduce their lack of understanding by
directing their attention to other aspects (through increased perception) and
leveraging information to make larger safety stocks in preparation for future
uncertainties (i.e., projection).

We also found a significant association between information sharing and mental
model development for Reactors (${\chi}^{2}_{2,435}=14.122$, $p<.001$), which
was indicated by a small effect (Cramer’s $V=.180$). In addition, the chi-
square post-hoc test revealed that being a Reactor and having access to
information is associated with significantly lower perception ($p=.004$) and
higher comprehension ($p<.001$)(see Table 5), which is the exact opposite of
Hoarders. While the results for projection is not significant, average counts
of SA aspects over time (see Figure 8-B1-3), display an increasing trend in
the number of comments reflecting a projection for Reactors in both groups
(Figure 8-B3). Reactors in the Info group show an increase in their perception
followed by a decline after the shortage (Figure 8-B1). The content of their
perceptions mainly reflects “profit-decrease” and “backlog-increase” (see
Figure 10-A2):

> “We incured [sic] a lot of losses this month. we are not able to meet up
> with the demands from the health centers.”–(PL2-26, male, 32)
> “not great our backlog is large”–(PL2-4, female, 36)

While Reactors in the Info group show significantly higher comprehension, the
content of their comprehension is mainly summarized in “general-positive”,
“general-negative”, and “general-neutral” codes similar to Hoarders. These
codes are represented in comments such as “good”, “perfect”, “fine”, “bad”,
“Not good”, and “terrible”. Such comments can be again a sign of poor
understanding of the system’s dynamics. In contrast with Hoarders, it seems
information sharing does not direct Reactors’ attention to other elements as
much (through perception), which affects how they comprehend their state.
Reactors in the Info-group also show a declining trend in the counts of their
comprehension-related codes over time (Figure 8-B2). Reactors start to hoard
when facing the disruption, but it seems that those who have access to their
supplier inventory rather wait and see how things unfold instead of being
proactive. Both Hoarders and Reactors engage in hoarding behavior as
represented in their actions. However, they show different hoarding patterns
that seem to originate from different mental model development, which is
affected by information sharing.

Finally, the chi-square test of independence did not show a significant
association between information sharing and mental model development of
Followers (${\chi}^{2}_{2,103}=1.085$, $p=.58$, Cramer’s $V=.103$). We report
the chi-square test result to be consistent with previous analyses. However,
we acknowledge that the contingency table for Followers (Table 5) violates the
assumption of the chi-square test by having expected frequencies less than 5.
We also performed Fisher’s exact test (Field, 2013), which similarly revealed
no significant association ($p=.65$). It seems information sharing does not
affect Followers mental model development. They continue to believe that
ordering the same as the system’s suggestion is the best strategy. The average
counts of SA aspects over time also did not reveal a specific trend (Figure
8-C1-3). These results supports our hypothesis (H4) by showing how information
sharing is associated with mental model development depending on the
behavioral profile.

Figure 8. Progression of different aspects of situation model over time
depending on behavioral profiles and level of information sharing in Study 2.
The vertical dashed line in each graph shows the time players were notified
about the disruption. The highlighted area illustrates the shortage period due
to manufacturing disruption. For each experimental group, count ratio was
calculated by dividing the number of comments that reflect each SA aspect by
total number of players in that group.

[nine lineplots showing the average count of SA concepts in Study 2]nine
lineplots showing the average count of SA concepts in Study 2 for different
behavioral profiles and by ploting separate lines for Info and No-Info
conditions.

###  6.3. Summary

Study 2 demonstrated the reliability of our qualitative coding approach for
measuring mental model concepts as evidenced by a sufficiently high agreement
rate ($\kappa$=$0.84$). We also connected mental model development to player
actions by characterizing Hoarders, Reactors, and Followers from the lens of
their observed behavior. Our results demonstrated that differences in decision
patterns result from differences in mental model development. In particular,
Hoarders’ proactive behavior is the outcome of more uncertainty about the
future as they fail to make sense of the situation. In contrast, the
Followers’ mental model development directs them to interpret the situation
positively and trust the order suggestions. Disruption also affects players’
mental model development over time, directing Hoarders’ and Reactors’
attention to elements of the environment, evidenced by increased perception
(see Figure 7-A).

Study 2 also complemented Study 1 by providing evidence on how the effect of
information sharing on observed behavior is, in fact, the outcome of the
mental model development of players over time. More specifically,

  * •

Hoarders exhibit higher perception, lower comprehension, and higher projection
when having access to their supplier inventory, leading them to be more
proactive and stockpile.

  * •

Reactors display lower levels of perception, and higher comprehension, which
directs them towards less strategic behavior and merely react to
uncertainties.

  * •

Followers’ mental model development does not seem to be affected by
information sharing.

These results are aligned with evidence we found in our previous research for
the effect of information sharing on the observed behavior of different player
types (Mohaddesi et al., 2022). Using thought bubbles, however, we opened a
window on mental model development (as a cognitive construct) over time and
demonstrated how information sharing plays a role in explaining such behaviors
from a cognitive aspect.

##  7\. Discussion and Conclusion

The results from our studies provided empirical support for the use of thought
bubbles. A striking finding was the complete opposite patterns in mental model
development of Hoarders and Reactors. Our results not only complemented the
behavioral patterns from our prior research (Mohaddesi et al., 2022), but
provided a deeper theoretical understanding of the cognitive aspects of human
decisions in their interaction with dynamic systems. Understanding cognitive
constructs of mental models is instrumental in making sense of human behavior
and has been of interest to the HCI field for decades (Hollan et al., 2000).
This is an elusive task and requires elicitation of mental models and
analyzing elicited concepts. We used thought bubbles for mental model
elicitation by evoking players’ thinking and capturing verbalization of their
thought processes. Of course, many other researchers have also studied
elicitation of mental models (Jones et al., 2014; Grenier and Dudzinska-
Przesmitzki, 2015). Our work is distinguished from these studies in the sense
that we attempted to elicit mental models in (1) a diegetic setting—in the
context and during the interaction, (2) using an open-ended prompt to collect
verbalization of thought processes, and (3) over time.

The concept of elicitation is fundamental in studying mental models. In the
HCI context, numerous studies on eliciting mental models exist (Gero et al.,
2020; Banks and Bowman, 2021). However, most of them approach the elicitation
process in a non-diegetic form, for example, using interviews before or after
the interaction. While a comparison with non-diegetic elicitation techniques
was out of the scope of our work, we showed how diegetic elicitation allows
for dynamic assessment of mental model development over time, and by having
minimal interruption to the user. The idea of diegetic elicitation pertains to
prior research studying how elicitation procedures affect mental models. Jones
et al. (Jones et al., 2014) provided evidence on how the interview process
affects mental model representations and found out that out-of-context
elicitation pertains to mental models stored in long-term memory. Doyle et al.
(Doyle et al., 2008) argued that interviewers or additional information after
the task can impact subjects’ mental models, and thus, subjects should be
isolated from any out-of-context influences. Therefore, non-diegetic
elicitation may measure mental models that are different from what players
relied on during the interaction. In addition, considering the context of our
study (i.e., dynamic decision-making), it made sense to elicit mental models
in situ and diegetically, as in this context people continuously interact with
the decision environment which affects their mental model development. We
designed thought bubbles as part of a meeting scene (Figure 2) to ensure
keeping players in the context.

We chose to elicit mental models through a verbal process, asking players to
articulate their thinking. We found inconsistencies in prior research
regarding the value of verbal elicitation. Some researchers advocated for the
use of textual data and argued that language is key to understanding mental
models (Carley and Palmquist, 1992). Others have pointed to the complex nature
of mental models as a cognitive construct that makes it difficult for
individuals to articulate them (Memon et al., 2013). While we acknowledge that
not all aspects of mental models can be verbalized, our results demonstrated
that the parts that are verbalized provide much insight, especially about the
mental model development of people with different behavioral profiles. We
accomplished this by designing thought bubbles with an open-ended prompt to
mitigate contextual and framing biases (Memon et al., 2013) and to avoid
taking players out of context. Although one might be skeptical of players’
engagement with thought bubbles in this format, despite a somewhat negative
trend, we found evidence for continuous engagement across the eight prompts
(see Appendix C). An interesting finding was differences in the average number
of words per comment depending on behavioral profile and manipulations. For
example, hoarders in the Info group and Reactors in the No-info group wrote
more words per comment (see Figure 12).

Using thought bubbles, we collected data on players’ thought processes over
time and in intervals. Mental model elicitation over time is particularly
important, considering that mental models are dynamic constructs (Crandall et
al., 2006). They dynamically form based on new information and as the result
of interaction with the environment. Therefore, elicitation over time allowed
us to consider this dynamic nature and opened up an opportunity to study
mental model development as a process. Of course, the way mental model
measurement should be conceptualized (i.e., as a process or an outcome)
depends on the research question and the underlying task. However, prior
studies also seem to advocate for the process view on mental models, pointing
to their dynamic nature (Staggers and Norcio, 1993), or because of measurement
errors arguing that measuring the change in mental models through repeated
elicitation is preferred (Doyle et al., 2008). Our results demonstrated the
benefit of elicitation over time by showing how experimental manipulations and
behavioral profiles influence the mental model development of players over
time, as represented by the aspect of their situation model (see Figures 6-8).

From a methodological perspective, our approach for testing thought bubbles
involved three main elements: (1) game environment, (2) dynamic decision-
making task, and (3) interaction with a complex dynamic system. Future
research can leverage thought bubbles by adapting any of these elements with
some considerations. First, while games are particularly useful setting for
knowledge elicitation (Van Daalen et al., 2014), we can envision implementing
thought bubbles in a non-game environment; for example, as part of a
simulation interface where subjects provide textual input (in intervals) on
their thought process. However, we argue that the characteristics of
simulation games (here gamettes)—including their realism in representing the
system, ability to foster communication, and active involvement (Lukosch et
al., 2018)—can lead players to engage with thought bubbles differently in a
game setting compared to a non-game one. Future research can study how well a
non-game interface can capture thought processes in relation to the task with
thought bubbles.

Second, we studied the use of thought bubbles in a dynamic decision-making
task. When studying a different type of task, one must ask if the focus is on
measuring mental models as a process or an outcome (Staggers and Norcio,
1993). If the focus is on the mental model representation and their content or
structure in relation to the task—while we advocate leveraging thought bubbles
for diegetic elicitation—querying subjects in intervals might be less
relevant. Although considering the studies that cast doubt on the
externalization of mental models as a static construct (Doyle et al., 2008),
and prefer the process view on mental models (Staggers and Norcio, 1993),
future research must scrutinize to what extent eliciting mental model
development as a process can be justified for other types of tasks. Another
key consideration is whether the goal is to understand mental models or
improve the mental models of the task. While improving mental models was not
the purpose of our study, we showed how experimental manipulations such as
information sharing affect mental model development. Therefore, future
research should look into how manipulations can improve mental models and
enhance performance, perhaps by incorporating smart nudging interventions
(Mele et al., 2021). Another way is to leverage AI for different framing of
the prompt, especially as prior research argued how wording and framing of
questions could affect mental models (Doyle et al., 2008).

The type of task also affects the choice of framework for analyzing
elicitation results. We used the Situation Awareness model introduced by
Endsley (Endsley, 1995) as it maps well to mental models theory and dynamic
decision-making context, and hence, suited our study in which participants
interacted with a dynamic decision task. However, when the task is not dynamic
decision-making (e.g., gestural interaction with a display (Soni et al.,
2020)), SA might be less relevant. While we advocate for grounding the
qualitative analysis process in theories, the choice of underlying theoretical
framework in non-dynamic context requires more scrutiny. We demonstrated the
reliability of our approach by using our generated codebook from Study 1 to a
new dataset (Study 2) and providing insight on mental model development of
players with distinct behavioral profiles.

Finally, we studied players’ mental model development in their interaction
with an interactive environment simulating a complex dynamic system (i.e.,
supply chain simulation that evolved both spontaneously and as the result of
players’ actions). However, HCI encompasses many forms of interaction
(Helander, 2014). Future research can study other types of interaction in
which thought bubbles can be leveraged, perhaps with some adaptions. For
example, researchers can investigate the use of thought bubbles for diegetic
elicitation through non-verbal queries such as card sorting (Martín and
Macías, 2023), or diagrammatic representations (Saint-Louis and McEwen, 2022).
In addition, recent advances in AI have inspired many researchers to study
human-AI interaction (Amershi et al., 2019; Sundar, 2020) and investigate
mental models of AI (Gero et al., 2020; Villareale et al., 2022). Thought
bubbles can be utilized to advance our understanding of users’ mental model
development of AI over time. Thought bubbles can also be viewed as a form of
reflection, and reflection can be used to improve learning (Helyer, 2015).
Therefore, future research can look into opportunities in which thought
bubbles can be utilized for reflection in game-based learning technologies
(Harteveld et al., 2020). For all this to work, we need to think about how to
scale up the use of thought bubbles with respect to qualitative analysis,
which is a time-consuming process. Future studies can look into leveraging
natural language processing (NLP) for qualitative analysis (Leeson et al.,
2019; Guetterman et al., 2018; Crowston et al., 2012). Last but not least,
thought bubbles can potentially elicit more than mental models of the tasks at
hand by uncovering cognitive aspects such as attitudes and motivation, which
is another interesting future avenue, as reflected in the following quote:

> “This is bad. I guess I was scared about having enough supply. I over
> ordered and now we are paying for it in surplus inventory. I should have
> gone to veterinarian school like my family wanted! But no! I was all like
> ”I’m going to be a great supply chain manager one day””

###### Acknowledgements.

This research was supported with funding from the National Science Foundation
(NSF: 1638302 & 2028449). We further thank the StudyCrafter and Drug Shortage
teams at Northeastern University.

## References

  * (1)
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Andersen and Rohrbaugh (1992) David F Andersen and John Rohrbaugh. 1992.  Some conceptual and technical problems in integrating models of judgment with simulation models.  _IEEE Transactions on systems, Man, and Cybernetics_ 22, 1 (1992), 21–34.
  * Andrews et al. (2022) Robert W. Andrews, J. Mason Lilly, Divya Srivastava, and Karen M. Feigh. 2022.  The role of shared mental models in human-AI teams: A theoretical review.  _Theoretical Issues in Ergonomics Science_ 0, 0 (2022), 1–47.  https://doi.org/10.1080/1463922X.2022.2061080
  * Appelt et al. (2011) Kirstin C. Appelt, Kerry F. Milch, Michel J. J. Handgraaf, and Elke U. Weber. 2011.  The Decision Making Individual Differences Inventory and guidelines for the study of individual differences in judgment and decision-making research.  _Judgment and Decision Making_ 6, 3 (2011), 252–262.  https://doi.org/10.1017/S1930297500001455
  * Azghandi et al. (2018) Rana Azghandi, Jacqueline Griffin, and Mohammad S Jalali. 2018\.  Minimization of drug shortages in pharmaceutical supply chains: A simulation-based analysis of drug recall patterns and inventory policies.  _Complexity_ 2018 (2018), 14 pages.
  * Banks and Bowman (2021) Jaime Banks and Nicholas David Bowman. 2021.  Some assembly required: Player mental models of videogame avatars.  _Frontiers in Psychology_ 12 (2021), 16 pages.  https://doi.org/10.3389/fpsyg.2021.701965
  * Bekebrede et al. (2015) Geertje Bekebrede, Julia Lo, and Heide Lukosch. 2015.  Understanding complex systems through mental models and shared experiences: A case study.  _Simulation & Gaming_ 46, 5 (2015), 536–562.
  * Bekebrede and Mayer (2006) Geertje Bekebrede and Igor Mayer. 2006.  Build your seaport in a game and learn about complex systems.  _Journal of design research_ 5, 2 (2006), 273–298.
  * Brewer (1987) William F. Brewer. 1987\.  _Schemas versus mental models in human memory_.  John Wiley & Sons, Oxford, England, 187–197.
  * Brewer and Treyens (1981) William F Brewer and James C Treyens. 1981.  Role of schemata in memory for places.  _Cognitive psychology_ 13, 2 (1981), 207–230.
  * Brown et al. (2009) Andrea Brown, Cheryl Karthaus, Lisa A Rehak, and Barb Adams. 2009.  _The role of mental models in dynamic decision-making_.  Technical Report. HUMANSYSTEMS INC GUELPH (ONTARIO).
  * Carley and Palmquist (1992) Kathleen Carley and Michael Palmquist. 1992.  Extracting, representing, and analyzing mental models.  _Social forces_ 70, 3 (1992), 601–636.
  * Chen et al. (2021) Hao Chen, Shuang Liu, Liping Pang, Xiaoru Wanyan, and Yufeng Fang. 2021.  Developing an Improved ACT-R Model for Pilot Situation Awareness Measurement.  _IEEE Access_ 9 (2021), 122113–122124.
  * Chihaoui et al. (2019) Faten Ben Chihaoui, Nouha Maddeh, Safa Bhar Layeb, Chokri Hamouda, and Jouhaina Chaouachi. 2019.  A decision support system for drug inventory management within an emergency department: A case study. In _2019 6th International Conference on Control, Decision and Information Technologies (CoDIT)_. IEEE, New York, NY, USA, 1889–1894.
  * Cohen (2013) Jacob Cohen. 2013\.  _Statistical power analysis for the behavioral sciences_ (2nd ed ed.).  L. Erlbaum Associates, New York.  https://doi.org/10.4324/9780203771587
  * Collins and Gentner (1987) Allan Collins and Dedre Gentner. 1987.  How people construct mental models.  _Cultural models in language and thought_ 243 (1987), 243–265.
  * Cooke (1994) Nancy J Cooke. 1994\.  Varieties of knowledge elicitation techniques.  _International journal of human-computer studies_ 41, 6 (1994), 801–849.
  * Craik (1943) Kenneth James Williams Craik. 1943\.  _The nature of explanation_.  Cambridge: Cambridge University Press, New York.
  * Crandall et al. (2006) Beth Crandall, Gary A Klein, and Robert R Hoffman. 2006\.  _Working minds: A practitioner’s guide to cognitive task analysis_.  MIT Press, Cambridge, Massachusetts.
  * Crowston et al. (2012) Kevin Crowston, Eileen E Allen, and Robert Heckman. 2012\.  Using natural language processing technology for qualitative data analysis.  _International Journal of Social Research Methodology_ 15, 6 (2012), 523–543.
  * Doroudi et al. (2018) Rozhin Doroudi, Rana Azghandi, Zlatan Feric, Omid Mohaddesi, Yifan Sun, Jacqueline Griffin, Ozlem Ergun, David Kaeli, Pedro Sequeira, Stacy Marsella, et al. 2018\.  An integrated simulation framework for examining resiliency in pharmaceutical supply chains considering human behaviors. In _Proceedings of the 2018 Winter Simulation Conference_. IEEE Press, New York, NY, USA, 88–99.
  * Doroudi et al. (2020) Rozhin Doroudi, Pedro Sequeira, Stacy Marsella, Ozlem Ergun, Rana Azghandi, David Kaeli, Yifan Sun, and Jacqueline Griffin. 2020\.  Effects of trust-based decision making in disrupted supply chains.  _PloS one_ 15, 2 (2020), e0224761.
  * Doyle and Ford (1998) James K Doyle and David N Ford. 1998.  Mental models concepts for system dynamics research.  _System dynamics review: the journal of the System Dynamics Society_ 14, 1 (1998), 3–29.
  * Doyle et al. (2008) James K Doyle, Michael J Radzicki, and W Scott Trees. 2008\.  Measuring change in mental models of complex dynamic systems.  In _Complex Decision Making_. Springer, Berlin, Heidelberg, 269–294.  https://doi.org/10.1007/978-3-540-73665-3_14
  * Edwards (1962) Ward Edwards. 1962\.  Dynamic decision theory and probabilistic information processings.  _Human factors_ 4, 2 (1962), 59–74.
  * Endsley (1988) M.R. Endsley. 1988\.  Situation awareness global assessment technique (SAGAT). In _Proceedings of the IEEE 1988 National Aerospace and Electronics Conference_. IEEE, Dayton, OH, USA, 789–795 vol.3.  https://doi.org/10.1109/NAECON.1988.195097
  * Endsley and Garland (2000) M.R. Endsley and D.J. Garland. 2000.  _Situation Awareness Analysis and Measurement_.  CRC Press, Boca Raton, FL.  https://books.google.com/books?id=WrJGDsjJakcC
  * Endsley (1995) Mica R Endsley. 1995\.  Toward a theory of situation awareness in dynamic systems.  _Human factors_ 37, 1 (1995), 32–64.
  * Endsley (2000) Mica R Endsley. 2000\.  Situation models: An avenue to the modeling of mental models. In _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_. SAGE Publications, Los Angeles, CA, 61–64.
  * Endsley (2015) Mica R Endsley. 2015\.  Situation awareness misconceptions and misunderstandings.  _Journal of Cognitive Engineering and Decision Making_ 9, 1 (2015), 4–32.
  * Endsley (2017) Mica R Endsley. 2017\.  Autonomous driving systems: A preliminary naturalistic study of the Tesla Model S.  _Journal of Cognitive Engineering and Decision Making_ 11, 3 (2017), 225–238.
  * Endsley (2018) Mica R. Endsley. 2018\.  _Expertise and Situation Awareness_ (2 ed.).  Cambridge University Press, Cambridge, 633–652.  https://doi.org/10.1017/9781316480748.037
  * Endsley (2021) Mica R Endsley. 2021\.  A systematic review and meta-analysis of direct objective measures of situation awareness: A comparison of SAGAT and SPAM.  _Human factors_ 63, 1 (2021), 124–150.
  * Field (2013) A. Field. 2013.  _Discovering Statistics Using IBM SPSS Statistics_.  SAGE Publications, London.  https://books.google.com/books?id=c0Wk9IuBmAoC
  * Fleiss (1971) Joseph L Fleiss. 1971\.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ 76, 5 (1971), 378\.
  * Ford and Sterman (1998) David N Ford and John D Sterman. 1998.  Expert knowledge elicitation to improve formal and mental models.  _System Dynamics Review: The Journal of the System Dynamics Society_ 14, 4 (1998), 309–340.
  * Fu and Gonzalez (2006) Wai-Tat Fu and Cleotilde Gonzalez. 2006.  Learning in dynamic decision making: Information utilization and future planning. In _The 28th Annual Conference of the Cognitive Science Society (CogSci 2006)_. Lawrence Erlbaum Associates, Vancouver, BC, Canada, 244–249.
  * Gary and Wood (2011) Michael Shayne Gary and Robert E Wood. 2011.  Mental models, decision rules, and performance heterogeneity.  _Strategic management journal_ 32, 6 (2011), 569–594.
  * Gentner and Stevens (2014) Dedre Gentner and Albert L Stevens. 2014.  _Mental models_.  Psychology Press, New York.
  * Gero et al. (2020) Katy Ilonka Gero, Zahra Ashktorab, Casey Dugan, Qian Pan, James Johnson, Werner Geyer, Maria Ruiz, Sarah Miller, David R. Millen, Murray Campbell, Sadhana Kumaravel, and Wei Zhang. 2020\.  Mental models of AI agents in a cooperative game setting. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3313831.3376316
  * Grenier and Dudzinska-Przesmitzki (2015) Robin S. Grenier and Dana Dudzinska-Przesmitzki. 2015\.  A Conceptual Model for Eliciting Mental Models Using a Composite Methodology.  _Human Resource Development Review_ 14, 2 (2015), 163–184.  https://doi.org/10.1177/1534484315575966 arXiv:https://doi.org/10.1177/1534484315575966
  * Guetterman et al. (2018) Timothy C Guetterman, Tammy Chang, Melissa DeJonckheere, Tanmay Basu, Elizabeth Scruggs, and VG Vinod Vydiswaran. 2018.  Augmenting qualitative text analysis with natural language processing: methodological study.  _Journal of medical Internet research_ 20, 6 (2018), e9702.
  * Harteveld et al. (2020) Casper Harteveld, Nithesh Javvaji, Tiago Machado, Yevgeniya V. Zastavker, Victoria Bennett, and Tarek Abdoun. 2020.  Gaming4All: Reflecting on Diversity, Equity, and Inclusion for Game-Based Engineering Education. In _2020 IEEE Frontiers in Education Conference (FIE)_. IEEE, Uppsala, Sweden, 1–9.  https://doi.org/10.1109/FIE44824.2020.9274176
  * Helander (2014) M.G. Helander. 2014\.  _Handbook of Human-Computer Interaction_.  Elsevier Science, New York.  https://books.google.com/books?id=6vnSAwAAQBAJ
  * Helyer (2015) Ruth Helyer. 2015\.  Learning through reflection: the critical role of reflection in work-based learning (WBL).  _Journal of Work-Applied Management_ 7, 1 (2015), 15–27.
  * Hollan et al. (2000) James Hollan, Edwin Hutchins, and David Kirsh. 2000\.  Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research.  _ACM Trans. Comput.-Hum. Interact._ 7, 2 (jun 2000), 174–196.  https://doi.org/10.1145/353485.353487
  * Holland et al. (1989) J.H. Holland, K.J. Holyoak, R.E. Nisbett, and P.R. Thagard. 1989.  _Induction: Processes of inference, learning, and discovery_.  MIT Press, Cambridge, MA, USA.  https://books.google.com/books?id=Z6EFBaLApE8C
  * Hotaling et al. (2015) Jared M Hotaling, Pegah Fakhari, and Jerome R Busemeyer. 2015\.  Dynamic decision making.  In _International encyclopedia of the social & behavioral sciences_. Vol. 8. Elsevier, New York, NY, 708–713.
  * Ifenthaler (2010) Dirk Ifenthaler. 2010\.  Relational, structural, and semantic analysis of graphical representations and concept maps.  _Educational technology research and development_ 58, 1 (2010), 81–97.
  * Iyengar et al. (2016) Swathi Iyengar, Lisa Hedman, Gilles Forte, and Suzanne Hill. 2016.  Medicine shortages: A commentary on causes and mitigation strategies.  _BMC medicine_ 14, 1 (2016), 1–3.
  * Johnson-Laird (1980) Philip N Johnson-Laird. 1980\.  Mental models in cognitive science.  _Cognitive science_ 4, 1 (1980), 71–115.
  * Johnson-Laird (1983) Philip Nicholas Johnson-Laird. 1983\.  _Mental models: Towards a cognitive science of language, inference, and consciousness_.  Number 6 in Cognitive science. Harvard University Press, Cambridge, MA, USA.
  * Johnson-Laird and Bara (1984) Phillip N Johnson-Laird and Bruno G Bara. 1984.  Syllogistic inference.  _Cognition_ 16, 1 (1984), 1–61.
  * Johnson-Laird et al. (1992) Philip N Johnson-Laird, Ruth M Byrne, and Walter Schaeken. 1992\.  Propositional reasoning by model.  _Psychological review_ 99, 3 (1992), 418.
  * Jones et al. (2014) Natalie A Jones, Helen Ross, Timothy Lynam, and Pascal Perez. 2014.  Eliciting mental models: A comparison of interview procedures in the context of natural resource management.  _Ecology and Society_ 19, 1 (2014), 7 pages.
  * Jones et al. (2011) Natalie A. Jones, Helen Ross, Timothy Lynam, Pascal Perez, and Anne Leitch. 2011.  Mental Models: An Interdisciplinary Synthesis of Theory and Methods.  _Ecology and Society_ 16, 1 (2011), 13 pages.  http://www.jstor.org/stable/26268859
  * Katok (2018) Elena Katok. 2018\.  _Designing and conducting laboratory experiments_.  John Wiley & Sons, Inc., Hoboken, NJ, USA, 1–33.
  * Klein et al. (1993) Gary A Klein, Judith Orasanu, Roberta Calderwood, Caroline E Zsambok, et al. 1993\.  _Decision making in action: Models and methods_. Vol. 3.  Ablex, Norwood, NJ.
  * LaMere et al. (2020) Kelsey LaMere, Samu Mäntyniemi, Jarno Vanhatalo, and Päivi Haapasaari. 2020.  Making the most of mental models: Advancing the methodology for mental model elicitation and documentation with expert stakeholders.  _Environmental Modelling & Software_ 124 (2020), 104589.
  * Landriscina (2013) Franco Landriscina. 2013\.  _Simulation and learning_.  Springer, New York.
  * Leeson et al. (2019) William Leeson, Adam Resnick, Daniel Alexander, and John Rovers. 2019.  Natural Language Processing (NLP) in qualitative public health research: A proof of concept study.  _International Journal of Qualitative Methods_ 18 (2019), 1609406919887021\.
  * Lukosch et al. (2018) Heide K Lukosch, Geertje Bekebrede, Shalini Kurapati, and Stephan G Lukosch. 2018.  A scientific foundation of simulation games for the analysis and design of complex systems.  _Simulation & gaming_ 49, 3 (2018), 279–314.
  * Martín and Macías (2023) Marina Martín and José A. Macías. 2023.  A supporting tool for enhancing user’s mental model elicitation and decision-making in user experience research.  _International Journal of Human–Computer Interaction_ 39, 1 (2023), 183–202.  https://doi.org/10.1080/10447318.2022.2041885 arXiv:https://doi.org/10.1080/10447318.2022.2041885
  * McDonald et al. (2019) Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019\.  Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice.  _Proceedings of the ACM on human-computer interaction_ 3, CSCW (2019), 1–23.
  * Mele et al. (2021) Cristina Mele, Tiziana Russo Spena, Valtteri Kaartemo, and Maria Luisa Marzullo. 2021\.  Smart nudging: How cognitive technologies enable choice architectures for value co-creation.  _Journal of Business Research_ 129 (2021), 949–960.
  * Memon et al. (2013) Tasneem Memon, Jie Lu, and Farookh Khadeer Hussain. 2013\.  An enhanced mental model elicitation technique to improve mental model accuracy. In _International Conference on Neural Information Processing_. Springer, Berlin, Heidelberg, 82–89.
  * Mohaddesi et al. (2022) Omid Mohaddesi, Jacqueline Griffin, Ozlem Ergun, David Kaeli, Stacy Marsella, and Casper Harteveld. 2022\.  To trust or to stockpile: Modeling human-simulation interaction in supply chain shortages. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 363, 18 pages.  https://doi.org/10.1145/3491102.3502089
  * Mohaddesi et al. (2020) Omid Mohaddesi, Yifan Sun, Rana Azghandi, Rozhin Doroudi, Sam Snodgrass, Ozlem Ergun, Jacqueline Griffin, David Kaeli, Stacy Marsella, and Casper Harteveld. 2020\.  Introducing gamettes: A playful approach for capturing decision-making for informing behavioral models. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376571
  * Pauwels et al. (2015) Kim Pauwels, Steven Simoens, Minne Casteels, and Isabelle Huys. 2015.  Insights into European drug shortages: A survey of hospital pharmacists.  _PloS one_ 10, 3 (2015), e0119322.
  * Richardson et al. (1994) George P Richardson, David F Andersen, Terrence A Maxwell, and Thomas R Stewart. 1994.  Foundations of mental model research. In _Proceedings of the 1994 International System Dynamics Conference_. EF Wolstenholme, Stirling, Scotland, 181–192.
  * Ridolfi et al. (2012) Guido Ridolfi, Erwin Mooij, and Sabrina Corpino. 2012\.  _Complex-systems design methodology for systems-engineering collaborative environment_.  IntechOpen, Rijeka, Croatia, 39–70.
  * Ritchie-Dunham (2001) James L Ritchie-Dunham. 2001\.  Informing mental models for strategic decision making with ERPs and the balanced scorecard: A simulation–based experiment. In _Proceedings of the 19th International Conference of the System Dynamics Society_. System Dynamics Society, Atlanta, Georgia, USA, 19 pages.
  * Rong et al. (2008) Ying Rong, Zuo-Jun Max Shen, and Lawrence V Snyder. 2008\.  The impact of ordering behavior on order-quantity variability: A study of forward and reverse bullwhip effects.  _Flexible services and manufacturing journal_ 20, 1-2 (2008), 95\.
  * Rouwette et al. (2004) Etiënne AJA Rouwette, Andreas Größler, and Jac AM Vennix. 2004.  Exploring influencing factors on rationality: A literature review of dynamic decision-making studies in system dynamics.  _Systems Research and Behavioral Science: The Official Journal of the International Federation for Systems Research_ 21, 4 (2004), 351–370.
  * Rumelhart (1984) David E Rumelhart. 1984\.  _Schemata and the cognitive system_. Vol. 1.  Lawrence Erlbaum Associates Publishers, New York, 161–188.
  * Rutherford and Wilson (1991) Andrew Rutherford and John R. Wilson. 1991.  Models of mental models: An ergonomist-psychologist dialogue.  In _Mental Models and Human-Computer Interaction 2_ , M.J. Tauber and D. Ackermann (Eds.). Human Factors in Information Technology, Vol. 2. North-Holland, Amsterdam, The Netherlands, 39–58.
  * Saint-Louis and McEwen (2022) Hervé Saint-Louis and Rhonda McEwen. 2022.  Diagrammatic mental representation: A methodological bridge.  _Visual Studies_ 37, 5 (2022), 664–680.  https://doi.org/10.1080/1472586X.2021.1878054 arXiv:https://doi.org/10.1080/1472586X.2021.1878054
  * Saldaña (2021) Johnny Saldaña. 2021\.  _The coding manual for qualitative researchers_.  SAGE publications Ltd, London, UK. 1–440 pages.
  * Salvendy (2012) Gavriel Salvendy. 2012\.  _Handbook of human factors and ergonomics_.  John Wiley & Sons, Hoboken, NJ, USA.
  * Sarkar and Kumar (2016) Sourish Sarkar and Sanjay Kumar. 2016.  Demonstrating the effect of supply chain disruptions through an online beer distribution game.  _Decision Sciences Journal of Innovative Education_ 14, 1 (2016), 25–35.
  * Snyder and Shen (2011) Lawrence V Snyder and Zuo-Jun Max Shen. 2011.  _Fundamentals of supply chain theory_.  Wiley Online Library, Hoboken, NJ, USA.
  * Soni et al. (2020) Nikita Soni, Schuyler Gleaves, Hannah Neff, Sarah Morrison-Smith, Shaghayegh Esmaeili, Ian Mayne, Sayli Bapat, Carrie Schuman, Kathryn A. Stofer, and Lisa Anthony. 2020\.  Adults’ and Children’s Mental Models for Gestural Interactions with Interactive Spherical Displays. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3313831.3376468
  * Staggers and Norcio (1993) Nancy Staggers and Anthony F. Norcio. 1993.  Mental models: concepts for human-computer interaction research.  _International Journal of Man-machine studies_ 38, 4 (1993), 587–605.
  * Sterman and Dogan (2015) John D Sterman and Gokhan Dogan. 2015.  “I’m not hoarding, I’m just stocking up before the hoarders get here.”: Behavioral causes of phantom ordering in supply chains.  _Journal of Operations Management_ 39 (2015), 6–22.
  * Sun et al. (2016) Yifan Sun, Chisheng Liang, Steven Sutherland, Casper Harteveld, and David Kaeli. 2016\.  Modeling player decisions in a supply chain game. In _2016 IEEE Conference on Computational Intelligence and Games (CIG)_. IEEE, Santorini, Greece, 1–8.  https://doi.org/10.1109/CIG.2016.7860444
  * Sundar (2020) S Shyam Sundar. 2020\.  Rise of machine agency: A framework for studying the psychology of human–AI interaction (HAII).  _Journal of Computer-Mediated Communication_ 25, 1 (2020), 74–88.
  * Teichmann and Motus (2021) Mare Teichmann and Leo Motus. 2021.  Situation awareness, mental models and understanding. In _2021 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA)_. IEEE, Tallinn, Estonia, 86–93.
  * Tucker et al. (2020a) Emily L Tucker, Yizhou Cao, Erin R Fox, and Burgunda V Sweet. 2020a.  The drug shortage era: A scoping review of the literature 2001–2019.  _Clinical Pharmacology & Therapeutics_ 108, 6 (2020), 1150–1155.
  * Tucker et al. (2020b) Emily L Tucker, Mark S Daskin, Burgunda V Sweet, and Wallace J Hopp. 2020b.  Incentivizing resilient supply chain design to prevent drug shortages: Policy analysis using two-and multi-stage stochastic programs.  _IISE Transactions_ 52, 4 (2020), 394–412.
  * Tversky and Kahneman (1985) Amos Tversky and Daniel Kahneman. 1985.  _The framing of decisions and the psychology of choice_.  Springer, Boston, MA, USA, 25–41.
  * Van Daalen et al. (2014) C Van Daalen, Martin Schaffernicht, and Igor Mayer. 2014\.  System dynamics and serious games. In _32nd International Conference of the System Dynamics Society, Delft, The Netherlands, 20-24 July 2014_. System Dynamics Society, Delft, The Netherlands, 26 pages.
  * Villareale et al. (2022) Jennifer Villareale, Casper Harteveld, and Jichen Zhu. 2022\.  ”I want to see how smart this AI really is”: Player mental model development of an adversarial AI player.  _Proc. ACM Hum.-Comput. Interact._ 6, CHI PLAY, Article 219 (oct 2022), 26 pages.  https://doi.org/10.1145/3549482
  * Wilson and Rutherford (1989) John R. Wilson and Andrew Rutherford. 1989.  Mental Models: Theory and Application in Human Factors.  _Human Factors_ 31, 6 (1989), 617–634.  https://doi.org/10.1177/001872088903100601
  * Yang et al. (2016) Caijun Yang, Lina Wu, Wenfang Cai, Wenwen Zhu, Qian Shen, Zongjie Li, and Yu Fang. 2016.  Current situation, determinants, and solutions to drug shortages in Shaanxi Province, China: A qualitative study.  _PLoS One_ 11, 10 (2016), e0165183.
  * Zhang et al. (2022) Tao Zhang, David Kaber, and Maryam Zahabi. 2022.  Using situation awareness measures to characterize mental models in an inductive reasoning task.  _Theoretical Issues in Ergonomics Science_ 23, 1 (2022), 80–103.  https://doi.org/10.1080/1463922X.2021.1885083 arXiv:https://doi.org/10.1080/1463922X.2021.1885083

##  Appendix A Distribution of Qualitative Codes Over Time for Behavioral
Profiles

Figure 9. Distribution of topics and descriptions from qualitative coding over
time depending on behavioral profiles in Study 2. The vertical dashed line in
each graph shows the time players were notified about the disruption. The
highlighted area illustrates the shortage period due to manufacturing
disruption.

[Distribution of topics and descriptions from qualitative coding over
time]nine area plots showing the count of qualitative topics and descriptions
for players in Study 2 for different behavioral profiles across perception,
comprehension and projection.

##  Appendix B Distribution of Qualitative Codes Over Time for Behavioral
Profiles and Depending on Information Sharing

Figure 10. Distribution of topics and descriptions from qualitative coding
over time depending on behavioral profiles and level of information sharing in
Study 2. The vertical dashed line in each graph shows the time players were
notified about the disruption. The highlighted area illustrates the shortage
period due to manufacturing disruption.

[Distribution of topics and descriptions from qualitative coding over time for
information sharing]18 area plots showing the count of qualitative topics and
descriptions for players in Study 2 for different behavioral profiles and
levels of information sharing across perception, comprehension and projection.

##  Appendix C Average Count of Word per Comments

Figure 11. Average count of words per comment for players in Study 1. The
vertical dashed line in each graph shows the time players were notified about
the disruption. The highlighted area illustrates the shortage period due to
manufacturing disruption.

[lineplots of word counts for Study 1]three separate lineplots showing the
average count of words per comment for players in Study 1

Figure 12. Average count of words per comment for players in Study 2. The
vertical dashed line in each graph shows the time players were notified about
the disruption. The highlighted area illustrates the shortage period due to
manufacturing disruption.

[lineplots of word counts for Study 2]three separate lineplots showing the
average count of words per comment for players in Study 2