*Hagemann V and Kluge A (2017) Complex Problem Solving in Teams: The Impact of Collective Orientation on Team Process Demands. Front. Psychol. 8:1730. doi: 10.3389/fpsyg.2017.01730* Complex problem solving is challenging and a high-level cognitive process for individuals. When analyzing complex problem solving in teams, an additional, new dimension has to be considered, as teamwork processes increase the requirements already put on individual team members. After introducing an idealized teamwork process model, that complex problem solving teams pass through, and integrating the relevant teamwork skills for interdependently working teams into the model and combining it with the four kinds of team processes (transition, action, interpersonal, and learning processes), the paper demonstrates the importance of fulfilling team process demands for successful complex problem solving within teams. Therefore, results from a controlled team study within complex situations are presented. The study focused on factors that influence action processes, like coordination, such as emergent states like collective orientation, cohesion, and trust and that dynamically enable effective teamwork in complex situations. Before conducting the experiments, participants were divided by median split into two-person teams with either high (*n* = 58) or low (*n* = 58) collective orientation values. The study was conducted with the microworld C3Fire, simulating dynamic decision making, and acting in complex situations within a teamwork context. The microworld includes interdependent tasks such as extinguishing forest fires or protecting houses. Two firefighting scenarios had been developed, which takes a maximum of 15 min each. All teams worked on these two scenarios. Coordination within the team and the resulting team performance were calculated based on a log-file analysis. The results show that no relationships between trust and action processes and team performance exist. Likewise, no relationships were found for cohesion. Only collective orientation of team members positively influences team performance in complex environments mediated by action processes such as coordination within the team. The results are discussed in relation to previous empirical findings and to learning processes within the team with a focus on feedback strategies.

Keywords: interdependence, team processes, complex problem solving, collective orientation, trust, cohesion, C3Fire, microworld

# INTRODUCTION

Complex problems in organizational contexts are seldom solved by individuals. Generally, interdependently working teams of experts deal with complex problems (Fiore et al., 2010), which are characterized by element interactivity/ interconnectedness, dynamic developments, non-transparency and multiple, and/or conflicting goals (Dörner et al., 1983; Brehmer, 1992; Funke, 1995). Complex problem solving "takes place for reducing the barrier between a given start state and an intended goal state with the help of cognitive activities and behavior. Start state, intended goal state, and barriers prove complexity, change dynamically over time, and can be partially intransparent" (Funke, 2012, p. 682). Teams dealing with complex problems in interdependent work contexts, for example in disaster, crisis or accident management, are called High Responsibility Teams. They are named High Responsibility Teams (HRTs; Hagemann, 2011; Hagemann et al., 2011) due to their dynamic and often unpredictable working conditions and demanding work contexts, in which technical faults and slips have severe consequences for human beings and the environment if they are not identified and resolved within the team immediately (Kluge et al., 2009). HRTs bear responsibility regarding lives of third parties and their own lives based on their actions and consequences.

The context of interdependently working HRTs, dealing with complex problems, is described as follows (Zsambok, 1997): Members of interdependently working teams have to reach illdefined or competing goals in common in poor structured, non-transparent and dynamically changing situations under the consideration of rules of engagement and based on several cycles of joint action. Some or all goals are critical in terms of time and the consequences of actions result in decisionbased outcomes with high importance for the culture (e.g., human life). In HRT contexts, added to the features of the complexity of the problem, is the complexity of relationships, which is called social complexity (Dörner, 1989/2003) or crew coordination complexity (Kluge, 2014), which results from the interconnectedness between multiple agents through coordination requirements. The dynamic control aspect of the continuous process is coupled with the need to coordinate multiple highly interactive processes imposing high coordination demands (Roth and Woods, 1988; Waller et al., 2004; Hagemann et al., 2012).

Within this article, it is important to us to describe the theoretical background of complex problem solving in teams in depth and to combine different but compatible theoretical approaches, in order to demonstrate their theoretical and practical use in the context of the analysis of complex problem solving in teams. In Industrial and Organizational Psychology, a detailed description of tasks and work contexts that are in the focus of the analysis is essential. The individual or team task is the point of intersection between organization and individual as a "psychologically most relevant part" of the working conditions (Ulich, 1995).Thus, the tasks and the teamwork context of teams that deal with complex problems is of high relevance in the present paper. We will comprehensively describe the context of complex problem solving in teams by introducing a model of an idealized teamwork process that complex problem solving teams pass through and extensively integrate the relevant teamwork skills for these interdependently working teams into the idealized teamwork process model.

Furthermore, we will highlight the episodic aspect concerning complex problem solving in teams and combine the agreed on transition, action, interpersonal and learning processes of teamwork with the idealized teamwork process model. Because we are interested in investigating teamwork competencies and action processes of complex problem solving teams, we will analyze the indirect effect of collective orientation on team performance through the teams' coordination behavior. The focusing of the study will be owed to its validity. Even though that we know that more aspects of the theoretical framework might be of interest and could be analyzed, we will focus on a detail within the laboratory experiment for getting reliable and valid results.

# Goal, Task, and Outcome Interdependence in Teamwork

Concerning interdependence, teamwork research focuses on three designated features, which are in accordance with general process models of human action (Hertel et al., 2004). One type is goal interdependence, which refers to the degree to which teams have distinct goals as well as a linkage between individual members and team goals (Campion et al., 1993; Wageman, 1995). A second type is task interdependence, which refers to the interaction between team members. The team members depend on each other for work accomplishment, and the actions of one member have strong implications for the work process of all members (Shea and Guzzo, 1987; Campion et al., 1993; Hertel et al., 2004). The third type is outcome interdependence, which is defined as the extent to which one team member's outcomes depend on the performance of other members (Wageman, 1995). Accordingly, the rewards for each member are based on the total team performance (Hertel et al., 2004). This can occur, for instance, if a team receives a reward based on specific performance criteria. Although interdependence is often the reason why teams are formed in the first place, and it is stated as a defining attribute of teams (Salas et al., 2008), different levels of task interdependence exist (Van de Ven et al., 1976; Arthur et al., 2005).

The workflow pattern of teams can be

Teams that deal with complex problems work within intensive interdependence, which requires greater coordination patterns compared to lower levels of interdependence (Van de Ven et al., 1976; Wageman, 1995) and necessitates mutual adjustments as well as frequent interaction and information integration within the team (Gibson, 1999; Stajkovic et al., 2009).

Thus, in addition to the cognitive requirements related to information processing (e.g., encoding, storage and retrieval processes (Hinsz et al., 1997), simultaneously representing and anticipating the dynamic elements and predicting future states of the problem, balancing contradictory objectives and decide on the right timing for actions to execute) of individual team members, the interconnectedness between the experts in the team imposes high team process demands on the team members. These team process demands follow from the required interdependent actions of all team members for effectively using all resources, such as equipment, money, time, and expertise, to reach high team performance (Marks et al., 2001). Examples for team process demands are the communication for building a shared situation awareness, negotiating conflicting perspectives on how to proceed or coordinating and orchestrating actions of all team members.

# A Comprehensive Model of the Idealized Teamwork Process

The cognitive requirements, that complex problem solving teams face, and the team process demands are consolidated within our model of an idealized teamwork process in **Figure 1** (Hagemann, 2011; Kluge et al., 2014). Individual and team processes converge sequential and in parallel and influencing factors as well as process demands concerning complex problem solving in teams can be extracted. The core elements of the model are situation awareness, information transfer, individual and shared mental models, coordination and leadership, and decision making.

Complex problem solving teams are responsible for finding solutions and reaching specified goals. Based on the overall goals various sub goals will be identified at the beginning of the teamwork process in the course of mission analysis, strategy formulation and planning, all aspects of the transition phase (Marks et al., 2001). The transition phase processes occur during periods of time when teams focus predominantly on evaluation and/or planning activities. The identified and communicated goals within the team represent relevant input variables for each team member in order to build up a Situation Awareness (SA). SA contains three steps and is the foundation for an ideal and goal directed collaboration within a team (Endsley, 1999; Flin et al., 2008). The individual SA is the start and end within the idealized teamwork process model. SA means the assessment of a situation which is important for complex problem solving teams, as they work based on the division of labor as well as interdependently and each team member needs to achieve a correct SA and to share it within the team. Each single team member needs to utilize all technical and interpersonal resources in order to collect and interpret up-to-date goal directed information and to share this information with other team members via "closed-loop communication."

This information transfer focuses on sending and receiving single SA between team members in order to build up a Shared Situation Awareness (SSA). Overlapping cuts of individual SA are synchronized within the team and a bigger picture of the situation is developed. Creating a SSA means sharing a common perspective of the members concerning current events within their environment, their meaning and their future development. This shared perspective enables problem-solving teams to attain high performance standards through corresponding and goal directed actions (Cannon-Bowers et al., 1993).

Expectations of each team member based on briefings, individual mental models and interpositional knowledge influence the SA, the information transfer and the consolidation process. Mental models are internal and cognitive representations of relations and processes (e.g., execution of tactics) between various aspects or elements of a situation. They help team members to describe, explain and predict circumstances (Mathieu et al., 2000). Mental models possess knowledge elements required by team members in order to assess a current situation in terms of SA. Interpositional knowledge refers to an individual understanding concerning the tasks and duties of all team members, in order to develop an understanding about the impact of own actions on the actions of other team members and vice versa. It supports the team in identifying the information needs and the amount of required help of other members and in avoiding team conflicts (Smith-Jentsch et al., 2001). This knowledge is the foundation for anticipating the team members' needs for information and it is important for matching information within the team.

Based on the information matching process within the team, a common understanding of the problem, the goals and the current situation is developed in terms of a Shared Mental Model (SMM), which is important for the subsequent decisions. SMM are commonly shared mental models within a team and refer to the organized knowledge structures of all team members, that are shared with each other and which enable the team to interact goal-oriented (Mathieu et al., 2000). SMM help complex problem solving teams during high workload to adapt fast and efficiently to changing situations (Waller et al., 2004). They also enhance the teams' performance and communication processes (Cannon-Bowers et al., 1993; Mathieu et al., 2000). Especially under time pressure and in crucial situations when overt verbal communication and explicit coordination is not applicable, SMM are fundamental in order to coordinate implicitly. This information matching process fosters the building of a shared understanding of the current situation and the required actions. In order to do so teamwork skills (see Wilson et al., 2010) such as communication, coordination, and cooperation within the team are vitally important. **Figure 1** incorporates the teamwork skills into the model of an idealized teamwork process.

Depending on the shared knowledge and SA within the team, the coordination can be based either on well-known procedures or shared expectations within the team or on explicit communication based on task specific phraseology or closedloop communication. Cooperation needs mutual performance monitoring within the team, for example, in order to apply task strategies to accurately monitor teammate performance and prevent errors (Salas et al., 2005). Cooperation also needs backup behavior of each team member, for example, and continuous actions in reference to the collective events. The anticipation of other team members' needs under high workload maintains the

teams' performance and the well-being of each team member (Badke-Schaub, 2008). A successful pass through the teamwork process model also depends e.g., on the trust and the cohesion within the team and the collective orientation of each team member.

Collective orientation (CO) is defined "as the propensity to work in a collective manner in team settings" (Driskell et al., 2010, p. 317). Highly collectively oriented people work with others on a task-activity and team-activity track (Morgan et al., 1993) in a goal-oriented manner, seek others' input, contribute to team outcomes, enjoy team membership, and value cooperativeness more than power (Driskell et al., 2010). Thus, teams with collectively oriented members perform better than teams with non-collectively oriented members (Driskell and Salas, 1992). CO, trust and cohesion as well as other coordination and cooperation skills are so called emergent sates that represent cognitive, affective, and motivational states, and not traits, of teams and team members, and which are influenced, for example, by team experience, so that emergent states can be considered as team inputs but also as team outcomes (Marks et al., 2001).

Based on the information matching process the complex problem solving team or the team leader needs to make decisions in order to execute actions. The task prioritization and distribution is an integrated part of this step (Waller et al., 2004). Depending on the progress of the dynamic, non-transparent and heavily foreseeable situation tasks have to be re-prioritized during episodes of teamwork. Episodes are "temporal cycles of goal-directed activity" in which teams perform (Marks et al., 2001, p. 359). Thus, the team acts adaptive and is able to react flexible to situation changes. The team coordinates implicitly when each team member knows what he/she has to do in his/her job, what the others expect from him/her and how he/she interacts with the others. In contrast, when abnormal events occur and they are recognized during SA processes, the team starts coordinating explicitly via communication, for example. Via closed-loop communication and based on interpositional knowledge new strategies are communicated within the team and tasks are re-prioritized.

The result of the decision making and action taking flows back into the individual SA and the as-is state will be compared with the original goals. This model of an idealized teamwork process (**Figure 1**) is a regulator circuit with feedback loops, which enables a team to adapt flexible to changing environments and goals. The foundation of this model is the classic Input-Process-Outcome (IPO) framework (Hackman, 1987) with a strong focus on the process part. IPO models view processes as mechanisms linking variables such as member, team, or organizational features with outcomes such as performance quality and quantity or members' reactions. This mediating mechanism, the team process, can be defined as "members' interdependent acts that convert inputs to outcomes through cognitive, verbal, and behavioral activities directed toward organizing taskwork to achieve collective goals" (Marks et al., 2001, p. 357). That means team members interact interdependently with other members as well as with their environment. These cognitive, verbal, and behavioral activities directed toward taskwork and goal attainment are represented as gathering situation awareness, communication, coordination, cooperation, the consolidation of information, and task prioritization within our model of an idealized teamwork process. Within the context of complex problem solving, teams have to face team process demands in addition to cognitive challenges related to individual information processing. That means teamwork processes and taskwork to solve complex problems co-occur, the processes guide the execution of taskwork.

The dynamic nature of teamwork and temporal influences on complex problem solving teams are considered within adapted versions (Marks et al., 2001; Ilgen et al., 2005) of the original IPO framework. These adaptations propose that teams experience cycles of joint action, so called episodes, in which teams perform and also receive feedback for further actions. The IPO cycles occur sequentially and simultaneously and are nested in transition and action phases within episodes in which outcomes from initial episodes serve as inputs for the next cycle (see **Figure 2**). These repetitive IPO cycles are a vital element of our idealized teamwork process model, as it incorporates feedback loops in such a way, that the outcomes, e.g., changes within the as-is state, are continuously compared with the original goals. Detected discrepancies within the step of updating SA motivate the team members to consider further actions for goal accomplishment.

When applying this episodic framework to complex problem solving teams it becomes obvious that teams handle different types of taskwork at different phases of task accomplishment (Marks et al., 2001). That means episodes consist of two phases, so-called action and transition phases, in which teams are engaged in activities related to goal attainment and in other time in reflecting on past performance and planning for further common actions. The addition of the social complexity to the complexity of the problem within collaborative complex problem solving comes to the fore here. During transition phases teams evaluate their performance, compare the as-is state against goals, reflect on their strategies and plan future activities to guide their goal accomplishment. For example, team members discuss alternative courses of action, if their activities for simulated firefighting, such as splitting team members in order to cover more space of the map, are not successful. During action phases, teams focus directly on the taskwork and are engaged in activities such as exchanging information about the development of the dynamic situation or supporting each other. For example, a team member recognizes high workload of another team member and supports him/her in collecting information or in taking over the required communication with other involved parties.

#### Transition and Action Phases

The idealized teamwork process model covers these transition and action phases as well as the processes occurring during these two phases of team functioning, which can be clustered into transition, action, and interpersonal processes. That means during complex problem solving the relevant or activated teamwork processes in the transition and action phases change as teams move back and forth between these phases. As this taxonomy of team processes from Marks et al. (2001) states that a team process is multidimensional and teams use different processes simultaneously, some processes can occur either during transition periods or during action periods or during both periods. Transition processes especially occur during transition phases and enable the team to understand their tasks, guide their attention, specify goals and develop courses of action for task accomplishment. Thus, transition processes include (see Marks et al., 2001) mission analysis, formulation and planning (Prince and Salas, 1993), e.g., fighting a forest fire, goal specification (Prussia and Kinicki, 1996), e.g., saving as much houses and vegetation as possible, and strategy formulation (Prince and Salas, 1993; Cannon-Bowers et al., 1995), e.g., spreading team members into different geographic directions. Action processes predominantly occur during action phases and support the team in conducting activities directly related to goal accomplishment. Thus, action processes are monitoring progress toward goals (Cannon-Bowers et al., 1995), e.g., collecting information how many cells in a firefighting simulation are still burning, systems monitoring (Fleishman and Zaccaro, 1992), e.g., tracking team resources such as water for firefighting, team monitoring and backup behavior (Stevens and Campion, 1994; Salas et al., 2005), e.g., helping a team member and completing a task for him/her, and coordination (Fleishman and Zaccaro, 1992; Serfaty et al., 1998), e.g., orchestrating the interdependent actions of the team members such as exchanging information during firefighting about positions of team members for meeting at the right time at the right place in order to refill the firefighters water tanks. Especially the coordination process is influenced by the amount of task interdependence as coordination becomes more and more important for effective team functioning when interdependence increases (Marks et al., 2001). Interpersonal

processes occur during transition and action phases equally and lay the foundation for the effectiveness of other processes and govern interpersonal activities (Marks et al., 2001). Thus, interpersonal processes include conflict management (Cannon-Bowers et al., 1995), like the development of team rules, motivation and confidence building (Fleishman and Zaccaro, 1992), like encourage team members to perform better, and affect management (Cannon-Bowers et al., 1995), e.g., regulating member emotions during complex problem solving.

Summing up, process demands such as transition processes that complex problem solving teams pass through, are mission analysis, planning, briefing and goal specification, visualized on the left side of the idealized teamwork process model (see **Figure 3**). The results of these IPO cycles lay the foundation for gathering a good SA and initiating activities directed toward taskwork and goal accomplishment and therefore initiating action processes. The effective execution of action processes depends on the communication, coordination, cooperation, matching of information, and task prioritization as well as emergent team cognition variables (SSA and SMM) within the team. The results, like decisions, of these IPO cycles flow back into the next episode and may initiate further transition processes. In addition, interpersonal processes play a crucial role for complex problem solving teams. That means, conflict management, motivating and confidence building, and affect management are permanently important, no matter whether a team runs through transition or action phases and these interpersonal processes frame the whole idealized teamwork process model. Therefore, interpersonal processes are also able to impede successful teamwork at any point as breakdowns in conflict or affect management can lead to coordination breakdowns (Wilson et al., 2010) or problems with monitoring or backing up teammates (Marks et al., 2001). Thus, complex problem solving teams have to face these multidimensional team process demands in addition to cognitive challenges, e.g., information storage or retrieval (Hinsz et al., 1997), related to individual information processing.

# Team Learning Opportunities for Handling Complex Problems

In order to support teams in handling complex situations or problems, learning opportunities seem to be very important for successful task accomplishment and for reducing possible negative effects of team process demands. Learning means any kind of relative outlasted changes in potential of human behavior that cannot be traced back to age-related changes (Bower and

Hilgard, 1981; Bredenkamp, 1998). Therefore, Schmutz et al. (2016) amended the taxonomy of team processes developed by Marks et al. (2001) and added learning processes as a fourth category of processes, which occur during transition and action phases and contribute to overall team effectiveness. Learning processes (see also Edmondson, 1999) include observation, e.g., observing own and other team members' actions such as the teammate's positioning of firewalls in order to protect houses in case of firefighting, feedback, like giving a teammate information about the wind direction for effective positioning of firewalls, and reflection, e.g., talking about procedures for firefighting or refilling water tanks, for example, within the team. Learning from success and failure and identifying future problems is crucial for the effectiveness of complex problem solving teams and therefore possibilities for learning based on repetitive cycles of joint action or episodes and reflection of team members' activities during action and transition phases should be used effectively (Edmondson, 1999; Marks et al., 2001). The processes of the idealized teamwork model are embedded into these learning processes (see **Figure 3**).

The fulfillment of transition, action, interpersonal and learning processes contribute significantly to successful team performance in complex problem solving. For clustering these processes, transition and action processes could be seen as operational processes and interpersonal and learning process as support processes. When dealing with complex and dynamic situations teams have to face these team process demands more strongly than in non-complex situations. For example, goal specification and prioritization or strategy formulation, both aspects of transition processes, are strongly influenced by multiple goals, interconnectedness or dynamically and constantly changing conditions. The same is true for action processes, such as monitoring progress toward goals, team monitoring and backup behavior or coordination of interdependent actions. Interpersonal processes, such as conflict and affect management or confidence building enhance the demands put on team members compared to individuals working on complex problems. Interpersonal processes are essential for effective teamwork and need to be cultivated during episodes of team working, because breakdowns in confidence building or affect management can lead to coordination breakdowns or problems with monitoring or backing up teammates (Marks et al., 2001). Especially within complex situations aspects such as interdependence, delayed feedback, multiple goals and dynamic changes put high demands on interpersonal processes within teams. Learning processes, supporting interpersonal processes and the result of effective teamwork are e.g., observation of others' as well as own actions and receiving feedback by others or the system and are strongly influenced by situational characteristics such as non-transparency or delayed feedback concerning actions. It is assumed that amongst others team learning happens through repetitive cycles of joint action within the action phases and reflection of team members within the transition phases (Edmondson, 1999; Gabelica et al., 2014; Schmutz et al., 2016). The repetitive cycles help to generate SMM (Cannon-Bowers et al., 1993; Mathieu et al., 2000), SSA (Endsley and Robertson, 2000) or transactive memory systems (Hollingshead et al., 2012) within the team.

#### Emergent States in Complex Team Work and the Role of Collective Orientation

IPO models propose that input variables and emergent states are able to influence team processes and therefore outcomes such as team performance positively. Emergent states represent team members' attitudes or motivations and are "properties of the team that are typically dynamic in nature and vary as a function of team context, inputs, processes, and outcomes" (Marks et al., 2001, p. 357). Both emergent states and interaction processes are relevant for team effectiveness (Kozlowski and Ilgen, 2006).

Emergent states refer to conditions that underlie and dynamically enable effective teamwork (DeChurch and Mesmer-Magnus, 2010) and can be differentiated from team process, which refers to interdependent actions of team members that transform inputs into outcomes based on activities directed toward task accomplishment (Marks et al., 2001). Emergent states mainly support the execution of behavioral processes (e.g., planning, coordination, backup behavior) during the action phase, meaning during episodes when members are engaged in acts that focus on task work and goal accomplishment. Emergent states like trust, cohesion and CO are "products of team experiences (including team processes) and become new inputs to subsequent processes and outcomes" (Marks et al., 2001, p. 358). Trust between team members and cohesion within the team are emergent states that develop over time and only while experiencing teamwork in a specific team. CO is an emergent state that a team member brings along with him/her into the teamwork, is assumed to be more persistent than trust and cohesion, and can, but does not have to, be positively and negatively influenced by experiencing teamwork in a specific team for a while or by means of training (Eby and Dobbins, 1997; Driskell et al., 2010). Thus, viewing emergent states on a continuum, trust and cohesion are assumed more fluctuating than CO, but CO is much more sensitive to change and direct experience than a stable trait such as a personality trait.

CO of team members is one of the teamwork-relevant competencies that facilitates team processes, such as collecting and sharing information between team members, and positively affects the success of teams, as people who are high in CO work with others in a goal-oriented manner, seek others' input and contribute to team outcomes (Driskell et al., 2010). CO is an emergent state, as it can be an input variable as well as a teamwork outcome. CO is context-dependent, becomes visible in reactions to situations and people, and can be influenced by experience (e.g., individual learning experiences with various types of teamwork) or knowledge or training (Eby and Dobbins, 1997; Bell, 2007). CO enhances team performance through activating transition and action processes such as coordination, evaluation and consideration of task inputs from other team members while performing a team task (Driskell and Salas, 1992; Salas et al., 2005). Collectively oriented people effectively use available resources in due consideration of the team's goals,

participate actively and adapt teamwork processes adequately to the situation.

Driskell et al. (2010) and Hagemann (2017) provide a sound overview of the evidence of discriminant and convergent validity of CO compared to other teamwork-relevant constructs, such as cohesion, also an emergent state, or cooperative interdependence or preference for solitude. Studies analyzing collectively and non-collectively oriented persons' decisionmaking in an interdependent task demonstrated that teams with non-collectively oriented members performed poorly in problem solving and that members with CO judged inputs from teammates as more valuable and considered these inputs more frequently (Driskell and Salas, 1992). Eby and Dobbins (1997) also showed that CO results in increased coordination among team members, which may enhance team performance through information sharing, goal setting and strategizing (Salas et al., 2005). Driskell et al. (2010) and Hagemann (2017) analyzed CO in relation to team performance and showed that the effect of CO on team performance depends on the task type (see McGrath, 1984). Significant positive relationships between team members' CO and performance were found in relation to the task types choosing/decision making and negotiating (Driskell et al., 2010) respectively choosing/decision making (Hagemann, 2017). These kinds of tasks are characterized by much more interdependence than task types such as executing or generating tasks. As research shows that the positive influence of CO on team performance unfolds especially in interdependent teamwork contexts (Driskell et al., 2010), which require more team processes such as coordination patterns (Van de Ven et al., 1976; Wageman, 1995) and necessitate mutual adjustments as well as frequent information integration within the team (Gibson, 1999; Stajkovic et al., 2009), CO might be vitally important for complex problem solving teams. Thus, CO as an emergent state of single team members might be a valuable resource for enhancing the team's performance when exposed to solving complex problems. Therefore, it will be of interest to analyze the influence of CO on team process demands such as coordination processes and performance within complex problem solving teams. We predict that the positive effect of CO on team performance is an indirect effect through coordination processes within the team, which are vitally important for teams working in intensive interdependent work contexts.

Hypothesis 1: CO leads to a better coordination behavior, which in turn leads to a higher team performance.

As has been shown in team research that emergent states like trust and cohesion (see also **Figure 1**) affect team performance, these two constructs are analyzed in conjunction with CO concerning action processes, such as coordination behavior and team performance. Trust between team members supports information sharing and the willingness to accept feedback, and therefore positively influences teamwork processes (McAllister, 1995; Salas et al., 2005). Cohesion within a team facilitates motivational factors and group processes like coordination and enhances team performance (Beal et al., 2003; Kozlowski and Ilgen, 2006).

Hypothesis 2: Trust shows a positive relationship with (a) action processes (team coordination) and with (b) team performance.

Hypothesis 3: Cohesion shows a positive relationship with (a) action processes (team coordination) and with (b) team performance.

### MATERIALS AND METHODS

In order to demonstrate the importance of team process demands for complex problem solving in teams, we used a computer-based microworld in a laboratory study. We analyzed the effectiveness of complex problem solving teams while considering the influence of input variables, like collective orientation of team members and trust and cohesion within the team, on action processes within teams, like coordination.

# The Microworld for Investigating Teams Process Demands

We used the simulation-based team task C3Fire (Granlund et al., 2001; Granlund and Johansson, 2004), which is described as an intensive interdependence team task for complex problem solving (Arthur et al., 2005). C3Fire is a command, control and communications simulation environment that allows teams' coordination and communication in complex and dynamic environments to be analyzed. C3Fire is a microworld, as important characteristics of the real world are transferred to a small and well-controlled simulation system. The task environment in C3Fire is complex, dynamic and opaque (see **Table 1**) and therefore similar to the cognitive tasks people usually encounter in real-life settings, in and outside their work place (Brehmer and Dörner, 1993; Funke, 2001). **Figure 4** demonstrates how the complexity characteristics mentioned in **Table 1** are realized in C3Fire. The screenshot represents the simulation manager's point of view, who is able to observe all units and actions and the scenario development. For more information about the units and scenarios, please (see the text below and the Supplementary Material). Complexity requires people to consider a number of facts. Because executed actions in C3Fire influence the ongoing process, the sequencing of actions is free and not stringent, such as a fixed (if X then Y) or parallel (if X then Y and Z) sequence (Ormerod et al., 1998). This can lead to stressful situations. Taking these characteristics of microworlds into consideration, team processes during complex problem solving can be analyzed within laboratories under controlled conditions. Simulated microworlds such as C3Fire allow the gap to be bridged between laboratory studies, which might show deficiencies regarding ecological validity, and field studies, which have been criticized due to their small amount of control (see Brehmer and Dörner, 1993).

In C3Fire, the teams' task is to coordinate their actions to extinguish a forest fire whilst protecting houses and saving lives. The team members' actions are interdependent. The simulation includes, e.g., forest fires, houses, tents, gas tanks, different

kinds of vegetation and computer-simulated agents such as firefighting units (Granlund, 2003). It is possible, for example, that the direction of wind will change during firefighting and the time until different kinds of vegetation are burned down varies between those. In the present study, two simulation scenarios were developed for two-person teams and consisted of two firefighting units, one mobile water tank unit (responsible for re-filling the firefighting units' water tanks that contain a predefined amount of water) and one fire-break unit (a field defended with a fire-break cannot be ignited; the fire spreads around its ends). The two developed scenarios lasted for 15 min maximum. Each team member was responsible for two units in each scenario; person one for firefighting and water tank unit and person two for firefighting and fire-break unit. The user interface was a map system (40 × 40 square grid) with all relevant geographic information and positions of all symbols representing houses, water tank units and so on. All parts of the map with houses and vegetation were visible for the subjects, but not the fire itself or the other units; instead, the subjects were close to them with their own units (restricted visibility field; 3 × 3 square grid). The simulation was run on computers networked in a client-server configuration. The subjects used a chat system for communication that was logged. For each scenario, C3Fire creates a detailed log file containing all events that occurred over the course of the simulation. Examples of the C3Fire scenarios are provided in the Figures S1–3 and a short introduction into the microworld is given in the video. Detailed information regarding the scenario characteristics are given in Table S1. From scenario one to two, the complexity and interdependence increased.

## Participants

The study was conducted from Mai 2014 until March 2015. Undergraduate and graduate students (N = 116) studying applied cognitive sciences participated in the study (68.1% female). Their mean age was 21.17 years (SD = 3.11). Participants were assigned to 58 two-person teams, with team assignments being based on the pre-measured CO values (see procedure). They received 2 hourly credits as a trial subject and giveaways such as pencils and non-alcoholic canned drinks. The study was approved by the university's ethics committee in February 2014.

#### Procedure

The study was conducted within a laboratory setting at a university department for business psychology. Prior to the experiment, the participants filled in the CO instrument online and gave written informed consent (see **Figure 5**). The median was calculated subsequently (Md = 3.12; range: 1.69–4.06; scale range: 1–5) relating to the variable CO and two individuals with either high (n = 58) or low (n = 58) CO values were randomly matched as teammates. The matching process was random in part, as those two subjects were matched to form a team, whose preferred indicated time for participation in a specific week during data collection were identical. The participants were invited to the experimental study by e-mail 1–2 weeks after filling in the CO instrument. The study began with an introduction to the experimental procedure and the teams' task. The individuals received time to familiarize themselves with the simulation, received 20 min of training and completed two practice trials. After the training, participants answered a questionnaire collecting demographic data. Following this, a

FIGURE 4 | Examples for the complexity characteristics in Table 1 represented within a simulation scenario in C3Fire.

simulation scenario started and the participants had a maximum of 15 min to coordinate their actions to extinguish a forest fire whilst protecting houses and saving lives. After that, at measuring time T1, participants answered questionnaires assessing trust and cohesion within the team. Again, the teams worked on the following scenario 2 followed by a last round of questionnaires assessing trust and cohesion at T2.

#### Measures

**Demographic data** such as age, sex, and study course were assessed after the training at the beginning of the experiment.

**Collective Orientation** was measured at an individual level with 16 items rated on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree) developed by the authors (Hagemann, 2017) based on the work of Driskell et al. (2010). The factorial structure concerning the German-language CO scale was proven prior to this study (χ <sup>2</sup> = 162.25, df = 92, p = 0.000, χ 2 /df = 1.76, CFI =0.97, TLI = 0.96, RMSEA = 0.040, CI = 0.030-0.051, SRMR = 0.043) and correlations for testing convergent and discriminant evidence of validity were satisfying. For example, CO correlated r = 0.09 (p > 0.10) with cohesion, r = 0.34 (p < 0.01) with cooperative interdependence and r = −0.28 (p < 0.01) with preference for solitude (Hagemann, 2017). An example item is "I find working on team projects to be very satisfying". Coefficient alpha for this scale was 0.81.

**Trust** in team members' integrity, trust in members' task abilities and trust in members' work-related attitudes (Geister et al., 2006) was measured with seven items rated on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). An example item is "I can trust that I will have no additional demands due to lack of motivation of my team member." Coefficient alpha for this scale was 0.83 (T1) and 0.87 (T2).

**Cohesion** was measured with a six-item scale from Riordan and Weatherly (1999)rated on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). An example item is "In this team, there is a lot of team spirit among the members." Coefficient alpha for this scale was 0.87 (T1) and 0.87 (T2).

#### Action Process: Coordination

Successful coordination requires mechanisms that serve to manage dependencies between the teams' activities and their resources. Coordination effectiveness was assessed based on the time the firefighting units spent without water in the field in relation to the total scenario time. This measure is an indicator of the effectiveness of resource-oriented coordination, as it reflects an efficient performance regarding the water refill process in C3Fire, which requires coordinated actions between the two firefighting units and one water tank unit (Lafond et al., 2011). The underlying assumption is that a more successful coordination process leads to fewer delays in conducting the refill process. Coordination was calculated by a formula and values ranged between 0 and 1, with lower values indicating better coordination in the team (see Jobidon et al., 2012).

> Coordination = time spent without water /total time spent in scenario

#### Team Performance

This measure related to the teams' goals (limiting the number of burned out cells and saving as many houses/buildings as possible) and was quantified as the number of protected houses and the number of protected fields and bushes/trees in relation to the number of houses, fields, and bushes/trees, respectively, which would burn in a worst case scenario. This formula takes into account that teams needing more time for firefighting also have more burning cells and show a less successful performance than teams that are quick in firefighting. To determine the worst case scenario, both 15-min scenarios were run with no firefighting action taken. Thus, the particularities (e.g., how many houses would burn down if no action was taken) of each scenario were considered. Furthermore, the houses, bushes/trees and fields were weighted according to their differing importance, mirroring the teams' goals. Houses should be protected and were most important. Bushes/trees (middle importance) burn faster than fields (lowest importance) and foster the expansion of the fire. Values regarding team performance ranged between 0 and 7.99, with higher values indicating a better overall performance. Team performance was calculated as follows (see **Table 2**):

TeamPerformance = ((a/ max a)∗5) + ((b/ max b)∗2) + ((c/maxc)∗1)

#### RESULTS

Means, standard deviations, internal consistencies, and correlations for all study variables are provided in **Table 3**.

Team complex problem solving in scenario 1 correlated significantly negative with time without water in scenario 1, indicating that a high team performance is attended by the coordination behavior (as a team process). The same was true for scenario 2. In addition, time without water as an indicator for team coordination correlated significantly negative with the team members' CO, indicating that team members with high CO values experience less time without water in the microworld than teams with members with low CO values.

In order to analyze the influence of CO on team process demands such as coordination processes and thereby performance within complex problem solving teams we tested whether CO would show an indirect effect on team performance through the teams' coordination processes. To analyze this assumption, indirect effects in simple mediation models were estimated for both scenarios (see Preacher and Hayes, 2004). The mean for CO was 3.44 (SD = 0.32) for teams with high CO values and it was 2.79 (SD = 0.35) for teams with low CO values. The mean concerning team performance in scenario 1 for teams with high CO values was 6.30 (SD = 1.64) and with low CO values 5.35 (SD = 2.30). The mean concerning time without water (coordination behavior) for teams with high CO values was 0.16 (SD = 0.08) and with low CO values 0.20 (SD = 0.09). In scenario 2 the mean for team performance was

TABLE 3 | Means, standard deviations, internal consistencies, and correlations for all study variables.

*Performance range from 0 to 7.99; Time without Water range from 0 to 1 (lower values indicate a more effective handling of water); CO range from 1 to 5.* \**p* < *0.05,* \*\**p* < *0.01.*

6.26 (SD = 2.51) for teams with high CO values and it was 4.36 (SD = 2.24) for teams with low CO values. The mean concerning time without water for teams with high CO values was 0.18 (SD = 0.08) and with low CO values 0.25 (SD = 0.11).

For analyzing indirect effects, CO was the independent variable, time without water the mediator and team performance the dependent variable. The findings indicated that CO has an indirect effect on team performance mediated by time without water for scenario 1 (**Table 4**) and scenario 2 (**Table 5**). In scenario 1, CO had no direct effect on team performance (b(YX)), but CO significantly predicted time without water (b(MX)). A significant total effect (b(YX)) is not an assumption in the assessment of indirect effects, and therefore the non-significance of this relationship does not violate the analysis (see Preacher and Hayes, 2004, p. 719). Furthermore, time without water significantly predicted team performance when controlling for CO (b(YM.X)), whereas the effect of CO on team performance was not significant when controlling for time without water (b(YX.M)). The indirect effect was 0.40 and significant when using normal distribution and estimated with the Sobel test (z = 1.97, p < 0.05). The bootstrap procedure was applied to estimate the effect size not based on the assumption of normal distribution. As displayed in **Table 4**, the bootstrapped estimate of the indirect effect was 0.41 and the true indirect effect was estimated to lie between 0.0084 and 0.9215 with a 95% confidence interval. As zero is not in the 95% confidence interval, it can be concluded that the indirect effect is indeed significantly different from zero at p < 0.05 (two-tailed).

Regarding scenario 2, CO had a direct effect on team performance (b(YX)) and on time without water (b(MX)). Again, time without water significantly predicted team performance when controlling for CO (b(YM.X)), whereas the effect of CO on team performance was not significant when controlling for time without water (b(YX.M)). This time, the indirect effect was 0.60 (Sobel test, z = 2.31, p < 0.05). As displayed in **Table 5**, the bootstrapped estimate of the indirect effect was 0.61 and the true indirect effect was estimated to lie between 0.1876 and 1.1014 with a 95% confidence interval and between 0.0340 and 1.2578 with a 99% confidence interval. Because zero is not in the 99% confidence interval, it can be concluded that the indirect effect is indeed significantly different from zero at p < 0.01 (two-tailed).

TABLE 4 | Indirect Effect for Coordination and Team Performance in Scenario 1.

*Y* = *Team Performance Scenario 1; X* = *Collective Orientation T0; M* = *Coordination (time without water in scenario 1); Number of Bootstrap Resamples 5000.*\**p* < *0.05,* \*\**p*< *0.01.*

The indirect effects for both scenarios are visualized in **Figure 6**. Summing up, the results support hypothesis 1 and indicate that CO has an indirect effect on team performance mediated by the teams' coordination behavior, an action process. That means, fulfilling team process demands affect the dynamic decision making quality of teams acting in complex situations and input variables such as CO influence the action processes within teams positively.

Trust between team members assessed after scenario 1 (T1) and after scenario 2 (T2) did not show any significant correlation with the coordination behavior or with team complex problem solving in scenarios 1 and 2 (**Table 3**). Thus, hypotheses 2a and 2b are not supported. Cohesion at T1 showed no significant relationship with team performance in both scenarios, one significant negative correlation (r = −0.22, p < 0.05) with the coordination behavior in scenario 1 and no correlation with the coordination behavior in scenario 2. Cohesion at T2 did not show any significant correlation with the coordination behavior or with

TABLE 5 | Indirect Effect for Coordination and Team Performance in Scenario 2.

*Y* = *Team Performance Scenario 2; X* = *Collective Orientation T0; M* = *Coordination (time without water in scenario 2); Number of Bootstrap Resamples 5000.* \**p* < *0.05,* \*\**p* < *0.01.*

team performance in both scenarios. Thus, hypotheses 3a and 3b could also not be supported. Furthermore, the results showed no significant relations between CO and trust and cohesion. The correlations between trust and cohesion ranged between r = 0.39 and r = 0.51 (p < 0.01).

#### DISCUSSION

The purpose of our paper was first to give a sound theoretical overview and to combine theoretical approaches about team competencies and team process demands in collaborative complex problem solving and second to demonstrate the importance of selected team competencies and processes on team performance in complex problem solving by means of results from a laboratory study. We introduced the model of an idealized teamwork process that complex problem solving team pass through and integrated the relevant teamwork skills for interdependently working teams into it. Moreover, we highlighted the episodic aspect concerning complex problem solving in teams and combined the well-known transition, action, interpersonal and learning processes of teamwork with the idealized teamwork process model. Finally, we investigated the influence of trust, cohesion, and CO on action processes, such as coordination behavior of complex problem solving teams and on team performance.

Regarding hypothesis 1, studies have indicated that teams whose members have high CO values are more successful in their coordination processes and task accomplishment (Eby and Dobbins, 1997; Driskell et al., 2010; Hagemann, 2017), which may enhance team performance through considering task inputs from other team members, information sharing and strategizing (Salas et al., 2005). Thus, we had a close look on CO as an emergent state in the present study, because emergent states support the execution of behavioral processes. In order to analyze this indirect effect of CO on team performance via coordination processes, we used the time, which firefighters spent without water in a scenario, as an indicator for high-quality coordination within the team. A small amount of time without water represents sharing information and resources between team members in a reciprocal manner, which are essential qualities of effective coordination (Ellington and Dierdorff, 2014). One of the two team members was in charge of the mobile water tank unit and therefore responsible for filling up the water tanks of his/her own firefighting unit and that of the other team member on time. In order to avoid running out of water for firefighting, the team members had to exchange information about, for example, their firefighting units' current and future positions in the field, their water levels, their strategies for extinguishing one or two fires, and the water tank unit's current and future position in the field. The simple mediation models showed that CO has an indirect effect on team performance mediated by time without water, supporting hypothesis 1. Thus, CO facilitates high-quality coordination within complex problem solving teams and this in turn influences decision-making and team performance positively (cf. **Figure 1**). These results support previous findings concerning the relationships between emergent states, such as CO, and the team process, such as action processes like coordination (Cannon-Bowers et al., 1995; Driskell et al., 2010) and between the team process and the team performance (Stevens and Campion, 1994; Dierdorff et al., 2011).

Hypotheses 2 and 3 analyzed the relationships between trust and cohesion and coordination and team performance. Because no correlations between trust and cohesion and the coordination behavior and team complex problem solving existed, further analyses, like mediation analyses, were unnecessary. In contrast to other studies (McAllister, 1995; Beal et al., 2003; Salas et al., 2005; Kozlowski and Ilgen, 2006), the present study was not able to detect effects of trust and cohesion on team processes, like action processes, or on team performance. This can be attributed to the restricted sample composition or the rather small sample size. Nevertheless, effect sizes were small to medium, so that they would have become significant with an increased sample sizes. The prerequisite, mentioned by the authors, that interdependence of the teamwork is important for identifying those effects, was given in the present study. Therefore, this aspect could not have been the reason for finding no effects concerning trust and cohesion. Trust and cohesion within the teams developed during working on the simulation scenarios while fighting fires, showed significant correlations with each other, and were unrelated to CO, which showed an effect on the coordination behavior and the team performance indeed. The results seem to implicate, that the influence of CO on action processes and team performance might be much more stronger than those of trust and cohesion. If these results can be replicated should be analyzed in future studies.

As the interdependent complex problem-solving task was a computer-based simulation, the results might have been affected by the participants' attitudes to using a computer. For example, computer affinity seems to be able to minimize potential fear of working with a simulation environment and might therefore, be able to contribute to successful performance in a computerbased team task. Although computers and other electronic

devices are pervasive in present-day life, computer aversion has to be considered in future studies within complex problemsolving research when applying computer-based simulation team tasks. As all of the participants were studying applied cognitive science, which is a mix of psychology and computer science, this problem might not have been influenced the present results. However, the specific composition of the sample reduces the external validity of the study and the generalizability of the results. A further limitation is the small sample size, so that moderate to small effects are difficult to detect.

Furthermore, laboratory research of teamwork might have certain limitations. Teamwork as demonstrated in this study fails to account for the fact that teams are not simple, static and isolated entities (McGrath et al., 2000). The validity of the results could be reduced insofar as the complex relationships in teams were not represented, the teamwork context was not considered, not all teammates and teams were comparable, and the characteristic as a dynamic system with a team history and future was not given in the present study. This could be a possible explanation why no effects of trust and cohesion were found in the present study. Maybe, the teams need more time working together on the simulation scenarios in order to show that trust and cohesion influence the coordination with the team and the team performance. Furthermore, Bell (2007) demonstrated in her meta-analysis that the relationship between team members' attitudes and the team's performance was proven more strongly in the field compared to the laboratory. In consideration of this fact, the findings of the present study concerning CO are remarkable and the simulation based microworld C3Fire (Granlund et al., 2001; Granlund, 2003) seems to be appropriate for analyzing complex problem solving in interdependently working teams.

An asset of the present study is, that the teams' action processes, the coordination performance, was assessed objectively based on logged data and was not a subjective measure, as is often the case in group and team research studies (cf. Van de Ven et al., 1976; Antoni and Hertel, 2009; Dierdorff et al., 2011; Ellington and Dierdorff, 2014). As coordination was the mediator in the analysis, this objective measurement supports the validity of the results.

#### Outlook

As no transition processes such as mission analysis, formulation, and planning (Prince and Salas, 1993), goal specification (Prussia and Kinicki, 1996), and strategy formulation (Prince and Salas, 1993; Cannon-Bowers et al., 1995) as well as action processes such as monitoring progress toward goals (Cannon-Bowers et al., 1995) and systems monitoring (Fleishman and Zaccaro, 1992) were analyzed within the present study, future studies should collect data concerning these processes in order to show their importance on performance within complex problem solving teams. Because these processes are difficult to observe, subjective measurements are needed, for example asking the participants after each scenario how they have prioritized various tasks, if and when they have changed their strategy concerning protecting houses or fighting fires, and on which data within the scenarios they focused for collecting information for goal and systems monitoring. Another possibility could be using eye-tracking methods in order to collect data about collecting information for monitoring progress toward goals, e.g., collecting information how many cells are still burning, and systems monitoring, e.g., tracking team resources like water for firefighting.

CO is an emergent state and emergent states can be influenced by experience or learning, for example (Kozlowski and Ilgen, 2006). Learning processes (Edmondson, 1999), that Schmutz et al. (2016) added to the taxonomy of team processes developed by Marks et al. (2001) and which occur during transition and action phases and contribute to team effectiveness include e.g., feedback. Feedback can be useful for team learning when team learning is seen as a form of information processing (Hinsz et al., 1997). Because CO supports action processes, such as coordination and it can be influenced by learning, learning opportunities, such as feedback, seem to be important for successful task accomplishment and for supporting teams in handling complex situations or problems. If the team is temporarily and interpersonally unstable, as it is the case for most of the disaster or crisis management teams dealing with complex problems, there might be less opportunities for generating shared mental models by experiencing repetitive cycles of joint action (cf. **Figure 2**) and strategies such as cross training (Salas et al., 2007) or feedback might become more and more important for successful complex problem solving in teams. Thus, for future research it would be of interest to analyze what kind of feedback is able to influence CO positively and therefore is able to enhance coordination and performance within complex problem-solving teams.

Depending on the type of feedback, different main points will be focused during the feedback (see Gabelica et al., 2012). Feedback can be differentiated into performance and process feedback. Process feedback can be further divided into task-related and interpersonal feedback. Besides these aspects, feedback can be given on a team-level or an individual-level. Combinations of the various kinds of feedback are possible and are analyzed in research concerning their influence on e.g., selfand team-regulatory processes and team performance (Prussia and Kinicki, 1996; Hinsz et al., 1997; Jung and Sosik, 2003; Gabelica et al., 2012). For future studies it would be relevant to analyze, whether it is possible to positively influence the CO of team members and therefore action processes such as coordination and team performance or not. A focus could be on the learning processes, especially on feedback, and its influence on CO in complex problem solving teams. So far, no studies exist that analyzed the relationship between feedback and a change in CO, even though researchers already discuss the possibility that team-level process feedback shifts attention processes on team actions and team learning (McLeod et al., 1992; Hinsz et al., 1997). These results would be very helpful for training programs for fire service or police or medical teams working in complex environments and solving problems collaboratively, in order to support their team working and their performance.

In summary, the idealized teamwork process model is in combination with the transition, action, interpersonal and learning processes a good framework for analyzing the impact of teamwork competencies and teamwork processes in detail on team performance in complex environments. Overall, the framework offers further possibilities for investigating the

#### REFERENCES

influence of teamwork competencies on diverse processes and teamwork outcomes in complex problem solving teams than demonstrated here. The results of our study provide evidence of how CO influences complex problem solving teams and their performance. Accordingly, future researchers and practitioners would be well advised to find interventions how to influence CO and support interdependently working teams.

#### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of Ethical guidelines of the German Association of Psychology, Ethics committee of the University of Duisburg-Essen, Department of Computer Science and Applied Cognitive Science with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Ethics committee of the University of Duisburg-Essen, Department of Computer Science and Applied Cognitive Science.

#### AUTHOR CONTRIBUTIONS

VH and AK were responsible for the conception of the work and the study design. VH analyzed and interpreted the collected data. VH and AK drafted the manuscript. They approved it for publication and act as guarantors for the overall content.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.01730/full#supplementary-material

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Hagemann and Kluge. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Beyond Psychometrics: The Difference between Difficult Problem Solving and Complex Problem Solving

#### Jens F. Beckmann<sup>1</sup> \*, Damian P. Birney<sup>2</sup> and Natassia Goode<sup>3</sup>

<sup>1</sup> School of Education, Durham University, Durham, United Kingdom, <sup>2</sup> School of Psychology, University of Sydney, Sydney, NSW, Australia, <sup>3</sup> Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Sunshine Coast, QLD, Australia

#### Edited by:

Wolfgang Schoppek, University of Bayreuth, Germany

#### Reviewed by:

Andreas Fischer, Forschungsinstitut Betriebliche Bildung (f-bb), Germany Manuel Bedia, University of Zaragoza, Spain

\*Correspondence: Jens F. Beckmann<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 26 June 2017 Accepted: 20 September 2017 Published: 10 October 2017

#### Citation:

Beckmann JF, Birney DP and Goode N (2017) Beyond Psychometrics: The Difference between Difficult Problem Solving and Complex Problem Solving. Front. Psychol. 8:1739. doi: 10.3389/fpsyg.2017.01739 In this paper we argue that a synthesis of findings across the various sub-areas of research in complex problem solving and consequently progress in theory building is hampered by an insufficient differentiation of complexity and difficulty. In the proposed framework of person, task, and situation (PTS), complexity is conceptualized as a quality that is determined by the cognitive demands that the characteristics of the task and the situation impose. Difficulty represents the quantifiable level of a person's success in dealing with such demands. We use the well-documented "semantic effect" as an exemplar for testing some of the conceptual assumptions derived from the PTS framework. We demonstrate how a differentiation between complexity and difficulty can help take beyond a potentially too narrowly defined psychometric perspective and subsequently gain a better understanding of the cognitive mechanisms behind this effect. In an empirical study a total of 240 university students were randomly allocated to one of four conditions. The four conditions resulted from contrasting the semanticity level of the variable labels used in the CPS system (high vs. low) and two instruction conditions for how to explore the CPS system's causal structure (starting with the assumption that all relationships between variables existed vs. starting with the assumption that none of the relationships existed). The variation in the instruction aimed at inducing knowledge acquisition processes of either (1) systematic elimination of presumptions, or (2) systematic compilation of a mental representation of the causal structure underpinning the system. Results indicate that (a) it is more complex to adopt a "blank slate" perspective under high semanticity as it requires processes of inhibiting prior assumptions, and (b) it seems more difficult to employ a systematic heuristic when testing against presumptions. In combination, situational characteristics, such as the semanticity of variable labels, have the potential to trigger qualitatively different tasks. Failing to differentiate between 'task' and 'situation' as independent sources of complexity and treating complexity and difficulty synonymously threaten the validity of performance scores obtained in CPS research.

Keywords: complex problem solving, semantic effect, complexity vs. difficulty, systematicity, person–task– situation

# INTRODUCTION

fpsyg-08-01739 October 7, 2017 Time: 19:36 # 2

Complex problem solving (CPS) is an umbrella term for a diverse range of approaches to research, learning and assessment. A common denominator of all these approaches is the use of a computerized simulation of some abstract or contextualized system. This is where the main commonality ends. However, when considering the various different ways problem solvers can interact with these simulations and the wide variety of different purposes of their use, it becomes apparent that the term CPS has many different meanings. One meaning refers to a research paradigm that aims to study "complex" cognition in the context of information processing, decision-making, causal reasoning, or learning (Beckmann, 1994; Beckmann and Guthke, 1995; Frensch and Funke, 1995; Guthke et al., 1995). In other domains (e.g., Greiff et al., 2015), CPS has been considered as an ability-related construct (or set of constructs). One example is the ability to deal with uncertainty (e.g., Osman, 2010) with its conceptual – yet not always empirically aligned – links to reasoning and (fluid) intelligence (Funke and Frensch, 2007; Stadler et al., 2015). CPS has also started to establish its use in relation to an assessment approach, be it in smaller-scale studies in relation to personnel decisions (Wood et al., 2009) or in relation to larger-scale educational attainment assessment exercises such as PISA (OECD, 2013; Funke et al., 2017). Within an assessment context, CPS is often discussed as a skill or competency (rather than ability). On the one hand, the shared use of the term CPS in these contexts tends to belie the conceptual, and concomitantly, methodological diversity in this field of research; on the other hand, such diversity in meaning raises the suspicion of an insufficient conceptual foundation of CPS.

As a look beyond CPS and at scientific theory building paradigms generally reminds us, a lack of conceptual grounding tends to result in definitions for the respective target constructs that are predominantly operational (rather than conceptual). This is evident in the CPS literature, as CPS has often been used as a descriptor of the kind of behavior observable when individuals are confronted with a specific kind of challenge (i.e., CPS is what problem solvers do when dealing with complex problems). As a corollary of a preponderance of operational definitions, research and subsequent publications seem to be heavily focussing on psychometric characteristics of CPS simulations as measurement tools. In its extreme, such a situation might be perceived as delegating conceptual decisions to statistical procedures.

With this paper, we aim to go beyond the psychometrically driven approach to CPS and to contribute to a more theory-based positioning of it within a nomological network of cognition. Our argumentation leads to an empirical investigation that explicitly differentiates manipulations of complexity (the conceptual) from experiences of difficulty (the psychometric) and in so doing, demonstrate the importance of separating statistical and conceptual issues in the investigation of CPS.

# COMPLEXITY vs. DIFFICULTY

One symptom of a predominantly psychometric view on CPS is the lack of a distinction between complexity and difficulty (Beckmann and Goode, 2017). Difficulty is a psychometric concept with rather limited explanatory value. In general terms, difficulty provides a descriptive account of some items being answered correctly by a smaller proportion of individuals than other items, thus creating the basis for them being labeled as more difficult. When interested in the reasons for their higher levels of difficulty, one is confronted with a tautological reference (common to classical-test-theory) to the lower proportion of correct responses these items tend to attract. Actual explanations as to why this might be the case, however, need to go beyond such circularity. An analysis of the cognitive behavior required to tackle the problem posed by an item, as well as a reflection of the circumstances under which the item is expected to be solved, feeds into the notion of complexity. In this regard, complexity reflects ex ante considerations of the cognitive demands imposed by the task and the circumstances under which the task is to be performed (i.e., predictions), which makes complexity a primarily cognitive concept. Difficulty is experiential, person-bound and by definition, statistical. It is a reflection of how well individuals (with their individual differences in ability, knowledge, skills, motivation, etc.) deal with complexity, which makes it a psychometric concept.

At a first and rather pragmatic glance, such distinction may seem pedantic. After all, so it could be claimed, the presumptions linking difficulty (statistics) with complexity (theory) are built on a wealth of replicable scientific evidence. Therefore, so one might argue, our criticism would be considered not only unfounded in practice, but even counterproductive to the pursuit of knowledge. However, it is very easy to demonstrate this is not the case, and that when considered through person–task–situation (PTS) interactions, the broader CPS paradigm proves to be particularly in need of such a distinction. In the following we present a framework that allows for a conceptual differentiation between complexity and difficulty in the context of CPS. We then empirically test core arguments derived from this framework.

We start by taking the perspective of CPS as a research paradigm that utilizes computerized scenarios as task stimuli. These computerized scenarios or microworlds can be conceptualized as systems (e.g., Funke, 1985, 1992). In their simplest form, such systems comprise two kinds of variables that are causally linked, input variables and output variables and the interconnectedness of these system variables can be algorithmically described through linear structural equations. Such systems are considered "dynamic" if output variables change both as an effect of problem solvers inputs and independently over time.

In the contexts of research, assessment and learning, problem solvers are usually asked to first explore the unknown causal structure of these systems. In general, such an exploration phase serves the purpose of knowledge acquisition. In a subsequent control phase, problem solvers are then asked to reach and maintain pre-defined goal states in the output variables. The objective here is the application or utilization of the knowledge

acquired during the exploration phase. In the terminology of generic problem solving, the typical CPS constellation is where a particular set of operations has to be identified (i.e., knowledge acquired) that will bring the system from a given initial state to a set target one (i.e., system control).

# CPS IN THE THREE DIMENSIONAL SPACE OF PERSON, TASK, AND SITUATION (PTS)

As has been discussed previously (Beckmann, 2010; Birney et al., 2016; Beckmann and Goode, 2017), psychological research takes place in the three dimensional space of Person, Task, and Situation variables. The definition of task has two sub-facets, the task qua task and the task as behavior requirement (McGrath and Altman, 1966; Hackman, 1969; Wood, 1986). The task qua task facet refers to the physical characteristics of the stimuli the problem solver is confronted with. In the context of CPS, these are the characteristics of the CPS scenario which include, but are not limited to, the number of variables or the density of their interconnectedness. The task as behavior requirement refers to what the problem solver is instructed to do. In the context of CPS, this could be, for example, to either freely interact with the given system to uncover its causal structure, or to control this system, i.e., to reach and maintain a set of target states in the output variables. Both, task qua task and task as behavior requirements contribute interactively to the complexity of the CPS task. That is, being confronted with the same system (task qua task) but with different instructions – as communicated task as behavior requirement – results in different tasks with different levels of complexity. In short, different tasks require different sets of expected<sup>1</sup> (cognitive) behavior and therefore contribute differently to complexity.

The definition of situation refers to the environment or the circumstances in which a given task is to be performed ("task environment" as described by Newell and Simon, 1972, p. 55). In the context of CPS this includes situational characteristics such as whether a causal diagram (i.e., a graphical representation of the causal structure) is available or not when being asked to control the system. Knowing or being able to anticipate the target states during the exploration phase (or not) would be another situational characteristic. As these and other variations in circumstantial characteristics are also expected to result in differing sets of cognitive behaviors (despite being confronted with the same system and the same instruction), the situation is conceptualized as another contributor to complexity.

So far we have identified that both the task (with its two facets task qua task and task as behavior requirement) and the situation contribute to complexity. The third category of variables is linked to Person and includes, inter alia, individual differences in reasoning ability, information processing capacity, motivation, working memory, experience, and knowledge. Observed performance is the resultant of the difficulty individuals' experience in dealing with the complexity imposed by the task and the situation. In short, difficulty is the observable, subjective reflection of complexity.

Experimental research in psychology, irrespective of its focus, builds on observing variation in one component of this tripartite system of variables (i.e., Person, Task, and Situation) whilst the variation in the other two is either controlled for or, more or less systematically manipulated. The dominant experimental paradigms can be defined by their focus on one of these three components. For instance, in an assessment context, test takers are confronted with a standardized set of tasks under standardized instructions (e.g., to control a particular microworld) and in standardized situations (e.g., after a knowledge acquisition phase that resulted in a causal diagram, which is made available on the computer screen). Standardization ensures that all test takers are dealing with the same level of complexity (as it is defined by the system, the instructed task and the situation), so that observed variability in performance scores between (and occasionally within) individuals can be attributed to individual differences in conceptually relevant person characteristics (e.g., reasoning ability).

In comparison, in the context of cognition research, participants are confronted with systems (task qua task) or situations that differ systematically as part of experimental manipulations. Randomization in the allocation of participants to conditions aims at controlling for potential effects of individual differences. This allows for observed variability in average performances scores across conditions to be attributed to differences in complexity caused by the variation in task and/or situation characteristics.

In the context of instructional design research, as another example, the situational features of a (learning) task are systematically varied (e.g., availability and location of information – say, 0, 1, or more mouse-clicks away) whilst the task as behavior requirements (e.g., to acquire structural knowledge) and the task qua task (i.e., the system) are kept the same across learners. Observed performance differences are then interpreted as indications of how various situational variables (e.g., interface features) make a learning task more or less complex.

In sum, complexity and difficulty are different concepts. Failing to differentiate between the two is problematic in at least two ways. First, equating (observation-based and psychometrically derived) difficulty with complexity serves to perpetuate the circular argument of that what is difficult must be complex, and what makes something complex is its difficulty. Second, equating (task and situation analysis based) complexity with difficulty creates the dilemma of not being able to "explain" why the same level of complexity (as set by the task and the situation) results in different individuals experiencing varying levels of difficulty (as observed via differences in performance scores). The first problem creates the risk of a tautological trap that is often associated with operational definitions of constructs; the second problem seems to negate the role of the individual or person and therefore promotes a rather "un-psychological" perspective per se. For CPS to be taken beyond a predominantly

<sup>1</sup>We emphasize expected here because it highlights that complexity is based on ex ante developed expectations regarding the set of cognitive behaviors necessary to deal with the challenge posed by the task in conjunction with the situation.

psychometric approach, the differentiation between complexity and difficulty is a necessary precondition. Otherwise, by remaining too narrowly focussed at a psychometric level, CPS could just as appropriately be labeled Difficult Problem Solving (i.e., DPS) – a term which can be readily recognized as data driven and theoretically vacuous.

Projecting CPS-related research and its findings onto the tripartite system of Person, Task, and Situation as briefly outlined above provides a framework for the necessary differentiation between complexity and difficulty. In the following we use the semantic effect (Beckmann, 1994; Beckmann and Guthke, 1995; Beckmann and Goode, 2014), as an exemplar for how the PTSbased differentiation between complexity and difficulty can help take CPS beyond a psychometric perspective and subsequently gain a better understanding of the cognitive mechanisms behind this effect.

# VARIABLE LABELS AS SITUATIONAL CHARACTERISTIC – A SOURCE OF COMPLEXITY OR DIFFICULTY?

Previous research has repeatedly shown that seemingly minor changes of situational characteristics such as using semantically laden labels for system variables in comparison to semantically neutral labels have profound effects on performance (Beckmann, 1994; Beckmann and Guthke, 1995; Beckmann and Goode, 2014). In these studies, problem solvers tend to acquire less knowledge and subsequently control the system rather poorly when the same system is presented as a 'Cherry Tree' with input variables labeled 'Light,' 'Water,' and 'Temperature' linked to output variables labeled 'Cherries,' 'Leaves,' and 'Beetles' in comparison to a 'Machine' with input variables labeled as control dials 'A,' 'B,' 'C' and output variables labeled display 'X,' 'Y,' 'Z.' This phenomenon was initially described as the 'Semantic Effect' (e.g., Beckmann, 1994).

Projecting the semantic effect onto the tripartite framework of Person, Task, and Situation (PTS) implies that whilst presenting problem solvers with the same system (i.e., keeping the task qua task constant) and instructing them to execute the same tasks (i.e., keeping tasks as behavior requirements constant) still creates systematic variability in performance (i.e., indicating differences in difficulty) when a situational characteristic (e.g., the semantic meaning of variable labels) is varied.

As previous research has suggested, problem solvers confronted with system labels high in semanticity tend to approach the task of exploring a complex, dynamic system with a set of presumptions regarding the interrelatedness of system variables, whilst problem solvers working with variable labels low in semanticity tend to start with a "blank slate" concerning the causal structure of the system (Beckmann and Goode, 2014). In the former situation, knowledge acquisition would require a process of systematically eliminating presumed, yet not existing relationships and therefore reducing the complexity of the internal representation of the system's causal structure. In the latter situation, knowledge acquisition from a "blank slate" perspective would require a process of systematically compiling knowledge and therefore building up the complexity of the internal representation of the system's causal structure. Predicting whether the cognitive processes involved in eliminating presumptions are more complex than those in relation to compiling knowledge would be challenging from a purely psychometric perspective.

Concomitantly, the observed performance differences in the context of the semantic effect are associated with differences in the systematicity of the exploration behavior (Beckmann and Goode, 2014). Systematicity in exploration behavior is reflected in a specific sequence of interventions. First, all inputs are left at zero. Any changes in the outputs can then be interpreted as autonomic changes (i.e., eigendynamics). Subsequent interventions should then focus on the effects of each input variable on any of the output variables in isolation, i.e., changing only one input at a time. Only such "Vary-One-or-None-At-a-Time" heuristic (VONAT, see Beckmann and Goode, 2014; p. 279; Beckmann and Goode, 2017) creates informative system state transitions that allow problem solvers to derive knowledge regarding the causal structure of the system. In contrast, changing more than one variable at a time or to miss the zero change intervention creates what de Jong and van Joolingen (1998, p. 185) describe as "inconclusive experiments," which impedes successful knowledge acquisition.

# AIMS AND HYPOTHESES

We use the phenomenon of the semantic effect as an exemplary case for testing the conceptual assumption derived from the PTS framework that situational variables – in addition to task variables – present a potential source of complexity.

First, and based on findings from previous research (e.g., Beckmann and Goode, 2014) we expect problem solvers working with variable labels high in semanticity to be less systematic in their exploration behavior (Systematicity Hypothesis). We then address the question whether the inferior CPS performance observed under semantically rich conditions (i.e., the semantic effect) can be explained by (1) supposedly higher cognitive demands associated with a process of reducing the complexity of an internal representation of the causal structure of the explored system, or by (2) problem solvers "simply" not employing the appropriate heuristic (i.e., systematically testing against a priori assumptions). In the context of the PTS framework, results in accordance to (1) would recommend situational variables as contributors to complexity; results in accordance to (2) would suggest that situational variables contribute to the difficulty of dealing with a complex dynamic system.

The validity of a conceptual distinction between complexity and difficulty, which is based on the PTS framework, can be tested by observing the effect of explicitly instructing problem solvers to systematically explore the system by either starting with the presumption that all relationships exist (thus requiring to eliminate non-existing relationships and to reduce the complexity of the mental representation of the causal structure) or by starting with the presumption that no relationships exist (thus requiring to compile the set of relationships that exist and to build up the

complexity of the mental representation of the causal structure). If eliminating presumptions to arrive at the correct model of the system is more complex (via imposing greater demands on cognitive behavior) than starting with a blank slate then performance scores should worsen (both knowledge and control performance). Should, however, performance scores improve, this would suggest that problem solvers fail to engage in cognitive behavior that they are in fact capable of (Complexity–Difficulty Hypothesis). In psychometric terminology, the latter outcome would suggest that semanticity has the potential of introducing "construct-irrelevant difficulty" (Messick, 1995), and therefore represents a threat to validity.

#### METHODS

#### Participants

The sample comprised of 240 students from two Australian universities across a wide range of subjects, including engineering, business studies, science related subjects and medicine (60% female, mean age 22.7 years).

#### Materials

To test both the Semanticity-Hypothesis and the Complexity–Difficulty Hypothesis four different versions of a CPS scenario with three input and three output variables were created (**Figure A1** shows the causal diagram and the underpinning equations that govern this system). These four versions were embedded in a 2 (semanticity: high vs. low) by 2 (instruction: compile vs. eliminate) between subject design. In the two high semanticity versions, variable labels related to a Cherry Tree were used (i.e., 'HEAT,' 'LIGHT,' and 'WATER' for the input variables and 'CHERRIES,' 'LEAVES,' and 'BEETLES' for the output variables). In the two low semanticity versions, variables low in semantic value were used (i.e., 'INPUT A,' 'INPUT B,' and 'INPUT C' and 'OUTPUT U,' 'OUTPUT V,' and 'OUTPUT W,' respectively), referring to a 'BLACK BOX.' For each of the two semanticity conditions two instruction conditions were created. In the compile conditions problem solvers were instructed to explore the causal structure of the given system by starting with '. . . the assumption that no relationship existed, and to systematically find out which of the possible links do, in fact, exist.' In the eliminate conditions problem solvers were instructed to explore the causal structure of the system by starting with '. . . the assumption that all the relationships existed, and to systematically find out which of the possible links do, in fact, not exist.'

#### Procedure

After completing a demographics questionnaire participants were randomly allocated to one of the four CPS conditions. The CPS systems were presented in a non-numerical, graphical format on the computer screen (see **Figure 1**). Prior to being instructed to start exploring the system under the assumption that either all or none of the relationships existed, participants allocated to the high semanticity condition (i.e., Cherry Tree) were asked to indicate their expectations regarding the causal

structure that might underpin the system. This information was used to test whether the actual implemented causal structure could be perceived as counterfactual to "common" expectations.

Phase 1 – Knowledge Acquisition: Participants were first instructed to acquire knowledge of the system variables' interconnectedness. To do so they were given two cycles with seven trials each where they could freely change the values of the three input variables in their respective system and observe the subsequent changes in the output variables. After each exploration trial participants were asked to record their insights regarding the causal structure of the system in form of a causal diagram presented on screen. After the first cycle of seven trials the values for the output variables were reset, the causal diagram, however, remained on the screen.

In the compile conditions, the initial causal diagram consisted of dotted arrows representing possible links (see left panel in **Figure 1** for the Cherry Tree version). Over the course of the knowledge acquisition phase these arrows had to be changed into either solid arrows (indicating assumed links) or deleted arrows (indicating assumed non-links).

In the eliminate conditions, the initial causal diagram comprised solid arrows for all possible relationships (see right panel in **Figure 1** for the Black Box version). During the process of knowledge acquisition in this condition, arrows linking variables that were in fact identified as being unrelated were expected to be deleted from the diagram leaving only those arrows in the causal diagram for which a link is assumed to exist.

Phase 2 – Control: In the second phase, participants were asked to control their respective system using their developed causal diagram, which represented their previously acquired causal knowledge. Participants had two control cycles with seven intervention trials each to reach and maintain two different target states, which were indicated as red horizontal lines in the respective panels of the output variables (**Figure 2**). After the first control cycle, the values for the output variables were reset and a different set of target values were given. Problem solvers were not informed about these target states prior to the respective control phase.

#### Operationalizations

fpsyg-08-01739 October 7, 2017 Time: 19:36 # 6

Systematicity of exploration behavior is operationalized via three ordinal categories. The intervention sequence necessary for being able to identify the underlying causal structure of the explored system comprises one exploration intervention where all inputs were left at zero followed by three exploration interventions where only one input was changed. Problem solvers who executed this sequence in this order at least once across their 14 exploration trials received a systematicity score of 2 (VONAT). Those who either failed to employ the zero intervention or where it did not precede the three single change interventions (i.e., traditional VOTAT) received a systematicity score of 1, otherwise a score of 0 was given. The rationale is that in systems with autoregressive dependencies an all preceding zero intervention is a necessary precondition to have the chance to correctly identifying direct effects using subsequent single change interventions.

Knowledge acquisition. The task of exploring a system to find out its underlying causal structure can be conceptualized as a "relationship detection task." Taking CPS beyond a mere psychometric approach (i.e., by looking at more than the percentage of correctly identified relationships) should be reflected in the performance score used. We therefore based the operationalization of knowledge acquisition performance on a signal detection model that Snodgrass and Corwin (1988) introduced in the context of recognition memory. In this model the combined probability of correctly identifying existing and non-existing relationships (i.e., hits and correct rejections, resp.) form the sensitivity index P<sup>r</sup> (Formula 1). Knowledge scores based on this operationalization have a theoretical range from −0.98 to 0.98, where a score below zero indicates inaccurate knowledge, whilst a score above zero indicates more accurate knowledge.

$$P\_{\mathbf{r}} = \text{(Hit rate)} - \text{(False Alarm rate)}\tag{1}$$

In this model a Bias Index (Br) can also be derived, which reflects a problem solver's tendency to either "see" or "not to see" relationships when in fact they are uncertain. Bias scores (Br, Formula 2) range from 0 to 1, where values below 0.5 indicate a conservative response tendency (i.e., "guessing that relationships do not exist") and values above 0.5 indicate a liberal response tendency (i.e., "guessing that relationships exist").

$$B\_{\rm t} = \frac{\text{False Alarm rate}}{1 - (\text{Hit rate}) - (\text{False Alarm rate})} \tag{2}$$

Control performance. An operationalization of control performance by means of a simple metric of the distance between actual and target state after the final control intervention with limited reflection of the process, resembles the psychometric notion of a criterion-based assessment. That is, it does not differentiate between problem solvers who have reached the target state earlier and having to spend most of the time stabilizing the system, and problem solvers who reached the target closer to the end of the control cycle. As discussed in the context of measuring knowledge acquisition, given our aim to take CPS beyond a psychometric approach, the operationalization of control performance needs to better reflect how problem solvers cope with the cognitive demands (i.e., complexity) imposed by the start-target state discrepancy and the system characteristics (e.g., the dependency structure of output variables).

Finding the correct control intervention (i.e., set of inputs) that brings the system at or closest to the target state can be conceptualized as navigating the problem space. Different systems differ in their size and navigability as a function of (a) system characteristics such as the number and kind of dependencies, and/or (b) situational characteristics, such as the start-target discrepancy problem solvers must bridge. In order to allow for comparisons of performance scores across different studies, using different systems and/or different starttarget discrepancies, performance scores need to be standardized against the size of the problem space of the respective system and start-target discrepancy. To achieve this standardization, we propose to operationalize control performance via the Euclidean distance between the intervention vector (i.e., values entered for the input variables) used by the participant and the vector of optimal interventions (i.e., inputs that would have brought the outputs at or closest to the target states<sup>2</sup> ) for each trial of a control cycle (i.e., at each decision-input point). A standardization against the size of the problem space can be achieved by dividing the trial specific deviation scores by the trial specific difference between the vectors of pessimal and optimal intervention inputs [see formula (3)]. Consequently, control performance scores represent the averaged (across the seven control trials) deviation of the actual from the optimal intervention relative to maximal possible deviation for each and every trial. Their theoretical range is from 0 (worst possible, i.e., pessimal) to 1 (i.e., optimal).

$$\Delta vEuXr = \frac{1}{\text{m}} \sum\_{t=1}^{m} \{ 1 - \left[ \frac{\sqrt{\sum\_{i=1}^{k} (\text{optimal}\_{\text{ti}} - \text{actual}\_{\text{li}})^2}}{\sqrt{\sum\_{i=1}^{k} (\text{residual}\_{\text{ti}} - \text{optimal}\_{\text{ti}})^2}} \right] \} \quad \text{(3)}$$

m: number of trials across control cycles (14 in this study), k: number of input variables (three in this study).

<sup>2</sup>The vector of ideal inputs would bring the system exactly to (or would maintain) the target state. Restrictions of the range of possible input variables (e.g., introduced for practical reasons) might prevent reaching the target state in any one single intervention. In such cases (i.e., the ideal values fall outside this range), the values were adjusted to the nearest possible values, and these then constituted the vector of optimal inputs. In cases when the ideal values are within the range of possible inputs, the ideal values were used as the optimal input.

# RESULTS

The analyses are presented in two parts. First, we test two prerequisites, (1) the potential incompatibility of the underlying causal structure with the common expectation associated with a "real" cherry tree, and (2) the effectiveness of the instructions to start with the assumption that either all relationships existed or none of the relationships existed (manipulation check). In the second set of analyses we focus on the Systematicity-Hypothesis and the Complexity–Difficulty Hypothesis. **Table 1** provides an overview of the descriptive statistics in study-related variables across the experimental groups.

As a first step, we tested whether a potential semanticity effect might simply be explained by the causal structure that underpins the CPS system being counterfactual to what one would expect in a "real" cherry tree. To this end we analyzed the problem solvers' expectations regarding the causal structure of the Cherry Tree prior to being instructed to explore the system (i.e., using the Sensitivity Index P<sup>r</sup> to operationalize prior expectations as prior knowledge). The resulting average Pr(0) of −0.03 (SD = 0.21 based on NCT = 124) indicates no systematic misalignment of common expectations with the actual causal structure (see **Figure A1**). In the case where the implemented system structure stood in contrast to common expectations (i.e., being "counterfactual") the sensitivity index would have been substantially closer to −1.00. In cases where the implemented system structure would agree with a commonly held set of expectations – if such consensus existed in the first place – the resulting sensitivity index would be closer to +1.00. In the latter case, problem solvers would have already possessed knowledge that they were expected to acquire during the subsequent exploration phase. Both the average

hboxtextitP<sup>r</sup> value of around zero and the fact that expectations regarding the existence of relationships are equally distributed across the 12 possible variable links replicates what was found in earlier studies contrasting CPS scenarios with high and low semanticity (Beckmann, 1994; Beckmann and Goode, 2014, 2017). A counterfactual causal structure can therefore be ruled out as an alternative explanation for a potential semantic effect.

Instruction Manipulation: In a next step, we checked whether the instruction to start with the assumption that either all relationships existed (eliminate condition) or none

#### TABLE 1 | Descriptive statistics.

fpsyg-08-01739 October 7, 2017 Time: 19:36 # 8

of the relationships existed (compile condition) was reflected in problem solvers' response behavior during the knowledge acquisition phase. Problem solvers' ability to follow these instructions should be identifiable in the trajectories of the bias scores (Br) over the course of the two exploration cycles with their seven trials each. We expect problem solvers in the eliminate condition to start with a bias score greater than 0.5 and close to 1.00 as this would indicate an instruction-induced tendency "to guess that there is" a relationship when in fact (still) in a state of not knowing. Problem solvers in the compile condition, however, were expected to start with a conservative bias (i.e., a B<sup>r</sup> score below 0.5 and close to zero), which would indicate a response tendency of "guessing that there is not" a relationship when in the state of (yet) not knowing. In both conditions, we expected bias scores to become more neutral (i.e., B<sup>r</sup> ≈ 0.5) over the course of the exploration trials and when progressing in acquiring knowledge. The left panel in **Figure 3** depicts the differing bias trajectories for the "Black Box" conditions; the right panel shows them for the "Cherry Tree" conditions. It is interesting to note that the final convergence occurs at a level of around 0.75 for all conditions. This seems to indicate a general propensity to slightly err on the positive, i.e., to rather assume that there are relationships than running the risk of missing one.

The trajectories seem to suggest that the instruction has led to the expected differences in response behavior, confirming the effectiveness of the instruction manipulation, in general. Two further suggestions seem to emerge. First, the slopes for the eliminate conditions are markedly less steep than the ones for the compile conditions (F4.1,968.<sup>5</sup> = 76.455, p < 0.001, η <sup>2</sup> = 0.224)<sup>3</sup> , which seems to suggest that reducing complexity is more challenging than increasing it, regardless of semanticity. Second, the starting point of the compile condition for "Cherry Tree" is not as low as it is for "Black Box" (Br[1]: t<sup>118</sup> = 4.13, p < 0.001, dcompile−BBvsCT = 0.76), which seems to suggest that adopting a "blank slate" perspective is more challenging in a system with high semanticity.

#### Systematicity

To test the Systematicity-Hypothesis we conducted an ordinal logistic regression analysis where problem solvers' VONAT score was regressed on the semanticity condition and the instruction condition they have been allocated to. The results indicate (see **Table 2**) that problem solvers who were asked to explore a system with low semanticity (i.e., Black Box) were 2.24 time more likely to employ a systematic exploration heuristic (i.e., using VOTAT or VONAT) than problem solvers working on a system that used variables labels high in semanticity (i.e., Cherry Tree). Being instructed to either systematically eliminate erroneously presumed relationships or to identify existing relationships in the causal model of the respective system did not, however, make a substantial difference in the level of systematicity with which problem solvers explored the system.

#### Complexity – Difficulty

To address the Complexity–Difficulty Hypothesis we tested in a final step whether the effect of the instruction differs between the two levels of semanticity in terms of the knowledge acquisition performance and control performance metrics. Given both metrics produced comparable effects and interpretations, we report them together. As expected, the ANOVAs resulted in a main effect of the situational factor "semanticity," with overall lower performance scores (knowledge acquisition: F1,<sup>236</sup> = 29.863, p < 0.001, η <sup>2</sup> = 0.11; control performance: F1,<sup>236</sup> = 14.048, p < 0.001, η <sup>2</sup> = 0.06) for the high semanticity condition (i.e., Cherry Tree) relative to the low semanticity condition ("Black Box"). This replicates the semantic effect once more (Beckmann, 1994; Beckmann and Guthke, 1995; Beckmann and Goode, 2014). Across the two semanticity conditions, the task factor "instruction" seems to have no effect on performance scores overall (knowledge acquisition: F1,<sup>236</sup> = 0.119, p = 0.730, η <sup>2</sup> ≈ 0.00, control performance: F1,<sup>236</sup> = 0.027, p = 0.870, η <sup>2</sup> ≈ 0.00). However, the presence of an interaction effect (knowledge acquisition: F1,<sup>236</sup> = 13.235, p < 0.001, η <sup>2</sup> = 0.05, control performance: F1,<sup>236</sup> = 7.544, p = 0.006, η <sup>2</sup> = 0.03), indicates that when being instructed to start with the assumption that all relationships existed and consequently systematically eliminate unjustified presumptions showed a positive effect on both knowledge acquisition and control performance in conditions of high semanticity (i.e., Cherry Tree), but it resulted in systematically lower performance scores in the condition where problem solvers were working with low levels of semanticity (i.e., Black Box, see **Figure 4**).

<sup>3</sup>Greenhouse-Geisser correction of dfs for the F-test was applied due to sphericity (χ 2 <sup>90</sup> = 2201.984, p < 0.001).

TABLE 2 | Ordinal logistic regression of systematicity (VONAT) on semanticity and instruction.

# Summary

Knowledge acquisition, especially in systems with high levels of semanticity, can be conceptualized as a process of transforming a presumption structure into a knowledge structure. The instruction to start with the assumption that all possible relationships between system variables existed aimed at creating a presumption structure with high levels of complexity. If we were to use the number of relationships in a system (NoR) as a crude quantifier of complexity (see Beckmann and Goode, 2017 for a more detailed discussion) the process of knowledge acquisition under this instruction requires the reduction of complexity from a NoRpresumed = 12 to NoRactual = 6. In contrast, the instruction to start with a "blank slate" (i.e., assuming that no relationship exists) aimed at creating a situation where complexity needed to be increased from NoRpresumed = 0 to NoRactual = 6. The slope differences in bias scores between the two instruction conditions suggest that decreasing the complexity of a presumption structure is more challenging than is building up the complexity of a knowledge structure, regardless of the semanticity of the explored system.

If we were to interpret the difference in the initial Bias-scores between the two instruction conditions as an indicator of how well problem solvers were able to adopt a "full slate" or "blank slate" perspective then the significant interaction effect between semanticity and instruction would indicate that the instruction–adoption differs between the two semanticity conditions. Problem solvers tend to struggle adopting a "blank slate" perspective under the high semanticity condition. From a cognitive task analysis point of view, we could surmise that adopting a "blank slate" perspective under high semanticity conditions requires the suppression of preconceived expectations regarding the causal structure of the system as they are triggered by the semanticity of the variable labels. The process of suppression or decontextualization seems to add to the complexity of the task of knowledge acquisition in CPS-systems high in semanticity. In short, semanticity, as a situational characteristic of CPS, has the potential of being a source of complexity.

We have also argued that systematicity (i.e., the creation of informative mini-experiments that help to identify the existence or non-existence of relationships between system variables) is a necessary precondition for successful knowledge acquisition, independent of instruction conditions or semantic embedment of the system. Our findings, however, suggest that problem solvers working under high semanticity conditions are on average less likely to engage in systematic exploration behavior. At this stage, it is difficult to conceive of a "cognitive argument" that would predict that the heuristic of systematically testing against presumptions (as required in the eliminate conditions) is cognitively more demanding than testing for evidence of the existence of relationships (as would be required in the compile conditions). This, in conjunction with the fact that problem solvers in the compile condition with low semanticity were able to be more systematic, leads to the conjecture of seeing the failing to employ a suitable or necessary heuristic as an indication of the greater difficulties problem solvers seem to have. In short, semanticity as a situational characteristic of CPS might also be a potential source of (unnecessary) difficulty.

In switching the focus from the bias score (indicating the adoption of the instructed behavior) and systematicity score (indicating the level of engaging in planned and coordinated behavior) onto performance (i.e., knowledge acquisition as well as control), the data suggest that in conditions of high semanticity, it is more effective to start with the presumption that all relationships might exist ("full slate") rather than to start pretending that none exist. This requires systematic testing

against a priori assumptions regarding the system's underlying causal structure, with the emphasis on "systematic." In conditions with low semanticity, however, it seems less effective to start with presumptions of existing relationships (it is safe to assume that such presumptions would likely be a result of conscious efforts to guess). The more "natural" starting position here would be something more akin to a "blank slate" (or, knowing that one does not know), which then would require a systematic testing for evidence regarding the system's underlying causal structure.

The fact that knowledge acquisition and control performances in the high semanticity conditions still fall short of those shown in low semanticity conditions (i.e., replicating the "semantic effect") can be explained via two factors. First a complexity factor, which reflects the additional cognitive demands associated with suppressing presumptions when trying to adopt a "blank slate" starting position under high semanticity conditions, and second a difficulty factor, which reflects the tendency of problem solvers to not adopt a systematic approach to exploration behavior.

# GENERAL DISCUSSION

These reflections should not be misunderstood as an unconditional plea against the use of semantically laden variable labels in CPS. The answer to the question of what kind of systems should be used is once more the infamous: it depends. It depends on the purpose of the use of CPS scenarios. If, for instance, we aim to measure problem solvers' ability to draw inferences based on observed outcomes of systematic experimentation, we need to consider that arguably minor changes in situational characteristics, such as the semanticity of variable labels, have the potential to prevent the spontaneous employment of systematic experimentation (see also Beckmann and Goode, 2014). Under these circumstances, it would be inappropriate to interpret performance scores as indicators of problem solvers' reasoning ability or to expect them to correlate highly with reasoning measures. If, however, the aim was to predict "real life decision making" and given that "real life problems" are always semantically anchored, then using systems with high semanticity might be appropriate. The "construct purity" (or uni-dimensionality, in psychometric terms) of the measure, however, is likely to be compromised, which needs to be reflected (a) in expectations regarding inter-test correlations and (b) in the way performance scores are interpreted. Schoppek and Fischer (2015) make a convincing case for conceptualizing CPS performance scores as indicators of a competency, whereby a competency is a conglomerate of knowledge, reasoning ability, thinking skills and motivational variables. The PTS framework proposed here can help draw attention to the often-overlooked potential impact that situational characteristics might have on the composition of knowledge, reasoning ability, thinking skills and motivational variables in performance scores obtained from dealing with supposedly homomorphous CPS systems. For instance, the practice of aggregating performance scores obtained in multiple minimal complex systems (e.g., Funke, 2014; Stadler et al., 2016) with various levels of semanticity might be psychometrically desirable (e.g., maximizing reliability). At the same time, however, this very practice could (inadvertently) turn out to be a threat to construct validity if performance scores are underpinned by qualitatively different cognitive processes (e.g., compiling vs. eliminating), varying levels of functional or dysfunctional prior knowledge, and/or differences in perceived personal relevance of the semantic context these systems are embedded in. The PTS framework might also be (in-)formative for on-going discussions as to whether CPS performance scores are more than g or not (e.g., Gonzales et al., 2005; Wüstenberg et al., 2012; Hundertmark et al., 2015; Schoppek and Fischer, 2015; Stadler et al., 2015).

The use of semantically laden cover stories or variable labels to induce a stronger sense of "real life" relevance of the CPS experience for the participants in our laboratory studies or large scale assessment exercises should also not be mistaken as a shortcut to what some might call ecological validity. If we were to define ecological validity as the meaningfulness, appropriateness and usefulness of inferences drawn based on

performance scores obtained using an assessment tool, one would have to convincingly demonstrate that the problems posed in the assessment situation have triggered the same cognitive or affective processes as they are expected to be involved when dealing with complexity, uncertainty and dynamics in the "real world." Otherwise we run the risk of simply falling prey to our own make-believe. Proper validation requires an ex ante specification of the cognitive or affective processes expected to be involved when dealing with complexity, uncertainty and dynamics in the "real world." The distinction between complexity and difficulty, as being proposed here, can help moving beyond psychometrics-driven post hocinterpretations of mean scores and correlation patterns.

The differentiation between complexity and difficulty can also help improving the conceptual and psychometric quality of the assessment or research tools we use in the context of CPS. For instance, result patterns indicating that problem solvers overall experienced fewer difficulties (i.e., better performance) than the ex ante specifications of complexity would have led us to expect, could suggest that problem solvers might not have had to engage in the sequences of cognitive processes anticipated. Certain task-independent situational features could have enabled the use of prior knowledge or chunked information cues (Wood, 1986) and consequently created "construct-irrelevant easiness" (Messick, 1995). Conversely, "construct-irrelevant difficulty" (Messick, 1995) could result from a misalignment between (empirically observed) performance scores and (theoretically pre-determined) complexity specifications, where the former is systematically lower than the latter would have suggested. This could have been triggered by situational variables (inadvertently) preventing problem solvers from engaging in the anticipated sequence of cognitive behaviors. Both instances present a threat to validity that might be overlooked if complexity and difficulty are treated synonymously.

The main intent of this paper was to contribute to the discussion around taking CPS beyond a narrowly defined psychometric approach. We are of the view that a predominantly psychometric perspective tends to fall short in appropriately capturing the essence of CPS, namely complexity. We identified the lack of a differentiation between complexity and difficulty as a major barrier to achieving conceptual progress in CPS research. To redress this, we introduced the Person–Task–Situation (PTS) framework which, through the theoretical distinction it makes between its constituent factors, enables a conceptual differentiation of complexity and difficulty. The differentiation provides a theory-based platform for studying cognition (e.g., information processing, learning, decision making, reasoning) beyond an atheoretical psychometric lens.

Complexity as a concept also includes a qualitative dimension, whilst difficulty is exclusively quantitative. Complexity is a cognitive concept that reflects the interactive effects of information processing demands imposed upon the cognitive system by task and situation characteristics (i.e., the T and the S in the PTS framework). Difficulty is a psychometric concept that reflects the level of success problem solvers have in dealing with complexity. The integration of the person (i.e., the P in the PTS framework) introduces individual differences in ability, memory, knowledge and attitudinal variables as potential explanatory factors for observed performance differences. Cognition research in general, and CPS research in particular, focuses on studying the links between complexity and difficulty. By ignoring their conceptual differences and treating them synonymously, CPS research runs the risk of loosing sight of its cognition-based origins and failing to utilize its potential.

As a case in point, we used the "semantic effect" to test these conceptualizations. We were able to show that by using the same system (i.e., keeping the task qua task constant) and asking problem solvers to freely explore the system to find out its underlying causal structure (i.e., keeping the task as behavior requirement constant), but varying the system's semantic embedment via using different variable labels (i.e., varying a situational variable) systematic differences in exploration behavior occurred. Failing to differentiate task and situation as independent sources of complexity and by treating complexity and difficulty synonymously the resulting performance differences would erroneously be attributed to individual differences in person-related variables.

The conceptual distinction between complexity and difficulty paves the path for taking CPS beyond a psychometric approach. In fact, it is instrumental to bringing the "psycho-" back into psychometric. Otherwise one tends to operate with a "metric" that is agnostic to theory, and can therefore not be scrutinized for validity. The validity question is the core element of empirical research in psychology that relies on a strong conceptual underpinning. Psychometrics is a tool for linking the theoretical and the empirical and should not be used as a substitute for either.

The study presented here is not intended as a comprehensive test of the PTS framework that underpins the complexity–difficulty distinction. Instead, the paper should be considered as an invitation and orientation for future work. The theoretical analyses and empirical outcomes we report support the proposed complexity framework in demonstrating that it is both specific enough to allow for testable hypotheses, yet broad enough to allow modifications and refinements. Our work also contributes to efforts to better understand the person–task–situation tripartite. Future conceptual and empirical contributions will be necessary to further develop and refine a common framework that considers the interplay of the person, the task and the situation and has complexity at its conceptual core. This, so we have argued, is particularly pertinent to a research paradigm such as CPS that carries complexity in its label. Efforts to this end will support the better integration of research findings from existing and future studies on CPS.

## ETHICS STATEMENT

This study has been reviewed by, and received ethics clearance through, the Human Research Ethics Committee (HREC), University of New South Wales, Australia (Approval No: 09616 and 06294). After being informed (a) what participation in the study entails, (b) that participation was voluntary and (c) withdrawal from participation was possible at any time without negative consequences, and (d) that anonymity was guaranteed, participants were asked to sign an informed consent form prior to participation.

#### AUTHOR CONTRIBUTIONS

fpsyg-08-01739 October 7, 2017 Time: 19:36 # 12

JB, DB, and NG certify that they have participated sufficiently in the work to take responsibility for the content, including participation in the conception, design, analysis, drafting the work, writing, and final approval of the manuscript. Each author agrees to be accountable for all aspects of the work.

#### REFERENCES

#### FUNDING

This research was supported under Australian Research Council's Linkage Projects funding scheme (project LP0669552). The views expressed herein are those of the authors and are not necessarily those of the Australian Research Council.

# ACKNOWLEDGMENT

We would like to thank Anissa Müller, Myvan Bui, and Lisa Reinecke for their support with the data collection.

European Contributions to the Dynamic Assessment, Vol. 3, ed. J. S. Carlson (Greenwich, CT: JAI Press), 117–143.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Beckmann, Birney and Goode. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# APPENDIX

# Common Process Demands of Two Complex Dynamic Control Tasks: Transfer Is Mediated by Comprehensive Strategies

Wolfgang Schoppek <sup>1</sup> \* and Andreas Fischer <sup>2</sup>

<sup>1</sup> University of Bayreuth, Bayreuth, Germany, <sup>2</sup> Forschungsinstitut Betriebliche Bildung, Nürnberg, Germany