The present results extend previous theoretical claims in the dual-task control literature. For example, Ellenbogen and Meiran (2008) claimed that dual-task costs might be attributed to conflicts that arise when during the execution of one task rules for another task need to be held active in memory in order to enhance preparation for the latter. This can, for example, yield parallel activation of response codes, which can thus interfere and produce crosstalk phenomena (see also Hommel, 1998a; Hommel and Eglau, 2002). Our present Experiment 1, which represents a typical dual-task experiment (i.e., in form of a manual and a saccade task triggered by separate stimuli), shows that even when the instructed task rules are held constant (i.e., for both effector systems participants were instructed to initiate a response which is spatially compatible with the stimulus), the actual size of the set of task-relevant response binding patterns plays an additional important role in determining the efficiency of task coordination, even across highly different response modalities such as manual and oculomotor responses. Experiment 2 also extends the claims of rule-based conflict as a major determinant of dual-response control efficiency by showing that S–R rule-based conflict not only affects dualresponse performance in typical dual-task settings (i.e., with two separate stimuli as in Experiment 1), but also when response compounds are triggered by a single dimension of a stimulus. Note that manipulating the number of S–R rule sets by utilizing opposing S–R mappings for the two rules in Experiment 2 was also associated with dimensional overlap between S–R rule sets (i.e., responses were bivalent). Therefore, the observed effects might not only have occurred due to the different number of S–R rule sets alone, but might additionally be based on response-based conflict across trials (e.g., Meiran, 2000; Brass et al., 2003). Therefore, future research could explore effects of the number of S–R rule sets in the absence of potential interference between S–R rule sets. Taken together, the present results represent a step forward in understanding the interaction of memory and action selection in the context of multiple-action control.

Our observations, especially those in Experiment 1, further strengthen our proposed framework of multiple-action control (Huestegge and Koch, 2010, see introduction) by confirming a prediction that was not explicitly tested previously, namely that the number of task-relevant binding patterns affects the efficiency of the coordination of two responses during action selection (as indexed by the amount of dual-response costs). This result further emphasizes that any model of dual-task control that only focuses on mechanisms within a trial (e.g., Pashler, 1994; Meyer and Kieras, 1997; Navon and Miller, 2002; Tombu and Jolicoeur, 2003) is necessarily incomplete, since contextual factors (e.g., which binding patterns are required in surrounding trials) strongly determine action selection efficiency (see also Janczyk, 2016). Our observation of rule switch costs further strengthens this claim by demonstrating that maintaining two rules in working memory affected dual-response efficiency on a trial-by-trial basis. The present results further highlight that it is important to study the interaction of action control and memory in order to specify the largely elusive notion of "response selection" in dual-task control models.

An interesting additional observation in Experiment 1 was that the number of response alternatives did not strongly affect overall RTs (Hick's law) in oculomotor control. While it should be noted that the underlying group comparison might be less powerful than a within-subjects design, the absence of Hick's law for oculomotor responses is well in line with previous studies (e.g., Kveraga et al., 2002). Interestingly, although for manual responses the number of response alternatives did not differ between groups (thus no effect in single-response

conditions was expected) we found a strong modulation of manual (but not oculomotor) dual-response costs when the number of alternatives in the oculomotor response varied, thus indicating a decrease of response coordination efficiency with a larger number of binding patterns. This result pattern suggests that it is important to analyze measures of coordination costs and not only overall RT levels when studying the impact of response alternatives in the context of multiple-action control. Additionally, the potential dependency of underlying mechanisms on the specific effector systems involved calls for further research utilizing other combinations of output systems (e.g., Huestegge and Hazeltine, 2011; Stephan et al., 2013; Huestegge et al., 2014).

In sum, the present study on cross-modal multiple action control demonstrated that retrieval competition between taskrelevant response binding patterns and task rules both have a strong impact on complex action control. These results further highlight the importance of studying the interplay of memory and action control (here: retrieval competition) in order to specify the mechanisms underlying the rather vague notion of response selection in multiple action control.

#### ETHICS STATEMENT

All procedures performed in studies involving human participants were in accordance with the ethical standards of the national research committee of the DGPs (Deutsche Gesellschaft für Psychologie) with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki (1964). The protocol was approved by the Deutsche Forschungsgemeinschaft (DFG) within the grant HU

#### REFERENCES

1847/3-1. This article does not contain any studies with animals performed by any of the authors.

#### AUTHOR CONTRIBUTIONS

LH had the idea for the study. AP and LH conjointly designed experiments and wrote and revised the manuscript. AP programmed the experiments and analyzed the data.

#### FUNDING

The present research was funded by the Deutsche Forschungsgemeinschaft (HU 1847/3-1) and the University of Wuerzburg in the funding programme Open Access Publishing.

#### ACKNOWLEDGMENTS

The authors thank Andrea Zahn, Sandra Wassermann and Lena Plückebaum for the collection of the data and those who participated in the study. The present research was funded by the Deutsche Forschungsgemeinschaft (HU 1847/3-1) and the University of Wuerzburg in the funding programme Open Access Publishing.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.00529/full#supplementary-material

memory: experiments and a computational model. Cogn. Psychol. 66, 157–211. doi: 10.1016/j.cogpsych.2012.11.001

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Pieczykolan and Huestegge. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Differential Impact of Visuospatial Working Memory on Rule-based and Information-integration Category Learning

#### Qiang Xing1,2 \* † and Hailong Sun2†

1 Department of Psychology, Guangzhou University, Guangzhou, China, 2 Management School, Jinan University, Guangzhou, China

Previous studies have indicated that the category learning system is a mechanism with multiple processing systems, and that working memory has different effects on category learning. But how does visuospatial working memory affect perceptual category learning? As there is no definite answer to this question, we conducted three experiments. In Experiment 1, the dual-task paradigm with sequential presentation was adopted to investigate the influence of visuospatial working memory on rule-based and information-integration category learning. The results showed that visuospatial working memory interferes with rule-based but not information-integration category learning. In Experiment 2, the dual-task paradigm with simultaneous presentation was used, in which the categorization task was integrated into the visuospatial working memory task. The results indicated that visuospatial working memory affects information-integration category learning but not rule-based category learning. In Experiment 3, the dualtask paradigm with simultaneous presentation was employed, in which visuospatial working memory was integrated into the category learning task. The results revealed that visuospatial working memory interferes with both rule-based and information-integration category learning. Through these three experiments, we found that, regarding the rule-based category learning, working memory load is the main mechanism by which visuospatial working memory influences the discovery of the category rules. In addition, regarding the information-integration category learning, visual resources mainly operates on the category representation.

Keywords: visuospatial working memory, visual processing, rule-based category structure, informationintegration category structure, executive function, dual-task paradigm

# INTRODUCTION

Categorization is a fundamental decision-making process that allows us to meaningfully parse the world and group similar objects together so that they can be treated equivalently (Rabi and Minda, 2014). It enables us to apply what we have learned about one thing and generalize that knowledge to other things of the same kind. For example, after learning the hard way that a

#### Edited by:

Tilo Strobach, Medical School Hamburg, Germany

#### Reviewed by:

Claudia C. von Bastian, Bournemouth University, UK Wei Deng, University of Macau, China

> \*Correspondence: Qiang Xing<EMAIL_ADDRESS>†Co-first authors

#### Specialty section:

This article was submitted to Cognition, a section of the journal Frontiers in Psychology

Received: 22 November 2016 Accepted: 23 March 2017 Published: 07 April 2017

#### Citation:

Xing Q and Sun H (2017) Differential Impact of Visuospatial Working Memory on Rule-based and Information-integration Category Learning. Front. Psychol. 8:530. doi: 10.3389/fpsyg.2017.00530

**Abbreviations:** COVIS, competition between verbal and implicit systems model; II, information-integration; II-C, information-integration category structure control group; II-V, information-integration category structure experimental group; RB, rule-based; RB-C, rule-based category structure control group; RB-V, rule-based category structure experimental group.

particular mushroom is probably poisonous, it is highly adaptive to generalize that knowledge to other similar mushrooms rather than to have to learn the hard way every time a new mushroom is encountered (Richler and Palmeri, 2014).

A large number of studies have indicated that category learning contains multiple classes of processing systems (Maddox et al., 2004; Richler and Palmeri, 2014; Xing and Sun, 2015), which have been explained by different theoretical models, such as exemplar-similarity (Patalano et al., 2001), family-resemblance (Yamauchi and Markman, 1998), and so on. The COVIS is so far the most influential multi-system theory, according to which there are at least two independent systems that exist in human category learning. One is the verbal system that is based on hypothesis testing and is under the control of consciousness, which is also influenced by working memory. The other is an implicit system that solves categorization tasks by learning to associate a response with regions of perceptual space, which is based on reinforcement and is independent of working memory (Ashby et al., 1998).

This has led to an extensive series of studies that have compared the learning of RB and II category structures (**Figure 1**). Given that the categorization of the RB and II structures depends primarily on the verbal and implicit systems, respectively, it is possible to test two kinds of prediction made by the COVIS model (Dunn et al., 2012). For the RB category structure, the classification rules are easy to verbalize and a judgment rule does not require the integration of two dimensions. For example, consider a category set in which round objects belong to one group and square objects belong to another group. These categories could be learned by applying the easy to verbalize rule that "category 1 objects are round." However, in contrast, the II category structure defines category membership according to the conjoint values on two or more dimensions using rules that are not easy to verbalize (e.g., if the size of a circle is greater than x and the orientation of a line is greater than y, then the stimulus is a member of category A). Consequently, such structures cannot be learned by the verbal system, which must eventually yield control of the response to the implicit system (Maddox et al., 2004; Worthy et al., 2013; Richler and Palmeri, 2014).

Furthermore, according to the COVIS model, working memory involves the ability to store information transiently and to perform cognitive activities, and it has different effects on the RB and II category structures. In other words, if working memory tasks are presented concurrently, the RB category learning will be disturbed, while the II category learning will not be affected; this has been verified by a large number of studies (Maddox et al., 2004; Zeithamova and Maddox, 2006; Filoteo et al., 2010). It is worth noting that the experiments mentioned above all involved verbal working memory. However, visuospatial working memory is another important type of working memory (Baddeley and Logie, 1999). There is good evidence that verbal and visuospatial working memory rely on different neural systems (Goldman-Rakic, 1998). Compared with verbal working memory, visuospatial working memory not only includes working memory load but also involves visual resources. Additionally, processing in the implicit system depends critically on the visual stimulus's representation in the inferotemporal cortex. This representation may be disrupted by the presence of a visuospatial task (Casale and Ashby, 2008). Thus, one hypothesis is that the presence of a visuospatial working memory task will affect II category learning. However, the existing COVIS model does not distinguish the types of working memory, and previous studies have mainly focused on the effects of verbal working memory on RB and II category learning. As such, the question naturally arises of how visuospatial working memory affects the RB and II category learning.

Using the dual-task paradigm that involves simultaneous presentation of visual stimuli, Miles and Minda (2011) found that a high level of working memory load could impair the RB category learning, which confirms that working memory plays a significant role in the RB category learning. In addition, their study found that the visual processing of the visuospatial working memory task affected the II category learning, which was not related to the level of working memory load. However, the study used the RB category structure that relies on a single dimension to perform categorization, whereas the II category structure takes two dimensions into consideration; thus, the difference in the difficulty of the category structure could affect the categorization results (Sun and Xing, 2014; Zaki and Kleinschmidt, 2014). Miles and Minda (2011) believe that visuospatial working memory mainly depends on a function that influences implicit category learning, which is not related to working memory load. Nevertheless, the study did not explain how visual processing affects category learning. Moreover, Miles et al. (2014) used a simultaneous-task paradigm in which the category learning task is integrated with a verbal working memory task, and the results showed that working memory load can affect II category learning.

It can be seen that there is still much debate about the influence of visuospatial working memory on the RB and II category structure, especially about how visuospatial working memory affects II category learning. If it does affect II structure category learning, would there be any difference in the results of the above-mentioned studies? Several studies may offer some insight to solve these problems. According to the hypothesis of Zeithamova and Maddox (2007), the process of category learning may include the following steps: (1) representation of the stimulus and (2) generation and testing of a categorization rule for the RB category learning (i.e., learning of a categorization criterion for the II category learning). Thus, we suppose that if the process of category learning really includes these steps, the working memory load from the visuospatial working memory task would be critical primarily for rule generation and testing (because the verbal system depends upon working memory load), while the visuospatial resource from the visuospatial working memory task may influence the representation of the stimulus for the II category learning (because the implicit system learns the association between a region of perceptual space and an overt response).

In practical terms, the implicit category learning system establishes a connection between a specific perceptual space and the specific action, and the representation of the category stimuli is involved in the category learning. This is indicated in the study by Dunn et al. (2012) in which a Gabor mask

structure, decisions are made based on two or more dimensions (in this example, frequency and orientation).

presented after the II category structure interfered with the visual processing of the category stimuli and affected the perceptual representation of the II category learning. Maddox et al. (2004) found that the addition of a working memory load in the sequential presentation impaired RB learning but had little effect on II learning. Furthermore, when studying the effects of working memory on category learning, dual-task paradigms are usually adopted, such as the dual task with sequential presentation and the dual task with simultaneous presentation, in which the different locations of working memory are manipulated (Miles and Minda, 2011).

Therefore, the inconsistencies in the previous studies are much more likely to be caused by the fact that visual resources and working memory load may affect the different processing stages of category learning. Based on this point of view, we conducted three experiments in which we manipulated the different dual tasks in order to examine whether they would influence the different cognitive processing stages of category learning. We aimed to investigate the process of cognitive processing during which visuospatial working memory affects II and RB category learning, especially visual resources and working memory. In Experiment 1, sequential presentation tasks were adopted. In Experiment 2, we used the embedded paradigm in which the category learning task was embedded in the visuospatial working memory task. In Experiment 3, we used a concurrent-task methodology in which the working memory task was embedded in the classification task.

### EXPERIMENT 1

# Materials and Methods

#### Participants

We randomly selected 84 participants (40 male, 44 female) who were participating in the secondary post-graduate examination held by the education school of Guangzhou University. The average age was 19.31 years (±2.15). All participants were righthanded, had normal or corrected-to-normal vision, and had no color blindness or color weakness problems. This study was carried out in accordance with the recommendations of the ethical committee of Guangzhou University with written informed consent from all participants. All participants gave written informed consent in accordance with the Declaration of Helsinki.

#### Experimental Materials

The categorization stimuli were generated using the same procedures as Dunn et al. (2012). The stimuli were sine wave gratings that varied in spatial frequency and orientation. Twenty stimuli in each of the four categories were generated by sampling randomly from the same four parameter distributions used by Dunn et al. (2012). The Psychophysics Toolbox in MATLAB (MathWorks, Natick, MA, USA) was used to generate the RB and II category structures (Brainard, 1997). Actual values of spatial frequency (f) and orientation (o) were generated from a random sample (x, y) from these distributions using the following transformations: f = 0.25 + x/50, o = y.π/500. All of the stimuli were 200 × 200 pixel images. The specific dimensions of the parameters are shown in **Table 1**.

A visuospatial working memory task was created that was analogous to the Sternberg working memory task used in Maddox et al. (2004). In this task, the participant was asked to remember four locations out of nine possible locations (analogous to remembering four numerical digits sampled from nine possible digits). First, a fixation cross (i.e., a "+") appeared in the middle of the screen, indicating the beginning of the dot pattern task. Next, nine gray dots appeared on the screen and the memory set turned red, followed by a series of four rapidly presented masks. Each mask was a 9 × 9 grid of gray dots, half of which had a red center. Next, the memory probe appeared on the screen along with the question "Was this dot originally red?" Participants made a response using the appropriate button and received feedback.

µX represents the average value, σ 2 X represents the variance, cov represents the variance covariance, RB represents the rule-based category structure, and II represents the information-integration category structure. According to the frequency and direction, each category structure was divided into four categories: A, B, C, and D.

#### Experimental Design

The experiment had a 2 (task condition: working memory group vs. control group) × 2 (category structure: II vs. RB) × 4 (block) repeated-measures design, in which task condition and category structure were the between-subjects variables and learning block was the within-subjects variable. The dependent variables were the accuracy of categorization in the visuospatial working memory task and the category learning. The number of participants followed that used by previous studies (Stanton and Nosofsky, 2007; Miles and Minda, 2011). All participants were assigned randomly to one of four groups, with 21 participants in each group. Two participants in the RB task control group (RB-C) were removed due to interruption during the experiment; thus, the data from 19 participants were used. One participant in the RB task experimental group (RB-V) was removed for the same reason; therefore, the data of 20 participants were used. The data of 21 participants were used in the II task experimental group (II-V), while that of 19 participants were used in the II control group (II-C) after deleting the data of two participants for the same reason.

#### Experimental Procedure

The dual-task experimental paradigm with sequential presentation was used (**Figure 2**). The experimental procedure included four blocks, each of which had 80 trials. First, participants tried to complete the category learning task. Within each block, all 80 stimuli were presented in a random order. Participants were told to learn which of four categories (labeled as 1, 2, 3, and 4) each stimulus belonged to. After the presentation of a fixation cross (i.e., a "+") for 800 ms, the screen was presented of the RB or II category structure, which could be considered by participants as belonging to one of the four categories of A, B, C, or D, and for which they pressed the 1, 2, 3, or 4 number key on the keyboard, respectively. After the responses were given, the stimuli disappeared, and the feedback was provided immediately; the participants were informed not only whether their responses were correct or not, but also to which category each of the stimuli belonged, and the correct sine wave grating was shown to the participants at the same time.

The visuospatial working memory task followed the category learning task. The gray squares were presented on the screen for 500 ms. Then, four randomly selected gray squares all turned red for 500 ms before disappearing. After this, another gray square turned red (which could be one of the four squares that had changed color from gray to red or it could be a new square), followed by a series of four quickly presented masks. Each mask was a 9 × 9 grid of gray squares, half of which had a red center. The participants were required to determine whether this square had appeared before or not. If they believed that it had been presented before, they pressed the "F" key. If they believed that it had not been presented before, they pressed the "J" key. After the responses were given, the participants were provided with feedback for 800 ms about whether they were right or wrong. In contrast, the control group was not presented with the visuospatial working memory task. They were required to only perform the category learning task, which was the same as for the experimental group.

#### Results

#### Visuospatial Working Memory Task Performance

The mean accuracy rates averaged across participants were analyzed. The average accuracy of the visuospatial working memory task in the RB-V group was 0.71 (±0.18), and that of the participants in the II-V group was 0.71 (±0.17). An independentsamples t-test showed that there was no significant difference between the two groups, t(38) = −0.097, p = 0.977, indicating that there was no difference in the degree of cognitive resources consumed by participants in the RB group and II group when performing the visuospatial working memory task.

#### Analysis of the Overall Results

We conducted a 2 (category structure) × 2 (condition) × 4 (block) mixed design analysis of variance. This revealed a main effect of block, F(3,225) = 57.28, p < 0.001, η 2 p = 0.43, indicating learning, and a main effect of condition, F(1,75) = 4.24, p = 0.043, η 2 p = 0.05, indicating superior accuracy overall for the control condition compared to the visuospatial working memory condition. There was no main effect of category structure, F < 1, and no significant interactions between block and category structure, F(3,225) = 1.37, p = 0.253, between block and condition, F(3,225) = 1.27, p = 0.287, or between category structure and condition, F(1,94) = 1.42, p = 0.237. However, there were significant interactions between block, category structure, and condition, F(3,225) = 4.53, p = 0.004, η 2 p = 0.06. The interactions with category structure indicate that the condition had a greater effect on RB learning than II learning and that this difference increased across the blocks (**Figure 3**).

Furthermore, for the RB category structure, a 2 (condition) × 4 (block) repeated-measures analysis of variance was performed (**Table 2**). The results showed that the main effect of the block was significant, F(3,111) = 18.45, p < 0.001, η 2 p = 0.33, indicating that learning occurred. The significant main effect of the condition, F(1,37) = 4.45, p = 0.042, η 2 p = 0.11, indicated that the participants' learning was significantly different

in the different conditions. Furthermore, the interaction between the condition and block was significant, F(3,111) = 4.50, p = 0.005, η 2 p = 0.11, indicating that, in the two conditions, the findings of the different blocks were significantly different. The analysis of the simple effects showed that the difference in results between the experimental group and control group was not significant in Block 1 (p = 0.553). In Block 2, the results of the RB task categorization of the experimental group were significantly lower than those of the control group (p = 0.050). In Block 3, the results of the categorization task of the experimental group were not significantly different from those of the control group (p = 0.077). In Block 4, the results of the categorization task of the experimental group were significantly lower than those of the control group (p = 0.008) (**Figure 3**). All of these findings indicate that performing the visuospatial working memory task immediately after the feedback impaired the RB category learning.

For the II category structure, the 2 (condition) × 4 (block) repeated-measures analysis of variance was performed in the same way. The results showed that the main effect of the block was significant, F(3,114) = 43.53, p < 0.001, η 2 p = 0.53, indicating that learning occurred. The main effect of the condition was not significant, F(1,38) = 0.46, p = 0.503, and the interaction between the condition and block was not significant, F(3,111) = 0.75, p = 0.525 (**Figure 3**). All of these findings indicate that the visuospatial working memory task conducted immediately after the feedback did not influence the II category learning.

To summarize, we found that a visuospatial working memory task interferes with RB but not II category learning when the dual-task experimental paradigm with sequential presentation is used. The significant effect of visuospatial working memory on RB category learning replicates the effect observed in Maddox et al. (2004) with a verbal working memory task, and extends

TABLE 2 | Effects of working memory on the II and RB category structures (M ± SD).

RB and II represent the ule-based category structure and the informationintegration category structure, respectively. RB-C represents the rule-based category structure control group, RB-V represents the rule-based category structure experimental group, II-C represents the information-integration Category structure control group, and II-V represents the information-integration category structure experimental group. The numbers 1, 2, 3, and 4 represent the four blocks of the learning process.

the effect to a visuospatial working memory task. As outlined in the introduction, the RB learning involves generating a representation of the stimulus, response, and feedback. Thus, placing a load on a separate visuospatial working memory store will affect the feedback processes. In contrast, the II category learning appears to occur incrementally in a fashion that is heavily dependent on immediate feedback. As such, would a nested form of visual working memory affect category learning? In Experiment 2, we used an embedded paradigm in which the category learning task was embedded in the visuospatial working memory task in order to examine the effect of the working memory on RB and II category learning.

# EXPERIMENT 2

# Materials and Methods

#### Participants

We randomly selected 87 students (40 male, 47 female) who were participating in the secondary post-graduate examination held by the education school of Guangzhou University. The average age was 19.61 years (±1.62). Twenty participants were assigned to the RB-C condition, 24 to the RB-V condition, and 22 and 21 to the II-V condition and the II-C condition, respectively. We aimed for 20 participants per condition based on previous studies, such as Miles and Minda (2011). All of the participants were righthanded, had normal or corrected-to-normal vision, and had no color blindness or color weakness problems. Written informed consent was obtained from all participants before starting the investigation in accordance with the Declaration of Helsinki, and the study was approved by the ethical committee of Guangzhou University.

#### Experimental Materials

These were the same as in Experiment 1.

#### Experimental Design

This was the same as in Experiment 1.

#### Experimental Procedure

The dual-task experimental paradigm with simultaneous presentation was employed in which the category learning task was integrated into the visuospatial working memory task (**Figure 4**). The whole experimental process was divided into three stages.

In the first stage, the gray squares were presented for 500 ms. Four random squares of the screen then turned red for 500 ms, after which the screen disappeared. The masking appeared four times in sequence, each one lasting for 250 ms (1000 ms in total), in which random flickering squares were presented on each screen. The participants did not have to respond during this stage.

The second stage was the category learning task. A fixation cross (i.e., a "+") was presented for 800 ms, after which it disappeared. The screen then showed the RB or II category structure for 200 ms, after which it disappeared, and the response screen of the category learning was presented, in

which there were four categories (i.e., A, B, C, and D). The participants determined which category structure it belonged to and pressed the relevant key on the keyboard for each category (i.e., 1, 2, 3, and 4, respectively). The screen disappeared after the responses were given, and the instant feedback was then provided.

In the third stage, after finishing the category learning task, the gray squares were randomly presented on the screen for 500 ms, followed by the detection screen, in which a gray square turned red (which may have occurred in the first stage or not) and the participants were required to determine whether this square had appeared before or not. If the participant believed that it had been presented before, they pressed the "F" key. If the participant believed that it had not been presented before, they pressed the "J" key. The detection screen disappeared after the responses were given, and the simple feedback was then provided. The whole experimental procedure included four blocks, each of which had 80 trials. If the participant's performance on the visuospatial working memory task in each block was lower than 80%, a warning window popped up at the end of the block. For the control group, there was no visuospatial working memory task and participants were required to only perform the category learning task, which was the same as for the experimental group.

# Results

#### Concurrent Task Performance

The average accuracy for the visuospatial working memory task of participants in the RB-V group was 0.72 (±0.15), and that of participants in the II-V group was 0.68 (±0.11). An independentsamples t-test showed that there was no significant difference between the two groups, t(44) = −0.943, p = 0.320, indicating that there was not a difference in the degree of cognitive resources consumed by participants in the RB and II groups when performing the visuospatial working memory task.

#### Analysis of the Category Learning

We conducted a 2 (category structure) × 2 (condition) × 4 (block) mixed design analysis of variance. This revealed a main effect of block, F(3,249) = 62.33, p < 0.001, η 2 p = 0.43, indicating learning, and a main effect of category structure, F(1,83) = 8.98,

p = 0.004, η 2 p = 0.10, indicating superior accuracy overall for the RB category structure compared to the II category structure. There was no main effect of condition, F(1,83) = 1.20, p = 0.276, and no significant interactions between block and category structure, F(3,249) = 2.26, p = 0.082, or between block, category structure, and condition, F < 1. However, there were significant interactions between block and condition, F(3,249) = 2.90, p = 0.036, η 2 p = 0.03 and between category structure and condition, F(1,83) = 3.99, p = 0.049 (**Figure 5**).

Furthermore, for the RB category structure, we performed a 2 (condition) × 4 (block) repeated-measures analysis of variance (**Table 3**). The results showed that the main effect of the block was significant, F(3,126) = 34.36, p < 0.001, η 2 p = 0.45, indicating the existence of a learning effect. The main effect of the condition was not significant, F(1,42) = 0.33, p = 0.567. In addition, the interaction between the condition and block was not significant, F(3,126) = 0.81, p = 0.493. All of these findings indicate that the visuospatial working memory task did not affect the RB category learning.

For the II structure, the 2 (condition) × 4 (block) repeatedmeasures analysis of variance showed that the main effect of the block was significant, F(3,123) = 29.07, p < 0.001, η 2 p = 0.42, indicating the existence of a learning effect. The main effect of the condition was significant, F(1,41) = 6.17, p = 0.017,

TABLE 3 | Effects of visuospatial working memory on the II and RB category structures under the condition of simultaneous presentation (M ± SD).

Frontiers in Psychology | www.frontiersin.org April 2017 | Volume 8 | Article 530 |

η 2 p = 0.11, as was the interaction between the condition and block, F(3,123) = 4.03, p = 0.009, η 2 p = 0.09, indicating that in these two conditions, the results of the different blocks were significantly different.

In order to further investigate the interaction between the conditions and blocks in detail, an analysis of the simple effects was conducted. The results showed that, in Block 1, there was no significant difference in the results between the II-C and II-V groups, p = 0.200; in Block 2, the results of the II-C group were significantly higher than those of the II-V group, p = 0.027; in Block 3, the results of the II-C group were significantly higher those of the II-V group, p = 0.008; and in Block 4, the results of the II-C group were significantly higher than those of the II-V group, p = 0.015 (**Figure 5**).

In Experiment 2, the dual-task paradigm with simultaneous presentation was used, in which the categorization task was integrated into the working memory task. The results indicated that visuospatial working memory affects the II category learning but not the RB category learning. On the contrary, the RB category learning was impaired by the visuospatial working memory task in Experiment 1. Due to there being a similar visuospatial working memory task, the two studies should have found the same effect of the visuospatial working memory task on RB and II category learning (according to the COVIS model, the study results are only affected by the degree of working memory load). Yet, the results of Study 2 showed different patterns of interference with the II category learning and RB category learning compared to the results of Study 1. These results are not explained by the COVIS model. However, Studies 1 and 2 differed in the location of the working memory task. We infer that the visuospatial resource may interfere with the perception of the stimuli. Experiment 2 provides the first piece of evidence that visuospatial working memory affects II category learning. This observed effect is consistent with Zeithamova and Maddox's (2007) hypothesis of the stages of cognitive processing.

# EXPERIMENT 3

fpsyg-08-00530 April 7, 2017 Time: 10:41 # 9

# Materials and Methods Participants

We randomly selected 67 students (33 male, 34 female) who were participating in the secondary post-graduate examination held by the education school of Guangzhou University. The average age was 18.98 years (±0.79). There were 20 participants assigned to the RB-C condition, 23 to the RB-V condition, and 24 and 23 to the II-V condition and the II-C condition, respectively. All of the participants were right-handed, had normal or correctedto-normal vision, and had no color blindness or color weakness problems. Written informed consent was obtained from all participants before starting the investigation in accordance with the Declaration of Helsinki, and the study was approved by the ethical committee of Guangzhou University.

#### Experimental Materials

These were the same as in Experiment 1.

#### Experimental Design

This was the same as in Experiment 1.

#### Experimental Procedure

The dual-task experimental paradigm with simultaneous presentation was adopted for the visuospatial working memory experimental group, in which visuospatial working memory was integrated into the category learning. The whole experimental procedure was divided into three stages (**Figure 6**).

In the first stage, the fixation cross (i.e., a "+") was presented for 800 ms, after which it disappeared. The screen then showed the RB or II category structure for 200 ms, after which it disappeared. During this stage, the participants were not required to respond.

The second stage included the visuospatial working memory task. The gray squares were randomly presented on the screen for 500 ms. Four random squares then changed color from gray to red, which lasted for 500 ms, after which the screen disappeared. The masking appeared four times in a row, with each lasting for 250 ms. After that, the detection screen was presented in which a gray square turned red (which may have appeared before or not) and the participants were required to determine whether this square had appeared before or not. If they believed that it had been presented before, the "F" key was pressed; otherwise, the "J" key was pressed. As soon as the responses were given, the detection screen disappeared, and the feedback about whether the participants were right or wrong was provided.

In the third stage, at the end of the visuospatial working memory task, the response screen of the category learning task was presented, in which there were four categories (i.e., A, B, C, and D). The participants decided to which category structure it belonged, and pressed the counterpart key on the keyboard (i.e., 1, 2, 3, and 4, respectively). The screen disappeared after the responses were given, and abundant feedback was provided instantly. The experimental procedure included four blocks, each of which had 80 trials. For the control group, there was no visuospatial working memory task, and participants were required to only perform the category learning task. There was a delay of 3000 ms (the shortest presentation time in the whole visuospatial working memory task) for the gray screen between the screen presenting the category structure and the response screen, so that it was the same as for the conditions of the experimental group. The category learning task of the control group was the same as that of the experimental group (**Figure 6**).

### Results

#### Concurrent Task Performance

The average accuracy of accomplishing the visuospatial working memory task of participants in the RB-V group was 0.84 (±0.14), and that of participants in the II-V group was 0.88 (±0.09). An independent-samples t-test showed that there was no significant difference between the two groups, t(45) = 1.16, p = 0.167, indicating that there was not a difference in the degree of cognitive resources consumed by participants in the RB and II groups when performing the visuospatial working memory task.

#### Analysis of the Overall Results for the Category Learning

We conducted a 2 (category structure) × 2 (condition) × 4 (block) mixed design analysis of variance. This revealed a main effect of block, F(3,258) = 74.76, p < 0.001, η 2 p = 0.47, indicating learning, and a main effect of condition, F(1,86) = 14.52, p < 0.001, η 2 p = 0.14, indicating superior accuracy overall for the control condition compared to the visuospatial working memory condition. There was no main effect of category structure, F < 1, and no significant interactions between block and category structure, F < 1, between block and condition, F(3,258) = 1.44, p = 0.232, between category structure and condition, F < 1, or between block, category structure, and condition, F(3,258) = 1.13, p = 0.336 (**Figure 7**).

Furthermore, for the RB category structure, we performed a 2 (condition) × 4 (block) repeated-measures analysis of variance (**Table 4**). The results showed that the main effect of the block was significant, F(3,123) = 25.36, p < 0.001, η 2 p = 0.38, indicating the existence of a learning effect. The main effect of the condition was also significant, F(1,41) = 7.87, p = 0.008, η 2 p = 0.16; the categorization results of participants in the RB-V group were significantly lower than those of participants in the RB-C group. In addition, the interaction between the condition and block was not significant, F(3,123) = 1.76, p = 0.158. The results showed that the visuospatial working memory task impaired the learning performance in the RB category structure.

For the II category structure, we also performed a 2 (condition) × 4 (block) repeated-measures analysis of variance. The results showed that the main effect of the block was significant, indicating the existence of a learning effect, F(3,135) = 58.41, p < 0.001, η 2 p = 0.57. In addition, the main effect of the condition was significant, F(1,45) = 6.49, p = 0.014, η 2 p = 0.13, indicating that the results of participants in the II-C group were significantly higher than those of participants in the II-V group. These findings suggest that the visuospatial working memory task can similarly affect the learning of the II category structure. The interaction between the condition and block was not significant, F(3,135) = 0.31, p = 0.817.

In Experiment 3, the dual-task paradigm with simultaneous presentation was employed, in which visuospatial working memory was integrated into the category learning task. The results revealed that visuospatial working memory interferes with both RB and II category learning, which means that any visual working memory task that involves visual resources, such as the one used in Experiment 2, also disrupts the II category learning system. This finding help to clarify the workings of the implicit system. This system could certainly be a procedural system but it could also rely heavily on visual resources to learn how to classify visually similar stimuli into the same category.

# GENERAL DISCUSSION

Previous research has made clear the importance of working memory for RB categories (Zeithamova and Maddox, 2006, 2007; DeCaro et al., 2008; Minda et al., 2008; Rabi et al., 2015). We were interested in further exploring the effect of visuospatial working memory on RB and II category learning, especially investigating the role of visual processing and executive functioning. In Experiment 1, the dual-task paradigm with sequential presentation was adopted to investigate the influence of visuospatial working memory on implicit and explicit category learning. The results showed that visuospatial working memory interferes with RB but not II category learning. In Experiment 2, the dual-task paradigm with simultaneous presentation was used, in which the categorization task was integrated into the working memory task. The results indicated that visuospatial working memory affects II category learning but not the RB learning system. In Experiment 3, the dual-task paradigm with simultaneous presentation was employed, in which visuospatial working memory was integrated into the category learning task. The results revealed that visuospatial working memory interferes with both RB and II category learning. Through these three experiments, we found that, regarding the RB category structure, executive function is the main mechanism by which visuospatial

TABLE 4 | Effects of working memory on the II and RB category structures under the condition of simultaneous presentation (M ± SD).

RB and II represent the rule-based category structure and the informationintegration category structure, respectively. RB-C represents the rule-based category structure control group, RB-V represents the rule-based category structure experimental group, II-C represents the information-integration category structure control group, and II-V represents the information-integration category structure experimental group. The numbers 1, 2, 3, and 4 represent the four blocks of the learning process.

working memory influences the rules and the discovery of the rules but not the category representation. In addition, regarding the II category structure, visual processing mainly operates on the category representation, which interferes with the connection between the interference space and the specific action.

# Visuospatial Working Memory Affects the RB Category Learning

Our study showed that visuospatial working memory affects the RB category learning, and that working memory plays an important role during this process. During the process of the RB category learning, working memory is used to update and retrieve the rules from memory that are tested by feedback, while executive function is also needed to restrain the interference of irrelevant dimensions. Presenting the visuospatial working memory tasks sequentially occupies working memory and, as a result, the verification of rules conducted by the feedback is interfered with (Zeithamova and Maddox, 2007; Grimm and Maddox, 2013). When the visuospatial working memory task was embedded in the category learning (as in Experiment 3), we believe that the visuospatial working memory task mainly interfered in the discovery of the categorization rules; as soon as the participants were successful in identifying the categorization rules, they were able to learn successfully and their accuracy increased significantly.

However, when the category learning task was embedded in the visuospatial working memory task, the results showed that the effect of visuospatial working memory on RB category learning disappeared. That is, visuospatial working memory did not affect the RB category structure. It is worth noting that, although they are both task paradigms with simultaneous presentation, the existing studies suggest that, compared with the condition in which the visuospatial working memory task is integrated into the category learning task (as in Experiment 3), the condition in which the category learning task is integrated into the visuospatial working memory task requires a higher level of executive function. This is because, during the process of accomplishing the category learning task, participants need to use working memory consistently to retain the beginning of the visuospatial working memory task (Miles and Minda, 2011).

Therefore, according to the assumption of the COVIS model, the RB category learning should be hindered more heavily when the category learning task is integrated into the visuospatial working memory task than when the visuospatial working memory task is integrated into the category learning task. However, our experimental results contradicted this. Why was there such a result? We think that this was caused by the fact that working memory or executive function can affect a specific phase of cognitive processing during category learning. Although the condition in which the category learning task is integrated into the visuospatial working memory task requires more executive function, in the RB category structure learning, the perception of the category stimuli does not rely on executive function. Zeithamova and Maddox (2007) indicated that visuospatial working memory is more likely to be used to represent the optimal categorization criteria, while during explicit category learning, it uses assumptions to examine the categorization rules and relies on working memory to keep these categorization rules in mind.

# Visuospatial Working Memory Affects the II Category Learning

According to the COVIS model, working memory does not affect the learning of the II category structure, because II category learning establishes a connection between a specific perceptual area of the brain and a specific action, relying on the implicit category learning system. However, our results showed that the visuospatial working memory task also affected the RB and II category structure. When simultaneous tasks were used, the executive function of the visuospatial working memory task, no matter whether at a high or low level, affected the results of the II category structure, which indicates that executive function is not the key factor that affects the II category structure, whereas the visual processing of the visuospatial working memory task plays an important role. Dunn et al. (2012) found that the type of grating mask presented after the category stimuli affected the perceptual representation of the implicit category learning, which to some extent indicates that in the learning of the II category structure, visual processing is more likely to affect the original perceptual representation of the category stimuli. In addition, it has been found that the visuospatial working memory of children is slower than that of adults, but the visual processing capacity of children is fully developed and is not lower than that of adults (Huang-Pollock et al., 2011). Minda et al. (2008) showed that the learning results of children (5–7 years old) for the II category structure was not significantly different from that of adults.

How does visual processing affect the implicit category system? We think that the visual processing of the visuospatial working memory task affects the different processing stages of the category learning. By comparing Experiment 1 with Experiment 3, we can observe that, in the condition of the sequential presentation of the dual tasks, the visuospatial working memory task did not affect the II category structure because the II category learning depends on the connection between a specific area of the brain and a specific action.

The primary role of feedback is to provide instant reinforcement, and this stage of forming the category criterion does not necessarily rely on working memory and visual processing; it is more likely to involve implicit unconscious processing. As a result, the visuospatial working memory task with sequential presentation does not influence the II category learning, whereas when dual tasks with simultaneous presentation are used, the intensity of the executive function when the visuospatial working memory task is integrated into the category learning task is the same as when the tasks are presented sequentially. This indicates that when visuospatial working memory influences the II category structure, it is the location of the visuospatial working memory rather than the intensity of the executive function that is actually operating.

By comparing Experiment 2 with Experiment 3, we can observe that the II-V group is always better than the II-C group, which indicates that presenting the visuospatial working memory task after the II category structure has a negative influence on the category learning results from the very beginning of the learning. This suggests that an individual is more likely to be influenced by visual processing during the stage of category representation. However, when the category learning was integrated into the visuospatial working memory task, as for the overall learning cycle, there was no significant difference in results between the II-C and II-V groups in Block 1, and the results of the II-V group were significantly higher than those of the control group from Block 2 onward. Visuospatial working memory involves visual processing and visual perception, while the implicit category system needs to project a specific representation to a specific area of the brain and depends on the visual and perceptual memory systems to improve the stimulus representation that has been recognized and processed, especially to distinguish between representations that are similar but not the same. Therefore, when the visuospatial working memory task is presented at the very beginning, it does not affect the process of the representation of stimuli from different category learning phases, but it does affect the establishment of the connection between the perceptual space and the specific action (i.e., it affects the representation of the category criterion).

# CONCLUSION

(1) Visuospatial working memory affects RB and II category learning.

(2) Regarding the RB category structure, visuospatial working memory influences the discovery of rules in particular.

(3) Regarding the II category structure, visual processing primarily operates on the category representation, which interferes with the connection between the perceptual space and the specific action.

# ETHICS STATEMENT

The study was approved by the ethical committee of Guangzhou University. Because the data were analyzed anonymously, and no apparent ethical research complication with participation could be identified, informed oral consent was recommended and obtained from participants before data collection. Participants were given the opportunity to refuse to participate, to omit questions or to withdraw from the study at any time without penalization.

# AUTHOR CONTRIBUTIONS

QX and HS conceived and designed experiments. HS carried out experiments and analyzed experimental results. HS and QX wrote the manuscript.

# FUNDING

This work was supported by the National Natural Science Foundation of China (31571144). The "12th Five-Year plan" of Guangzhou Education Science (No.<PHONE_NUMBER>): the psychological mechanism of self regulated learning and efficient learning of junior high school students.

# REFERENCES

fpsyg-08-00530 April 7, 2017 Time: 10:41 # 13

adults. J. Exp. Psychol. Learn. Mem. Cogn. 34, 1518–1524. doi: 10.1037/a001 3355

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Xing and Sun. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Effect of Redundant Haptic Information on Task Performance during Visuo-Tactile Task Interruption and Recovery

#### Hee-Seung Moon1, 2, Jongsoo Baek 2 and Jiwon Seo1, 2 \*

*1 School of Integrated Technology, Yonsei University, Incheon, Korea, 2 Yonsei Institute of Convergence Technology, Yonsei University, Incheon, Korea*

Previous studies have revealed that interruption induces disruptive influences on the performance of cognitive tasks. While much research has focused on the use of multimodal channels to reduce the cost of interruption, few studies have utilized haptic information as more than an associative cue. In the present study, we utilized a multimodal task interruption scenario involving the simultaneous presentation of visual information and haptic stimuli in order to investigate how the combined stimuli affect performance on the primary task (cost of interruption). Participants were asked to perform a two-back visuo-tactile task, in which visual and haptic stimuli were presented simultaneously, which was interrupted by a secondary task that also utilized visual and haptic stimuli. Four experimental conditions were evaluated: (1) paired information (visual stimulus + paired haptic stimulus) with interruption; (2) paired information without interruption; (3) nonpaired information (visual stimulus + non-paired haptic stimulus) with interruption; and (4) non-paired information without interruption. Our findings indicate that, within a visuotactile task environment, redundant haptic information may not only increase accuracy on the primary task but also reduce the cost of interruption in terms of accuracy. These results suggest a new way of understanding the task recovery process within a multimodal environment.

#### Edited by:

*Tilo Strobach, Medical School Hamburg, Germany*

#### Reviewed by:

*Zaifeng Gao, Zhejiang University, China Ahu Gokce, Kadir Has University, Turkey*

> \*Correspondence: *Jiwon Seo<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Cognition, a section of the journal Frontiers in Psychology*

Received: *22 July 2016* Accepted: *23 November 2016* Published: *08 December 2016*

#### Citation:

*Moon H-S, Baek J and Seo J (2016) Effect of Redundant Haptic Information on Task Performance during Visuo-Tactile Task Interruption and Recovery. Front. Psychol. 7:1924. doi: 10.3389/fpsyg.2016.01924* Keywords: task interruption and recovery, multitasking, multimodal task, working memory, haptic stimuli

# INTRODUCTION

In daily life, people face various cognitive tasks, such as sending an e-mail or entering data into a computer in their workspace or home. Usually, these tasks are quite simple and completed with no errors. However, people often encounter circumstances in which another unexpected task interrupts the execution of the prior task. In practice, interruptions between multiple cognitive tasks occur frequently, and researchers have investigated these shifts in attention in workspaces via observational studies (Chisholm et al., 2000; Czerwinski et al., 2004; González and Mark, 2004). Numerous studies have attempted to identify how interruptions affect tasks and how people resume their original tasks after interruptions within a typical workspace (Czerwinski et al., 2004; Mark et al., 2005; Iqbal and Horvitz, 2007). For instance, a ring tone from a phone call, the arrival of a new e-mail, or a question from a colleague can all represent external interruptions that occur while performing a primary task that engages a person's attention (examples given by Fisher, 1998). With advanced technology, the number of complex situations and potential interruptions that divide people's attention has rapidly increased.

Recently, psychologists and human-computer interaction researchers have begun to focus on understanding the role of interruptions in cognitive control. A number of studies have revealed that interruptions are disruptive: Interruption by a secondary task causes interference in performing the primary task. Baddeley et al. (1984) demonstrated concurrent decreases in performance on two simultaneous tasks that require cognitive resources and therefore use working memory. Recent studies also focused on what makes interruptions disruptive, confirming this in two ways. Firstly, resuming the primary task requires more time following an interruption, a phenomenon referred to as resumption lag (Hodgetts and Jones, 2006; Monk et al., 2008; Brumby et al., 2013). In addition, interruptions lead to an increase in the likelihood of errors within the recovered task (Trafton et al., 2011; Brumby et al., 2013). These two prominent influences are common within different kinds of tasks, such as simple data-entry tasks (Zish and Trafton, 2014), sequential tasks (Trafton et al., 2011), cognitively demanding tasks (Borst et al., 2015), and decision-making tasks (Gathmann et al., 2015). Research in the field of human-computer interaction has also examined task switching and cognitive control in order to predict human task performance (Hornof et al., 2010). In addition, task switching has been noted for its effects on performance and mental load in both single-modal (Bailey et al., 2001) and multimodal user interfaces (Lu et al., 2013).

In order to make a precise prediction of performance on novel tasks, researchers have endeavored to elucidate the entire cognitive recovery process. Working memory is utilized for the maintenance and processing of information in the task at hand (Barrouillet et al., 2011) and is considered crucial for shifting cognitive tasks (Drews and Musters, 2015). Barrouillet et al. (2004) proposed a model of time-based resource management with regard to the maintenance and processing aspects of working memory. According to this model, information associated with the current task can undergo a decay process when attention toward the task is switched. In addition, task switching results in decreased recall performance (Liefooghe et al., 2008).

When people are faced with a situation in which their primary task is interrupted by a secondary task, information regarding the primary task is stored in working memory until resumption of the task following completion of the secondary task (Trafton et al., 2003). This ability to multitask is a common capability that allows most people to deal with interruptions without grave hardship. However, due to the limited capacity of working memory, the new information relevant to the secondary task can interfere with the information related to the primary task (Drews and Musters, 2015). The storage capacity of working memory has been researched for decades, and it is now wellknown that the central capacity is limited to a few chunks of information at a time (Cowan, 2001). Beyond the central capacity (i.e., shared memory capacity for several modalities), Saults and Cowan (2007) revealed the capacity of the peripheral memory for specific modalities (e.g., visual or auditory modality). As each modality has its own peripheral resources, humans can recall more information when both central and peripheral memory systems of different modalities are involved (Cowan et al., 2014).

The Memory-For-Goals (MFG) theory represents one of the most popular frameworks for conveying the effect of interruptions (Altmann and Trafton, 2002). The MFG theory states that the interruption and recovery processes are based on the idea that human memory has a required activation level for each task and its associated goal. Like the working memory model proposed by Barrouillet et al. (2004), the MFG theory asserts that activation associated with a cognitive task decays over time (Altmann and Trafton, 2002). Borst et al. (2015) also specified the interruption and recovery processes in terms of information transference between the problem state and declarative memory. The problem state is a resource that stores requisite information for performing a cognitive task. When a primary task is interrupted, the existing information in the problem state moves to declarative memory, and novel information associated with the interrupting task becomes stored in the problem state. After transference to declarative memory, information associated with the primary task decays over time in terms of a power function (Borst et al., 2015). In addition, the interrupting task increases its own activation level, which produces increased interference on the primary task (Altmann and Trafton, 2002). Several studies have supported this MFG framework, revealing that, when participants are interrupted such that they are required to perform a longer task, increases in resumption lag and the number of errors are observed (Hodgetts and Jones, 2006; Monk et al., 2008; Brumby et al., 2013; Altmann et al., 2014; Borst et al., 2015).

A large proportion of studies have conducted simple visual tasks in laboratory environments using monitors, whereas relatively few studies have utilized a multi-sensory task environment. As real-world tasks occur under multi-sensory circumstances, the recovery process should be studied within multimodal task environment. Hodgetts et al. (2014) and Keus van de Poll and Sörqvist (2016) focused on the auditory modality and investigated the effects of auditory distraction on a visual task recovery. Hodgetts et al. (2014) implemented a command and control task interrupted by yes/no questionnaires with auditory noise. Keus van de Poll and Sörqvist (2016) utilized a writing task interrupted by arithmetic problems with background speech. The results of both studies indicated that the interruption recovery process in a visual modality is affected by distraction from an auditory modality.

Haptic sensation is another less-studied modality involved in multimodal interruption recovery processes. Haptic feedback has been applied in various fields such as remote surgery (Prattichizzo et al., 2012), in-car messaging (Ardoin and Ferris, 2016), and virtual reality (Corbett et al., 2016). For example, Corbett et al. (2016) demonstrated that haptic feedback enhances users' performance in a virtual pointing task. Nam et al. (2008) further revealed that realistic haptic feedback regarding

**Abbreviations:** ANOVA, Analysis of variance; MFG, Memory-For-Goals; Hp, paired haptic stimuli; Hn non-paired haptic stimuli; Ip, interruption present; Ia, interruption absent.

the movement of the puck and stick improves performance during a virtual air hockey game. Furthermore, the presence of haptic feedback increases participant immersion in a virtual surgery environment (Meijden and Schijven, 2009). However, these studies do not explain the effects of redundant haptic information during multimodal task interruption and recovery. As haptic information is always perceived naturally during our daily cognitive tasks, it is important to understand the precise cognitive processes underlying the influence of this complex modality.

Studies of multimodal task interruption have utilized haptic sensation as an associative cue in order to enhance the activation of the primary task (Hopp et al., 2005; Prewett et al., 2012). When an associative cue and a primary task occur simultaneously, a link between the two is generated, allowing activation of the primary task following the presentation of the associative cue (Altmann and Trafton, 2002). Several studies have therefore applied associated cues in order to increase performance during multitasking (Altmann and Trafton, 2004; Hopp et al., 2005; Hodgetts and Jones, 2006; Smith et al., 2009). Furthermore, Prewett et al. (2012) demonstrated that using a vibrotactile cue (which is obviously non-visual) as an alert or message is more effective than using a visual cue when the primary task is visual. The multiple resource theory (Wickens, 2002) supports the effectiveness of vibrotactile cueing in this multitasking scenario. Wickens (2002) suggested that attentional resources from a separate resource pool distinguished by different sensory modalities can be successfully divided in parallel. Within the framework of the multiple resource theory, Hopp et al. (2005) also suggested that vibrotactile cues help alleviate the cost of interruption by reducing reaction time when the primary decision making task is visual.

However, previous studies utilizing simple vibration motors to implement vibrotactile cues have a clear limitation in that only directional or spatial cue information may be provided (Prewett et al., 2012). Thus, more general haptic sensations beyond vibrotactile cueing should be investigated, particularly in multimodal situations.

Though few in number, some studies have indeed utilized dual task situations that included haptic stimuli. Lu et al. (2013) performed a meta-analysis of studies regarding multimodal dual task situations in which a primary visual task was interrupted by secondary tasks of various modalities, including a haptic modality. As previously noted, interruption of a primary visual task with a secondary haptic task resulted in increased performance relative to interruption of a primary visual task with a secondary visual task. The multiple resource theory proposed by Wickens (2002) may account for such a result. However, in real-world situations, both the primary and secondary tasks rely on multiple modalities. Among various possible combinations of multiple modalities, we aimed to focus on the combination of visual and haptic modalities (visuo-tactile primary task + visuo-tactile secondary task).

In the present study, we implemented a combined visuotactile task in order to investigate the effect of redundant haptic stimuli during a task interruption situation. We first investigated the effect of redundant haptic information on the primary task. Our results align with those obtained by Lu et al. (2013), who studied the effect of redundant auditory information, which has been shown to increase accuracy as well as reaction time during task performance. We then studied the role of haptic stimuli in the interruption recovery process. Specifically, we analyzed how the combined information from paired visual and haptic modalities affects the recovery process relative to non-paired visual and haptic information. Based on the MFG theory, we speculate that priming from the redundant haptic stimulus may exert beneficial effects on the task recovery process. In this paper, we define "task recovery" as the retrieving process of a primary task information after the primary task is distracted by an interrupting task. As few studies have examined this topic, we addressed the following research questions:

# MATERIALS AND METHODS

Our experiment was characterized by a 2 (Interruption: present vs. absent) × 2 (Haptic Information: paired vs. non-paired) within-subject factorial design. Therefore, four experimental conditions were used: the two-back visuo-tactile task with paired haptic stimuli (Hp), with and without interruption, and the same two-back task with non-paired haptic stimuli (Hn), also with and without interruption. To create a visuo-tactile task environment that included haptic stimuli, a seven degree-offreedom haptic device was used in conjunction with a PC, monitor, and keyboard.

## Participant

Twenty-one students from Yonsei University (14 men; 7 women; age range: 20–26 years, mean age: 22.1 ± 2.36 years) participated in the present study. All participants were righthanded, with no visual or manual impairments, and remained naïve to the purpose of the experiments. No participant had any previous experience with relevant task interruption and recovery experiments. Participants conducted the procedure in a laboratory environment with one experimenter. All individual sessions lasted approximately 1 h. The present study was performed in accordance with the ethical standards laid down by the 1964 Declaration of Helsinki. All study participants provided informed written consent. Following the relevant Act and Enforcement Rules, which are specified below, from the Korean Ministry of Health and Welfare, our experimental procedure is exempt from local ethics committee approval. According to Article 15 (2) of the Bioethics and Safety Act and Article 13 of the Enforcement Rule of Bioethics and Safety Act, a research project "which utilizes a measurement equipment with simple physical contact that does not cause any physical change in the subject" (Korean to English translation by the authors) is exempt from

FIGURE 2 | Experimental setup with a haptic device and a PC.

such approval. Our entire experimental procedure was designed to use only a PC and a haptic device that does not cause any physical change in the participant.

# Materials

Two kinds of cognitive tasks were used in the present study: a two-back visuo-tactile task and a virtual needle penetration task (**Figure 1**). Each task was implemented as the primary task and the interrupting task. The primary task (two-back visuo-tactile task) is based upon the N-back design, which is widely utilized for the measurement of working memory in cognitive neuroscience (Kane et al., 2007). Furthermore, this cognitively demanding task has been used to assess the effects of task interruption (Monk et al., 2008; Borst et al., 2015). In order to provide participants with a visuo-tactile experimental environment including precise haptic feedback, our experimental system was composed of one haptic interface (Omega.7, a highly precise force-feedback haptic device with seven degrees-of-freedom produced by Force Dimension, Switzerland) and one PC (**Figure 2**). A 27-inch display monitor and a keyboard were set up in front of the participant. The haptic device was placed near the dominant hand; because all participants were right-handed, the haptic device was positioned on the right side of the monitor. The haptic interface can be connected and accessed through the Haptik Library (De Pascale and Prattichizzo, 2007). In addition, both haptic tasks were developed based on CHAI3D, an open-source set of C++ libraries for real-time haptic simulation, and driven with the Windows 8.1 operating system.

#### Two-Back Visuo-Tactile Task (Primary Task)

Participants were asked to perform the two-back visuo-tactile task as the primary task. A stream of visuo-tactile stimuli (cards with visual images and haptic stimuli) were presented sequentially, and participants were required to determine whether the information on the current item was the same as that occurring two items before. Hence, participants were required to keep this information in their working memory while recognizing new information.

In total, nine visuo-tactile stimuli (nine visual cards and nine tactile stimuli occurring in conjunction with one another) were used in the primary task. Since we presented visual and haptic stimuli to a participant at the same time, the information given to the participant was divided into two channels. **Figure 3** shows an overall schematic diagram of the two-back visuo-tactile task. First, visual information was provided via a 27-inch monitor as a series of rectangular cards, measuring 5 in × 5 in (**Figure 3**). All nine visual cards were distinguishable as nine different images with nine different colors. A haptic device provided haptic stimuli paired with the aforementioned visual cards. Nine haptic stimuli were generated using CHAI3D and the haptic simulation library as follows. Note that these stimuli are more general than the vibrotactile stimulus utilized in previous studies.

The different visual cards and haptic stimuli were paired with one another and simultaneously presented to the participant in the Hp session. Each paired item was presented for 2400 ms, followed by a mask of 240 ms. Participants responded to each item by pressing the corresponding key on the keyboard for each answer: "1" or "y" for "Yes" (i.e., the current item is the same as the item that occurred two items before) and "2" or "n" for "No." For participants who were uncomfortable with pressing two distant keys ("y" and "n"), two nearby keys ("1" and "2") were also offered as an alternative option. Each participant chose and

used one option (i.e., "y"/"n" or "1"/"2") depending on his or her preference throughout the whole experiment. If a participant did not respond within the given time (2400 ms), the response was recorded as a failure (i.e., wrong answer).

Two experimental sessions were implemented in order to investigate the effects of paired haptic information: a paired haptic stimulus (Hp) session and a non-paired haptic stimulus (Hn) session. In the Hn session, participants were presented with a non-paired haptic stimulus and a visual card. The non-paired haptic stimulus was arbitrarily chosen for each visual card. Both sessions of the primary task were interrupted every five to eight items (randomly assigned). Each session consisted of five sets, and one set consisted of 60 items.

#### Virtual Needle Penetration Task (Interrupting Task)

As an interrupting task, a virtual needle penetration task, which demands cognitive resources from both visual and haptic senses, was implemented. This interrupting task was adapted from a needle insertion simulation toward haptic-rendered soft tissue originally designed by Gerovich et al. (2004) and Prattichizzo et al. (2012).

A participant was instructed to move a virtual needle on the screen using the haptic device, find an invisible vessel inside the visible virtual skin, and place the tip of the needle inside the vessel (**Figure 4**). Throughout the interrupting task, the participant used only the right hand to manipulate the haptic device to control the virtual needle and also receive force feedback from the haptic device. The force feedback closely simulates the haptic sensation corresponding to the actual act of touching. The location of the invisible vessel was randomly assigned at each trial; however, the intensity of the force feedback at the moment penetrating the vessel was identical. Since the participant had become familiar with the force feedback intensity upon vessel penetration in the training period conducted prior to the actual

experiment, the participant with proper concentration could successfully locate the needle inside the vessel. Given a 7200-ms time limit for the interrupting task, a participant was required to use his or her visual sense to penetrate the visible skin and haptic sense to locate the needle inside the vessel. The vessel had a certain thickness, zv, and the needle would pass through the other side of the vessel if the participant applied excessive force. When the participant held the tip of the needle inside the vessel for more than 1000 ms without passing through the vessel, the interrupting task successfully terminated.

Adapted from the simulation designed by Gerovich et al. (2004) and Prattichizzo et al. (2012), the following haptic renderings are implemented in this task. Three kinds of soft tissue were generated as virtual renderings of the skin, inward vessel wall, and outward vessel wall. Each layer was assigned distinct spring and damping coefficients. Therefore, participants could be provided haptic feedback such as spring stiffness during contact with the tissue as well as damping effect when the needle passed through any kind of tissue. A detailed haptic model was implemented as follows (based on the haptic model from Gerovich et al. (2004) and Prattichizzo et al. (2012), but simplified by removing some layers and viscous effects).

When the needle contacts and punctures the outermost layer (i.e., virtual skin):

$$F = k\_{\sharp}z, \quad \qquad 0 < z < z\_{s.th}$$

$$F = \, ^b \imath z \nu, \qquad \qquad z\_{s.th} < z < z\_{\sharp} \tag{1}$$

When the needle interacts with the vessel wall, inward-bound:

$$\begin{aligned} F &= k\_{\text{vi}} (z - z\_s) + b\_s z\_s \nu, & \quad z\_s < z < z\_s + z\_{\text{vi}.th} \\ F &= \begin{pmatrix} b\_{\text{vi}} z + b\_s z\_s \end{pmatrix} \nu, & \quad z\_s + z\_{\text{vi}.th} < z < z\_s + z\_{\text{v}} \end{aligned} \tag{2}$$

When the needle interacts with the vessel wall, outward-bound:

$$F = k\_{\rm vo} \left( z - z\_{\rm v} - z\_{\rm s} \right) + \left( b\_{\rm vi} z\_{\rm v} + b\_{\rm s} z\_{\rm s} \right) \nu, \quad z\_{\rm s} + z\_{\rm v} < z < z\_{\rm s} + z\_{\rm v}$$

$$+ z\_{\rm vo.th}$$

$$F = \left( b\_{\rm vo} z + b\_{\rm vi} z\_{\rm v} + b\_{\rm s} z\_{\rm s} \right) \nu, \quad \qquad \qquad z\_{\rm s} + z\_{\rm v} + z\_{\rm vo.th} < z \text{(3)}$$

where ks , kvi, and kvo represent the spring coefficients of corresponding tissues; bs , bvi, and bvo represent the damping coefficients of corresponding layers (per unit thickness); zs and zv represent the thickness of the outer skin and vessel layers, respectively; and zs.th, zvi.th, and zvo.th represent the thresholds for penetration shown in **Figure 4**.

In the interrupting task, the haptic modality was mainly used to determine the vessel's location. Meanwhile, the visual modality was used to monitor the movement of the virtual needle and confirm whether the skin had been penetrated. Unlike in the primary two-back task, the interrupting task was identical in the Hp and Hn sessions.

### Procedure

Each individual 1-h session was conducted in a laboratory environment. All participants were given a tutorial regarding the experimental procedures, including a clear explanation of the tasks. Prior to the actual experiment, participants engaged in a training session in order to familiarize them with the haptic interface and experimental tasks. The primary two-back task during the training period consisted of 40 Hp (paired haptic stimulus) items and 40 Hn (non-paired haptic stimulus) items. The goal of the training period was to ensure that participants had become accustomed to the nine visuo-tactile stimulus pairs of the primary two-back task and the force feedback intensity upon vessel penetration during the interrupting task.

After the training period, the actual experiment was conducted. Participants performed five sets of 60 items in each Hp and Hn session. Therefore, the entire experiment consisted of 10 sets per person. After every two sets, a participant was given a 3-min break. In order to reduce the potential effect due the order of the tasks, participants were equally divided into two groups; one performed the Hp session prior to the Hn session, while the other conducted the Hn session prior to the Hp session.

# Measures

To measure the effects of task interruption in a combined visuo-tactile task environment, we examined reaction time and accuracy as dependent variables. As previously mentioned, increased reaction time and decreased accuracy for the primary task have been highlighted as the major cost of task interruption (Hodgetts and Jones, 2006; Monk et al., 2008; Trafton et al., 2011; Brumby et al., 2013). For every primary two-back task item, the time interval between the moment when a participant received the visuo-tactile stimulus and the moment when the participant pressed the response key was recorded as the reaction time. Each Hp or Hn session consisted of five sets of 60 primary two-back task items, and the average reaction time of each participant for each session was measured. In addition, accuracy was also measured by recording the proportion of correct responses, and the average accuracy of each participant was also recorded for statistical analysis.

Reaction time and accuracy were measured under four conditions: paired haptic stimulus with interruption present (Hp + Ip), paired haptic stimulus with interruption absent (Hp + Ia), non-paired haptic stimulus with interruption present (Hn + Ip), and non-paired haptic stimulus with interruption absent (Hn + Ia). **Figure 5** depicts an example sequence of the task items and an interruption. Interruptions occurred at arbitrary points in the sequence. The next two items after an interruption were classified as interrupted items, while other items were classified as non-interrupted items. The performance of interrupted items was recorded as the condition with interruption present (Ip). On the other hand, the performance of non-interrupted items was recorded as the condition with interruption absent (Ia). Since initial responses can be extreme outliers (Borst et al., 2015) we also excluded the initial responses until the first interrupted items.

# RESULTS

The cost of interruption recovery can be measured in two ways: reaction time and accuracy. As per the MFG theory, performance of the interrupted task would be degraded in terms of both reaction time and accuracy relative to the non-interrupted task (Altmann and Trafton, 2002; Monk et al., 2008; Altmann et al., 2014). In the present study, we investigated the effects of paired haptic stimulus presentation during interruption recovery. Thus, we first examined whether the haptic stimulus affected the performance of the primary task differently depending on the presence of interruption. Significant interactions between haptic stimulus presentation and the presence of an interruption with regard to accuracy or reaction time indicate that the haptic

Moon et al. Haptic Information in Task Interruption

stimulus influences the interruption recovery process. As we observed these effects in our analysis, we further analyzed the effects of the haptic stimulus on accuracy. However, we observed no significant interaction between presentation of the haptic stimulus and the presence of interruption with regard to reaction time.

# Interactions between Haptic Stimulus Presentation and the Presence of Interruption on Reaction Time and Accuracy

The interaction between presentation of a haptic stimulus and the presence of interruption can be simply analyzed by examining the haptic benefit depending on the presence of interruption. In the present study, the haptic benefit was defined as an improvement in reaction time or accuracy due to the presence of the paired haptic stimulus (i.e., reaction time or accuracy under Hp condition minus reaction time or accuracy under Hn condition). Our analysis based on the subtracted data under two conditions is similar to the approach of Olesen et al. (2004).

We used a paired samples t-test to analyze the haptic benefit depending on the presence of interruption. The haptic benefit in terms of reaction time under the interrupted condition was similar to the haptic benefit under the non-interrupted condition (t = 1.526, p = 0.143, Cohen's d = 0.333, according to the 5 percent-standard level; non-interrupted task mean = 220.43, SE = 31.11; interrupted task mean = 133.38, SE = 51.69). Thus, we observed no significant interaction between presentation of the haptic stimulus and the presence of interruption with regard to reaction time. In contrast, the haptic benefit in terms of accuracy under the interrupted condition was significantly better than the haptic benefit under the non-interrupted condition (t = −5.640, p < 0.01, Cohen's d = 1.231, according to the 5-percent-standard level; non-interrupted task mean = 0.70, SE = 0.68; interrupted task mean = 8.03, SE = 1.32). Thus, we observed a significant interaction between the presentation of the haptic stimulus and the presence of interruption with regard to accuracy. The same result was obtained by the two-way repeated measures analysis of variance (ANOVA) described in the next section.

# Effect of Haptic Stimulus on Accuracy during Interruption Recovery Process

We measured the accuracy of participants in the primary twoback task under the aforementioned four conditions (i.e., Hp + Ip, Hp + Ia, Hn + Ip, Hn + Ia). Mean accuracy values for the 21 included participants are presented in **Table 1**. As we predicted based on the MFG theory, participants made more errors in the interrupted condition than in the non-interrupted condition (90.18 vs. 71.23%, on average). Two-way repeated measures ANOVA were used to discover significant differences among each condition. The presence of paired haptic stimuli and the presence of interruption were used as within-subject factors in the two-way repeated measures ANOVA.

**Figure 6** depicts participant accuracy under the four conditions of the present study. When participants were interrupted by the interrupting task, their accuracy on the

#### TABLE 1 | Accuracy in the primary two-back task.

primary two-back task decreased significantly [F(1, 20) = 237.100, MSE = 31.801, p < 0.01, η 2 = 0.922]. In addition, participants achieved significantly better accuracy when paired haptic stimuli were provided than when non-paired haptic stimuli were provided [F(1, 20) = 28.122, MSE = 14.254, p < 0.01, η 2 = 0.584]. Furthermore, we observed a significant interaction between the presentation of a haptic stimulus and the presence of interruption with regard to accuracy, in accordance with our t-test results discussed in the previous section [F(1, 20) = 31.815, MSE = 8.870, p < 0.01, η 2 = 0.614]. That is, the paired haptic stimulus further enhances accuracy when the task is interrupted relative to when the task is not interrupted.

Our data analysis confirms the following results regarding the effect of paired haptic stimuli on task performance during visuotactile task interruption and the subsequent recovery process: (1) The presence of a paired haptic stimulus improves accuracy on the primary two-back task; (2) the paired haptic stimulus affects the accuracy differently depending on the presence of interruption; the paired haptic stimulus further enhances the accuracy when the interruption is present.

### DISCUSSION

Multiple sensory channels have been studied to understand the cognitive processes involved in multimodal task interruption and recovery. Previous studies have focused on visual and auditory modalities in particular, while studies involving haptic stimuli have been limited to an investigation of their role as associative cues in enhancing performance during the primary task. In the present study, we conducted a visuo-tactile task interruption and recovery experiment in order to examine the role of redundant haptic information in such processes. Our results confirm that the use of redundant haptic information enhances participant accuracy on the primary visuo-tactile task. Noticeably, the redundant haptic information is especially helpful for participants to recover the interrupted task. This increase in accuracy is significantly greater during interruption recovery than during a non-interrupted task.

• Q1: Does the cost of task interruption fit the MFG theory in a visuo-tactile task environment?

Our results align with the MFG theory, revealing an increased cost of task interruption with regard to accuracy. Regardless of the haptic stimulus, participants demonstrated decreased accuracy when they were interrupted. Hence, our data support the results of previous studies regarding interruption in a multimodal task environment (Hodgetts et al., 2014; Keus van de Poll and Sörqvist, 2016).

• Q2: How does the presence of redundant haptic information affect performance in a visuo-tactile cognitive task?

Our results indicate that redundant haptic information provided in the form of a paired haptic stimulus induces improvements in accuracy regardless of the presence of interruption. As mentioned earlier, Lu et al. (2013) investigated the use of redundant auditory information, which resulted in improved accuracy on a single visuo-auditory task without interruption. Our results indicated that similar enhancements are observed when redundant haptic information is provided during a visuo-tactile task.

• Q3: Is there any benefit of using redundant haptic information, especially in a visuo-tactile task interruption and recovery process?

The presence of the paired haptic stimulus resulted in significantly greater improvements in accuracy in the interrupted condition than in the non-interrupted condition, a comparison not studied in Lu et al. (2013). Such a result demonstrates that redundant haptic information exerts a specific influence on the interruption recovery process.

#### REFERENCES

Similar to performance enhancements observed when associative cues are presented during the recovery process, increases in accuracy due to the presentation of redundant haptic information may be explained as a result of enhanced activation of the primary task, according to the MFG theory (Altmann and Trafton, 2002). An associative cue boosts the activation of the primary task (i.e., priming occurs) due to generation of a link between the cue and the primary task. Likewise, redundant haptic information can boost the activation of the primary task during the recovery process because a similar link between the haptic information and the primary task is generated.

The results of the present study suggest that haptic information may be effective for interruption management. In their meta-analysis, Lu et al. (2013) suggested various ways of using multimodal information for designing efficient multimodal interfaces. For example, haptic modalities can be effectively utilized to deliver low-complexity information such as simple notifications, while auditory modalities can be used to deliver high-complexity information such as informative alerts. However, such suggestions are based on limited experiments that have utilized a vibrotactile motor unable of delivering a complex haptic stimulus. However, the highly precise force feedback haptic device used in the present study is capable of generating highly complex haptic stimuli that can vary in terms of viscosity, stiffness, vibration, magnetic force, various textures, etc. The significant improvements in accuracy observed during the present study demonstrate the ability of haptic stimuli to provide such highly complex information for the management of interruptions. Our results may provide a foundation for elucidating the mechanisms underlying the recovery process in a multimodal sensory environment.

#### AUTHOR CONTRIBUTIONS

HM and JS designed the study. HM developed the experimental software and performed the experiment. HM, JB, and JS analyzed the data and discussed the results. HM drafted the manuscript, and JB and JS revised the manuscript. All authors approved the final manuscript.

## FUNDING

This research was supported by the Ministry of Science, ICT, and Future Planning (MSIP), Korea, under the "IT Consilience Creative Program" (IITP-R0346-16-1008) and supervised by the Institute for Information & Communications Technology Promotion (IITP).

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Moon, Baek and Seo. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Phone Conversation while Processing Information: Chronometric Analysis of Load Effects in Everyday-media Multitasking

#### Michael B. Steinborn\* and Lynn Huestegge

Psychologie III, Universität Würzburg, Würzburg, Germany

This is a pilot study that examined the effect of cell-phone conversation on cognition using a continuous multitasking paradigm. Current theorizing argues that phone conversation affects behavior (e.g., driving) by interfering at a level of cognitive processes (not peripheral activity) and by implying an attentional-failure account. Within the framework of an intermittent spare–utilized capacity threading model, we examined the effect of aspects of (secondary-task) phone conversation on (primary-task) continuous arithmetic performance, asking whether phone use makes components of automatic and controlled information-processing (i.e., easy vs. hard mental arithmetic) run more slowly, or alternatively, makes processing run less reliably albeit with the same processing speed. The results can be summarized as follows: While neither expecting a text message nor expecting an impending phone call had any detrimental effects on performance, active phone conversation was clearly detrimental to primary-task performance. Crucially, the decrement imposed by secondary-task (conversation) was not due to a constant slowdown but is better be characterized by an occasional breakdown of information processing, which differentially affected automatic and controlled components of primary-task processing. In conclusion, these findings support the notion that phone conversation makes individuals not constantly slower but more vulnerable to commit attention failure, and in this way, hampers stability of (primary-task) information processing.

#### Keywords: vigilance, sustained attention, cell phone conversation, variability, effort

# INTRODUCTION

Everyday experience tells us that people have profound multitasking abilities since multitasking activities are extremely common in people's everyday-life routines (cf. Bills, 1943, pp. 151–185; Salvucci and Taatgen, 2011, pp. 3–24). For example, researchers are often talking of running multiple projects concurrently, or are concurrently consuming multiple media sources at work and leisure. Sufficient practice provided, people might even be able to acquire superior everydaylife multitasking abilities (Ophir et al., 2009; Schubert et al., 2015), which is particularly true within the area of multimedia applications and gaming (Strobach et al., 2012). Not surprisingly, since the majority of actions and decisions is governed by routinized programs (Norman and Shallice, 1986; Langner and Eickhoff, 2013), the commonsense view of multitasking would lead one to expect that

#### Edited by:

Mike Wendt, Medical School Hamburg, Germany

#### Reviewed by:

Rico Fischer, University of Greifswald, Germany Alexander Soutschek, University of Zurich, Switzerland

\*Correspondence: Michael B. Steinborn<EMAIL_ADDRESS><EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Cognition, a section of the journal Frontiers in Psychology

Received: 31 January 2017 Accepted: 15 May 2017 Published: 06 June 2017

#### Citation:

Steinborn MB and Huestegge L (2017) Phone Conversation while Processing Information: Chronometric Analysis of Load Effects in Everyday-media Multitasking. Front. Psychol. 8:896. doi: 10.3389/fpsyg.2017.00896

seemingly skilled behavior can concurrently be performed with ease, without any considerable impairment in performance (Finley et al., 2014). Thus, multitasking implies an advantage in time saving in the majority of standard situations, which often leads people to neglect the fact that it might also entail a disadvantage in unexpected situations, where rapid adaptations to changes are required (Hockey, 1997; Wickens, 2008; Parmentier, 2014). Taking up these issues, we focused on multitasking-induced performance (un)reliability, examining effects of phone usage on automatic and controlled components of information processing.

# Empirical Findings: Loading And Distraction Effects

Current empirical findings on continuous multitasking effects in the applied domain are largely dominated by two lines of research, by experiments on the effect of multitasking on learning and studying (Rohrer and Pashler, 2003; Pashler et al., 2013), and experiments on the effect of cell phone usage during driving (Alm and Nilsson, 1994; Horrey and Wickens, 2006). The experimental design in such applied studies is usually unconstrained, which brings about the benefits of retaining ecological validity (in some cases, at the cost of experimental control). Although there are (on principle) a variety of design options, the most typical experimental set-up found in the empirical literature usually consists of the following essential elements, a primary task which is usually performed in streams of continuous action, and a secondary task which is conceptualized either as a discrete event (e.g., an infrequent probe task) or a distractor such as a phone call (Sanders, 1998, pp. 271–285). The main line of empirical evidence stems from continuous tracking (or serial responding by key pressing) as the primary task and discrete manual or vocal responses to probe stimuli as the secondary task (Pashler, 1998, pp. 298–317). It is usually asked whether the loading/distractor affects primary-task performance, and the research question is mostly of practical relevance (Strayer and Johnston, 2001; Hancock et al., 2003; Strayer and Drews, 2007).

Empirically, there are three main determinants that affect primary task performance in natural contexts, in particular, the temporal predictability and task predictability of the secondary task, and controllability of the entire task ensemble (Sanders, 1998, pp. 330–359). These factors are ubiquitous and occasionally recognized as such (Kalsbeek and Sykes, 1967; Salvucci and Taatgen, 2008; Steinborn and Langner, 2011, 2012; Reissland and Manzey, 2016), albeit not strictly accounted for by the prevalent multiple-resource theory (Wickens, 1980, 1984). Specifically, when probe stimuli (as secondary task) occur at a constant rate within blocks of trials during the primary task, participants know exactly about when they will occur, and thus, are more likely to engage in appropriate processing strategies (Steinborn et al., 2016b). This again goes better when the nature of the secondary task is also constant, since task operations can better be prepared when these features are predictable as compared to when they are not (Kalsbeek and Sykes, 1967; Thomaschke and Dreisbach, 2015). Finally, multitasking depends greatly on whether individuals are enabled to do it their own way, that is, when they decided their own scheduling, than when they were to follow fixed schedules. For example, experiments reported by Hockey and Earle (2006) demonstrate that control over the regulation of multitask office work has an eminent impact on the way in which fatigue develops in response to demanding work goals (cf. Sanders, 1998, pp. 394–441).

Much of research in the applied domain is devoted to the effects of cell phone use during driving (e.g., Alm and Nilsson, 1994; Horrey and Wickens, 2006; Cooper and Strayer, 2008; Nijboer et al., 2016), and there is no doubt that this issue is of great practical importance and contributes much to a science-based approach to road safety policy (cf. Strayer and Drews, 2007). The essential finding can be summarized such that the use of mobile phones during driving leads to impairments at a purely cognitive (not peripheral) level thus increasing the risk of an accident. Strayer et al. (2003) examined the hypothesis of whether the observed impairment could be attributed to a disengagement of attention from the visual scene. Their results indicated that although an object is fixated, it is not being processed sufficiently (cf. Huestegge and Adam, 2011). In most countries, therefore, placing and receiving a phone call while driving are only allowed via hands-free systems. Yet, recent findings point to a reduction in attention directed toward the driving task even when using hands-free system car kits indicating that the source of interference produced by phone conversation originates not from manual operations related to phone use but from processing information related to conversation (Drews et al., 2008, 2009; Atchley et al., 2011a,b; Bergen et al., 2013).

Our theorizing given in the following section will finally converge toward an integrated spare–utilized capacity threading model as general framework (Kahneman, 1973). Two key aspects are of great importance. First, findings suggest that distraction effects by phone conversation are primarily caused through the cognitive effects of conversation and not (solely) by peripheral activities related to phone use. Second, the evidence delivers clues as to the possibility that phone-related distraction does not arise from a general slowing of relevant information-processing operations necessary for driving, but from an increase in the probability of attentional failure. For example, Casner and Schooler (2015, p. 38) examined vigilancelike phenomena in pilots performing routine tasks, concluding that people do not gradually become fatigued under vigilance conditions but occasionally jump into a rather discrete state of task-unrelated thoughts, or mind-wandering, respectively (Kurzban et al., 2013; Langner and Eickhoff, 2013; Smallwood, 2013; Steinborn et al., 2016b, for related accounts). Such a view of attention failure implies effects on performance variability which requires a theoretical model capable to explain the mechanism underlying performance fluctuations in active sustained-attention (i.e., mental-concentration) tasks (Pieters, 1983, 1985; Van Breukelen et al., 1995; Steinborn and Huestegge, 2016), and a spare–utilized capacity threading model offers a generic and integrated way of talking about performance variability.

# Theoretical Models: Spare–Utilized Capacity Threading

As already mentioned, we examined phone-related interference in active sustained-attention tasks to enable a chronometric approach to study applied-multitasking phenomena1 . Central to any theorizing on performance speed and variability in active tasks is a distinction between utilized and spare capacity as structural and a continual swing between these processes as dynamic component, referred to as capacity threading. According to Kahneman (1973, 2013), the most generic way to theorize on the energetic regulation of capacity during continuous mental work is to consider information processing as composed of two qualitatively distinct and constantly alternating classes of mechanisms which he termed operating and monitoring (Craik, 1948; Welford, 1959). Both processes serve different purposes (within the same goal area) and are complementary to each other with regard to energetic requirements. This means that it is subjectively more demanding to engage in mental operations than to not engage (i.e., than to keep spare capacity available). A measure of spare capacity is obtained by analyzing the response to an infrequent probe signal, presented to the individual at an unpredictable time during the primary task (cf. Kahneman et al., 1967; Posner and Boies, 1971; Shulman and Greenberg, 1971). By means of this method, it is possible to determine the amount of (utilized) capacity that is deployed to the task at the instant of probe-signal presentation, and a failure to identify or an unusually slow response to the probe-signal indicates that the individual is currently absorbed in the effective mental operations of the task at hand.

Kahneman (1973) argued that as individuals actually engage in the mental operations of the task at hand, spare (fluctuating) capacity is conveyed to utilized capacity and the corresponding increase in task focus would lead to a (temporary) decrease in monitoring. For example, Kahneman et al. (1967) demonstrated that when people engage in highly demanding mental operations (in the add-1 task) for a short period of time (i.e., when they perform a cognitive sprint), they are virtually blind during that period as revealed by measures of the probe-signal technique. In this way, he considered capacity allocation for an impending task as mobilization of mental energy (recruited from available spare capacity) to enable active mental operations. Mobilization is transient and time-sensitive, which means that it is virtually impossible to voluntarily sustain attention for more than a few seconds within one continuous stream of mental work. From this perspective, sustained attention is considered a mere re-implementation of successive efforts to redirect attention (to retransform spare to utilized capacity) to the task at hand. Thus, even when individuals have the intention to deliberately concentrate on the task for a while, capacity will never fully be utilized at any point during that period, but there is always spare capacity left for monitoring, evaluating, and adjusting pre-set performance standards (cf. Steinborn et al., 2016b).

Such a perspective of spare–utilized capacity threading offers a very natural way to explain variability in active mental tasks where response time (RT) is the primary performance measure. Instead of attributing experimental effects on RT variability to unspecified or umbrella-like terms often used in the literature (e.g., mental noise, ego depletion, lack of motivation, etc.), the model provides a generic and clearly defined mechanism based on spare–utilized capacity threading, according to which variations occur because the allocation policy sometimes channels capacity to other activities, resulting in slower responses during that period of trials (Steinborn et al., 2016b). This directly implies that the RT distribution of an individual is composed of a mixture between two operating mental states, an attentive state and a non-attentive state (cf. Luce, 1986, pp. 273–311; Ulrich and Miller, 1994, pp. 34–36; Van Breukelen et al., 1995, pp. 150–169). In the attentive state, the individual is effectively carrying out mental operations while in non-attentive periods, the individual is not effectively working because utilized capacity is conveyed to spare capacity. Note that this view has some decidedly important properties to explain performance fluctuations beyond mere scaling-variability (Wagenmakers and Brown, 2007), as indicated by relativized indices such as the RT coefficient of variation (RTCV), which is obtained by dividing the intraindividual RT standard deviation by the mean (cf. Steinborn et al., 2016b).

Although the advantage of RT variability and distributional analysis is widely recognized in the basic-research domains, researchers and practitioners in applied-research domains still rely on traditional measures of central tendency. A chronometric approach to studying performance speed and its fluctuation strictly implies a methodology beyond measures of central tendency, which can be studied in a comfortable way by analyzing the cumulative distributive function (CDF) of responses. The reason is that effects on RT mean are not interpretable by itself if they originate from a selective slowing at longer CDF percentiles (Miller, 2006). RT distributions are typically asymmetrical, having a steep slope on the left side (due to a rather narrow range of very fast responses) but an elongated right tail (arising from more broadly distributed slow responses)2 .

1 In response to a reviewer's comment, we would like to note that it is important to distinguish between active and more passive (i.e., vigilance-like) sustainedattention tasks when theorizing within a spare–utilized capacity threading model (cf. Langner and Eickhoff, 2013). To effectively engage in mental operations (e.g., such as counting, mental addition, or subtraction), the individual has to utilize capacity from available spare capacity, which is termed energetic mobilization (cf. Sanders, 1998, pp. 332–348). In contrast, in rather passive watchkeeping tasks (also referred to as vigilance tests, monitoring task, etc.), the individual's primary task is to wait and watch for relevant targets, and the factor demand is typically increased by lowering target occurrence probability and by increasing negative consequences of missing the target (cf. Broadbent, 1971, pp. 76–111). In this situation, there is no threading between operating and monitoring but a conflict between taskrelated target monitoring and task-unrelated (mind-wandering) tendencies, which are extremely difficult to resolve. The difficulty originates from a basic attentional principle, namely that attention primarily serves action and is to be attained and maintained through acting (cf. Neumann and Prinz, 1987). According to Kahneman (1973, pp. 13–27), it is virtually impossible to mobilize capacity in waitand-watch tasks because they do not require action for most of the time (see also, Casner and Schooler, 2015, for a similar view).

2Response to a reviewer's comment. As mentioned earlier, the responses of an individual in RT tasks are not symmetrically distributed around the mean but are typically skewed such that they have an elongated tail toward the right. This distributional asymmetry is due to the fact that there is fundamental limit to maximizing response speed but none to response slowing. For example, the classic work of Bills (1931, 1935) devoted particular attention to the occurrence of incidental extra-long responses (which he termed "mental blocks") after periods

Thus, RT variability expresses itself chiefly in responses above RT mean, and many variables affect RT mean only indirectly by selectively affecting stability (cf. Steinborn and Huestegge, 2016; Steinborn et al., 2016b). In the foreground of a research project within a spare–utilized capacity threading model thereby stand the goals of manipulating effort mobilization directly and measuring its effects with high precision by analyzing the entire RT distribution instead of only analyzing RT means (Steinborn et al., 2016a, 2017). This might provide a principal advancement to previous studies in this domain (cf. Sanders, 1998, pp. 394–451).

#### Present Study

Most work on driver distraction by cell phone conversation focused on the assessment of the impairment rather than on a delineation of the cognitive mechanisms underlying deficits in driving performance. Yet, studies that focused on this question imply an attention-failure account rather than a constant slowing of information-processing activity. In the present study, we aimed to precisely estimate potential impairments of cognitive performance by everyday-life cell phone usage, particularly by talking and texting. Our study can be characterized by two key aspects: First, we used an unconstrained continuous-multitasking paradigm. This is commonly accepted in applied-research domains, however, our approach differs in some way to previous studies since we used self-paced mental arithmetic as primary task, examining performance alone and in combination with unconstrained cell-phone conversation as secondary-task. We decided to use continuous arithmetic in order to enable the application of chronometric methods of RT measurement (Manzey and Lorenz, 1998; Haque and Washington, 2014). Further, we used a naturalistic conversation as secondary task, according to the methodical suggestions of Drews et al. (2008, pp. 393–395), to retain maximal ecological validity (Amado and Ulupinar, 2005; Horrey and Wickens, 2006). Second, in the foreground of our research thereby stands the use of advanced performancemeasurement methodology to critically capture aspects of performance reliability.

Notably though, the bulk of current research on cell-phone distraction neglected this important aspect of measurement. Whether the hypothesized effects on performance originate from a constant slowing of the speed of information processing, or alternatively, by an increase in the probability of attention failure is fundamental to the analysis and understanding of cell-phone distraction. In order to distinguish between both theoretical alternatives, we need to go beyond traditional measures of central tendency but instead must consider its effect at critical density zones of the entire RT distribution (Balota and Spieler, 1999; Spieler et al., 2000). We computed a CDF for each experimental condition, asking whether phonerelated impairments during continuous cognitive processing makes information processing run more slowly, or alternatively, makes processing run less reliably albeit with the same processing speed. Notably, this distinction is critically implied by current theorizing, albeit not explicitly measured in driving tasks (Groeger, 1999). Consequently, we examined whether experimental effects on RT mean originate from a global slowdown that is equally present at all CDF percentiles (parallel effect) or only from a local effect at slower percentiles (mixture effect). The former would indicate a true influence of continuous information-processing speed while the latter would indicate a destabilization of performance (Steinborn et al., 2016a, 2017).

Globally, we expected to observe an effect of cell-phone conversation on measures of RT and accuracy. That is, responses should be faster and somewhat less erroneous under the single-task condition as compared to a multi-task condition (main effect of context). We further expected faster responses for easy mental arithmetic as compared to hard mental arithmetic (main effect of demand). Whether cell-phone conversation differentially affects easy versus hard mental arithmetic performance is an empirical question, since previous research on continuous multitasking does not deliver enough reliable information on the impact of conversation on automatic versus controlled information processing components (cf. Ashcraft and Battaglia, 1978; Logan, 1979; Borst et al., 2013). To examine behavioral variability, we analyzed both the classic parameters of RT variability and parameters of distributional skewness based on the ex-Gaussian model (cf. Heathcote et al., 1991; Leth-Steensen et al., 2000). Remind that from the perspective of an energetic spare–utilized capacity threading model, it is crucial to know whether cell-phone conversation during cognitive processing leads to a generic (vs. selective) slow-down of all (vs. only long) CDF percentiles. Theorizing within an energetic-capacity framework, conversation is expected to hamper information-processing by increasing the probability of attentional failure. We examined both the effect of expecting and performing phone talking (Experiment 1) and text communication (Experiment 2) on cognitive performance, using the same sample of participants (within the sequence of both experiments counterbalanced across participants).

# MATERIALS AND METHODS

#### Participants

A student-based sample of 39 (29 female, 10 male) volunteers (mean age = 23.5 years, SD = 6.5) took part in the experiment. All

of normal work speed in self-paced color naming. Crucially, it is not a matter of the scale properties (i.e., being finite toward the left but infinite toward the right), as occasionally stated in the literature, but because of a limitation in the speed of processing even when performed with maximum mental efficiency (Steinborn et al., 2016b). To illustrate this, consider a formula-one driver on a particular training day where everything clicks into place (e.g., driver is fully concentrated, check processes occur at exactly the critical moments, team coordination is effective, etc.). On this day, the hypothetical lap times of the driver will be almost always near to ideal line (e.g., 73, 71, 71, 74, 76, 73, 74 s, etc.). Now consider a day where everything is not going as well as it should (e.g., driver not concentrated, team coordination ineffective, etc.). On this day, the driver may likely succeed in some (even in the majority of the) laps but may fail in other ones due to the particular circumstances on this particular day (e.g., 72, 71, 85, 74, 75, 93, 73 s, etc.). Critically, inspecting only measures of central tendency would lead to the conclusion that the driver was simply slower on the bad (vs. the good) day, which is convenient albeit incorrect (or at least incomplete) given that the overall slowing originated from an unfortunate combination of circumstances in some but not all of the rounds yielding extraordinarily slow lap times.

participants were in standard condition (reported to be healthy) and had normal or corrected-to-normal vision.

# Apparatus and Stimuli

The experiment was programmed using PsychoPy (Peirce, 2009). Participants sat about 60 cm in front of the screen. To mimic the characteristic (i.e., self-regulated) features of active continuous information-processing, we used mental arithmetic as one of the primary cultural techniques (Thorndike, 1922; Bills, 1943), practiced among identifiable cultural groups, and amenable to advanced psychometric analysis. In particular, we used a version of the mental-addition and verification task that contained both easy and hard items, using a short response–stimulus interval of 50 ms, which is particularly suitable to examine performance fluctuations (Sanders and Hoogenboom, 1970; Soetens et al., 1985; Steinborn et al., 2010, 2012). In each trial, an addition term together with the result is presented and participants indicated whether the result is either correct or incorrect. They were instructed to verify a correct result by pressing the right key (right index finger) and to falsify an incorrect result by pressing the left key (left index finger). The task contained easy and difficult items differing with respect to the chain length. Items categorized as easy included simple additions (e.g., 4 + 5 = 9; 4 + 5 = 8) while items categorized as difficult included chained additions (e.g., 4 + 5 + 1 + 2 = 12; 4 + 5 + 1 + 2 = 11). There were 24 easy items and 24 hard items. Each item was presented randomly and equally often (total of 865 trials).

# Automatic and Controlled Processing Components

In the present study, we used easy (chain length = 1) and hard (chain length = 4–5) mental-addition items as a proxy for automatic versus controlled processing components in mental arithmetic, which is well-agreed and theoretically backed-up by exemplar-based theories of cognition, learning, and automaticity. This consequently leads to a distinction between two general modes of solving mental-addition problems, a calculation-based mode and one that is based on memory retrieval. In the human-factors domain, this is often referred to as workload (albeit in a more intuitive way) and in most cases, not further specified. For example, Logan (1988) considered performance as automatic when it is based on single-step, direct-access retrieval of solutions from memory, while he considered performance as controlled when it is based on algorithmic processing mechanisms such as counting, addition, memorizing, or borrowing (Groen and Parkman, 1972; Ashcraft, 1992; Imbo et al., 2007). It should be clear that the use of this terminology only makes sense when the context in which the terminology is employed, is also specified (Logan, 1988, pp. 493– 495). Crucial is the assumption that every encounter of a stimulus (e.g., 4 + 5 = 9) results in episodic recording and retrieval, given the individual is sufficiently attentive and responsive. More formally, this leads to a set of fundamental assumptions: Attending deliberately to an event such as a single mental-arithmetic problem furnishes obligatory encoding and obligatory retrieval of separate instances in memory. Stimulus processing is characterized in terms of a race between algorithmic processing and memory retrieval such that whichever finishes first in a particular trial controls the response. In other words, any mental-arithmetic problem in a particular trial is finally solved either by the former or the latter process.

# Procedure