# EFFECTS OF GAME AND GAME-LIKE TRAINING ON NEUROCOGNITIVE PLASTICITY

EDITED BY: Guido P. H. Band, Chandramallika Basak, Heleen A. Slagter and Michelle W. Voss PUBLISHED IN: Frontiers in Human Neuroscience

#### *Frontiers Copyright Statement*

*© Copyright 2007-2016 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-840-5 DOI 10.3389/978-2-88919-840-5

## About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

## Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

## Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

## What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office<EMAIL_ADDRESS>
## **EFFECTS OF GAME AND GAME-LIKE TRAINING ON NEUROCOGNITIVE PLASTICITY**

Topic Editors:

**Guido P. H. Band,** Leiden University, Netherlands **Chandramallika Basak,** University of Texas at Dallas, USA **Heleen A. Slagter,** University of Amsterdam, Netherlands **Michelle W. Voss,** University of Iowa, USA

Cognitive training is not always effective. This is also the case for the form of cognitive training that this Research Topic focuses on: prolonged performance on game-like cognitive tasks. The ultimate goal of this cognitive training is to improve ecologically-valid target functions. For example, cognitive training should help children with ADHD to stay focused at school, or help older adults to manage the complexity of daily life. However, so far this goal has proven too ambitious. Transfer from trained to non-trained tasks is not even guaranteed in a laboratory, so there is a strong need for understanding how, when and for how long cognitive training has effect. Which cognitive functions are amenable to game training, for whom, and how? Are there mediating factors for success, such as motivation, attention, or age? Are the improvements real, or can they be attributed to nonspecific factors, such as outcome expectancy or demand characteristics? Are there better strategies to improve cognitive functions through game training?

This Research Topic of Frontiers in Human Neuroscience charts current insights in the determinants of success of game training.

**Citation:** Band, G. P. H., Basak, C., Slagter, H. A., Voss, M. W., eds. (2016). Effects of Game and Gamelike Training on Neurocognitive Plasticity. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-840-5

# Table of Contents

Aki Nikolaidis, Michelle W. Voss, Hyunkyu Lee, Loan T. K. Vo and Arthur F. Kramer

## Editorial: Effects of Game and Game-Like Training on Neurocognitive Plasticity

Guido P. H. Band1 \*, Chandramallika Basak 2 , Heleen A. Slagter 3 and Michelle W. Voss 4

*1 Leiden Institute for Brain and Cognition, Leiden, Netherlands, 2 School of Behavioral and Brain Sciences, University of Texas at Dallas, Dallas, TX, USA, 3 Department of Psychology, University of Amsterdam, Amsterdam, Netherlands, 4 Department of Psychological and Brain Sciences, University of Iowa, Iowa City, IA, USA*

Keywords: videogame, learning, training effects, cognitive control, prefrontal cortex, neuromodulators, working memory, plasticity

**The Editorial on the Research Topic**

## **Effects of Game and Game-Like Training on Neurocognitive Plasticity**

Playing videogames is a widely practiced leisure activity. Because of the time spent playing and the interactive content, game-guided learning could potentially supplement or replace more traditional mediums for learning and rehabilitation. There are many shades of gray between pure cognitive training, where the participant pursues the goal of a cognitive benefit directly, and game-based training, where training is contained in activities with their own intrinsic motivational value. Finding the optimal approach requires a better understanding of the underlying mechanisms (cf. Karbach and Schubert, 2013). We therefore conducted a research topic entitled "Effects of game and game-like training on neurocognitive plasticity," composed of eight empirical papers, an opinion paper and a review that contribute to the quest for effective ways to employ game-guided learning.

Several studies have shown benefits from videogames for cognitive functions such as visual attention (Green and Bavelier, 2003, 2007), and multitasking (Basak et al., 2008; Anguera et al., 2013). However, transfer of cognitive training to criterion tasks is not always observed. For a reliable picture of game effects, methodological rigor is necessary; to recalibrate positive findings (Redick and Webster) and to give way to publication of reliable null-results (Boot et al., 2011). Optimizing methodology is an ongoing process, as Zelinski et al. demonstrated using structural equation modeling to validly evaluate transfer.

How do games make training more effective? Games can help overcome cognitive limitations by employing adaptive difficulty, informative feedback, and a sufficient dosage (Mishra and Gazzaley). But the positive role that is often attributed to increased motivation (Habgood and Ainsworth, 2011), prolonged immersion and fun inherent in games may be overestimated. Wang et al. tested the optimal way to schedule a working memory training intervention. They found that transfer from training to fluid intelligence scores occurred only after spreading training sessions over 20 days, and not after any of the denser schedules. Thus, long game sessions may not be the most efficient way to learn. Katz et al. performed a systematic study to identify motivational game features that could boost children's working memory. Surprisingly, game elements such as real-time score displays modulated learning only negatively. This illustrates that mere addition of game elements to disguise the burden of active training works counterproductive. Rather, game-guided training should comply with the same

#### Edited and reviewed by:

*Srikantan S. Nagarajan, University of California, San Francisco, USA*

#### Reviewed by:

*Ian Spicer Ramsay, San Francisco VA Medical Center, USA*

#### \*Correspondence:

*Guido P. H. Band<EMAIL_ADDRESS>
Received: *04 February 2016* Accepted: *07 March 2016* Published: *30 March 2016*

#### Citation:

*Band GPH, Basak C, Slagter HA and Voss MW (2016) Editorial: Effects of Game and Game-Like Training on Neurocognitive Plasticity. Front. Hum. Neurosci. 10:123. doi: 10.3389/fnhum.2016.00123* evidence-based principles as traditional forms of training (e.g., Schmidt and Bjork, 1992; Pashler et al., 2007).

What are the cognitive mechanisms that make games beneficial for training? Both game designers and trainers acknowledge the difficulty of merging gaming with training features. When there is too much focus on cognitive benefit, game elements may fail to raise motivation. Conversely, when game elements dominate, they exert a cognitive load at the expense of capacity available for target training (cf. Sweller et al., 1998), or at least draw attention away from the target training. It is fair to assume that the amount of learning is a function of the intensity and the focus of attention for relevant learning material. Indeed, Nikolaidis et al. demonstrated that individual differences in the working memory related brain response to game training are predictive of post-intervention changes on the Sternberg memory search task.

The influence of attentional focus and intensity in games can be translated to game effects on neuromodulation (e.g., Deveau et al.). Reward is a common game ingredient, associated with the release of dopamine (Koepp et al., 1998) and consequently, new habit formation. Gaming elements can help preserve reward responsiveness as measured in ventral striatum after training (Lorenz et al.). Furthermore, games often trigger arousal, characterized by phasic norepinephrine release, which facilitates attention and memory encoding (Tully and Bolshakov, 2010). Note, however, that trainees vary in neuromodulatory activity across age groups and between trainees with and without psychopathology (e.g., schizophrenia, ADHD). Thus, game mechanics should ideally adapt to individual differences in optimal levels of reward, reward responsiveness (cf. Gray, 1982) and arousal.

A different form of brain stimulation targets brain activity at EEG spectral bands associated with attention, working memory and control. Reedijk et al. used binaural beats to entrain target frequencies alpha (10 Hz) or gamma (40 Hz), which led to improved divergent thinking scores, at least for participants with lower dopamine levels. The same reasoning, that entrainment of the brain activity spectrum can improve cognitive performance, is followed in applying neurofeedback and transcranial AC stimulation. Mishra and Gazzaley argued that integrating such entrainment techniques into games, the closed-loop approach, may be able to boost game efficacy.

## REFERENCES

Finally, games can target qualitative changes in task performance. Bavelier et al. (2012) argued that transfer occurs if games help in "learning to learn," which involves better selection of relevant information and recognition of a problem structure. Stamenova et al. found data challenging this principle, however. They trained older adults to better distinguish targets from foils in a recollection training task. Although false alarm rates decreased, there was no transfer to non-trained memory tests, possibly because the stimulus set used for training was too limited. Generalization of learning increases as variability in input and training tasks increases (Schmidt and Bjork, 1992). High variability at the level of the learning material likely fosters learning at a more abstract level of representation (Green and Bavelier, 2008; Slagter, 2012). Another example of a qualitative cognitive change by game-guided training is the observation of Connors et al. who trained blind participants navigation through a building, using either audio- or game-guided training. Although standard performance improved equally for both groups, the gameguided training group could apply the same spatial knowledge faster and more flexibly, suggesting deeper or more implicit learning.

In sum, game elements have clear potential to improve training via multiple mechanisms. Perhaps the largest challenge in optimizing the efficacy of game-guided training is to gear interventions to individual differences. Variable brain anatomy, connectivity, baseline performance, age, neuromodulation, and strategy use can all modulate effects of game training parameters. As more is known about the neurocognitive properties of the trainee, games can be adapted accordingly. Ultimately, this approach would merge game design with insights from cognitive neuroscience and educational neuroscience.

## AUTHOR CONTRIBUTIONS

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

## FUNDING

The third author was supported by a VIDI award of the Netherlands Organisation for Scientific Research.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Band, Basak, Slagter and Voss. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## Videogame interventions and spatial ability interactions

## **Thomas S. Redick\* and Sean B. Webster**

Department of Psychological Sciences, Purdue University, West Lafayette, IN, USA

### **Edited by:**

Michelle W. Voss, University of Iowa, USA

#### **Reviewed by:**

Michael Dougherty, University of Maryland at College Park, USA Walter R. Boot, Florida State University, USA

#### **\*Correspondence:**

Thomas S. Redick, Department of Psychological Sciences, Purdue University, 703 Third Street, West Lafayette, IN 47907, USA e-mail<EMAIL_ADDRESS>
Numerous research studies have been conducted on the use of videogames as tools to improve one's cognitive abilities. While meta-analyses and qualitative reviews have provided evidence that some aspects of cognition such as spatial imagery are modified after exposure to videogames, other evidence has shown that matrix reasoning measures of fluid intelligence do not show evidence of transfer from videogame training. In the current work, we investigate the available evidence for transfer specifically to nonverbal intelligence and spatial ability measures, given recent research that these abilities may be most sensitive to training on cognitive and working memory tasks. Accordingly, we highlight a few studies that on the surface provide evidence for transfer to spatial abilities, but a closer look at the pattern of data does not reveal a clean interpretation of the results. We discuss the implications of these results in relation to research design and statistical analysis practices.

**Keywords: video games, training, transfer, intelligence, cognitive interventions**

## **VIDEOGAME INTERVENTIONS AND SPATIAL ABILITY INTERACTIONS**

In the past 10 years, there has been substantial interest in the idea that playing videogames may serve to improve certain cognitive functions. A recent meta-analysis (Powers et al., 2013) provided a quantitative summary of the numerous studies in which a videogame-playing group was compared against a control group that did not receive the videogame "treatment" of interest. As is typically done in meta-analyses, Powers et al. (2013) used broad operational definitions of cognitive outcomes, such as combining together various outcome measures of "executive functions", which included multitasking, inhibition, taskswitching, short-term/working memory, and intelligence. Overall, videogame training effects on executive functions was statistically significant (*d* = 0.16), although the effect would be classified as small according to Cohen (1992). Notably, transfer to intelligence was not significant, *d* = 0.06, and inhibition was the only executive function that was significantly improved by videogame training. On the other hand, tests of spatial imagery, such as mental rotation tasks, exhibited stronger meta-analytic effects, *d* = 0.43. Our current work investigates specifically the effect of videogame training on spatial ability transfer outcomes, given recent research that argues these spatial ability tests may be most sensitive to cognitive training (Colom et al., 2013).

Basak et al. (2008) provide an illustrative example of almost ideal intelligence transfer results, as a function of videogame training. In their study, older adults in the training group played a videogame (*Rise of Nations*) during 15 sessions. Raven Advanced Progressive Matrices, a matrix reasoning test commonly used to measure fluid or nonverbal intelligence, was among the battery of tests administered during pre-test and post-test transfer sessions. As seen in **Figure 1A**, the training group improved on Raven scores from pre- to post-test, whereas the control group that did not do anything between pre- and post-test (a *no-contact control group*) showed no improvement on Raven. In addition, the Raven mean pre-test scores were similar for the two groups. The interaction pattern in Basak et al. (2008) is straightforward to interpret, and the transfer data easily support the argument that the training "worked" for those subjects.

The pattern of transfer results observed in Basak et al. (2008) is what one would predict if the pre- to post-test change for the training group is what drives the observed significant interaction. However, videogame training studies such as Maillot et al. (2012), van Muijden et al. (2012), and Cherney (2008) are more difficult to interpret, because the pattern of transfer results is not similar to the Basak et al. (2008) example above. As outlined in the summaries below, each of these studies has been used to provide support for the efficacy of videogame interventions upon spatial abilities, yet a closer examination of the results warrants a more cautious interpretation. Note that our discussion in the current manuscript focuses on interactions and how strongly one can interpret the results as support for the efficacy of videogame interventions; recent articles have discussed many other methodological and measurement issues in videogame and brain training studies (e.g., Boot et al., 2011, 2013; Shipstead et al., 2012; Green et al., 2013).

## **VAN MUIJDEN ET AL. (2012)**

Two groups of participants completed the study: the video game group (*n* = 53) and the documentary group (*n* = 19). Participants in the videogame training group played five custom-built games (cf. **Figure 1**; van Muijden et al., 2012). Participants in the documentary control group watched documentaries of about 30 min in length followed by a three-to-five multiple-choice question quiz about the documentary. Participants in both the videogame and documentary control groups were instructed to

complete one 30-min session daily for 7 weeks, and participants that completed the study completed on average over 21 h of the prescribed activity. Before and after the videogame and documentary sessions, all participants completed a cognitive test battery that consisted of nine cognitive tests, including Raven standard progressive matrices. Raw scores from Raven were converted to IQ scores. van Muijden et al. (2012) concluded that "the results from the present study suggest that modest improvements of inductive reasoning can also be achieved by means of playing cognitive training games" (p. 10).

The group × session interaction on Raven was significant (η 2 *p* = 0.068), and is displayed in **Figure 1B**. However, this pattern is quite different from the Raven interaction pattern shown by Basak et al. (2008). The pre-test Raven IQ mean score for the videogame group was 116.4 and the post-test mean score was 119.4, an increase of 3.0 IQ points. The pre-test Raven IQ mean score for the documentary group was 120.1 and the post-test mean score was 116.8, a decrease of 3.3 IQ points. Although the between-groups comparison of the pre-test scores reported by van Muijden et al. (2012) was not significant ("*p* > 0.05", p. 4), the effect size for the pre-test scores indicated a small-tomedium difference (Cohen's *d* = 0.38). van Muijden et al. (2012) conducted subsequent simple main effect analyses to examine the pre- to post-test change within each group separately. Within the videogame group only, the pre- to post-test Raven increase was considered marginally significant (*p* = 0.05), whereas the preto post-test Raven decrease for the documentary group was not significant (*p* > 0.10). Critically, however, there was also a large difference in the group sample sizes (*n* = 53 and *n* = 19 for the videogame training group and the documentary control group, respectively). The sample size difference is extremely important as it pertains to the aforementioned follow-up tests—the reported effect size for the pre- to post-test Raven change score was actually larger for the nonsignificant control group (η 2 *p* = 0.128) than it was for the significant training group (η 2 *p* = 0.071). Thus, despite the significant interaction and associated effect size, we view the crossover pattern of Raven results as relatively weak evidence for the efficacy of videogame training to improve spatial abilities.

## **MAILLOT ET AL. (2012)**

Two groups of older adults (between the ages of 65 and 78 years old) completed the study: one group (*n* = 15) was assigned to the exergame training condition and the other (*n* = 15) was assigned to the no-training control group. The exergame training group completed two 1-h exergame sessions per week for 12 weeks for a total training time of 24 h. During these sessions participants played *Wii Sports*, *Wii Fit*, and *Mario and Sonic on Olympic Games*. In pre- and post-test sessions, all participants completed a battery of cognitive assessment tasks—we focus here on the matrix reasoning test (a subtest of the Wechsler Abbreviated Scale of Intelligence), a mental rotation task, and a directional headings test, all measures of spatial ability. Maillot et al. (2012) concluded "exergame training, which combines cognitive and physical demands in an intrinsically attractive activity, might be an effective way to promote physical and cognitive improvements among older adults" (p. 597).

In the published article, only the pre-test/post-test change scores were reported for each test, but the pre- and post-test values were provided upon request (P. Maillot, personal communication, 10/9/13), and are shown in **Figure 1C**. First, Maillot et al. (2012) reported that the mental rotation task change scores were not significantly different between the exergame and control groups (*p* = 0.24, η 2 *p* = 0.019), so this result is not discussed further. Maillot et al. (2012) reported that the matrix reasoning test change scores were significantly different between the exergame and control groups (*p* < 0.01) with a very large effect size of η 2 *p* = 0.531. For the directional headings test, Maillot et al. (2012) reported that the change scores were significantly different between the exergame and control groups (*p* = 0.02), again with a large effect size of η 2 *p* = 0.149. Critically, across all three dependent variables, the control group shows a numerical decrease from pretest to post-test on each test, and the control group pre-test scores were numerically larger than the training group. Given the small sample sizes, between-groups comparisons of the pre-test scores were not significant (all *p*'s > 0.13), yet the effect sizes for the pre-test scores indicated small-to-medium differences (Cohen's *d* = 0.14–0.59). Although the matrix reasoning and directional headings difference scores were significantly different between the training and control groups, examination of the pre- and posttest scores instead of only the change scores reveals an interaction pattern that complicates a strong interpretation of the exergame training efficacy. Combined with the absence of exergame effects on the mental rotation task and the use of a no-contact control group, we view the spatial ability transfer results here as modest at best, which contrasts with the large effect sizes reported in the article.

## **CHERNEY (2008)**

There has been substantial interest in the idea that videogames could reduce or eliminate gender effects in spatial ability, following the study by Feng et al. (2007). Cherney (2008) investigated gender differences in videogame effects upon spatial abilities, reported in a paper titled "Mom, let me play more computer games: They improve my mental rotation skills". Separate groups of male (*n* = 30) and female (*n* = 31) undergraduate students completed the study. Participants were randomly assigned to one of the three conditions: 3D videogame training (*n* = 20), 2D videogame training (*n* = 21), or control (*n* = 20) group, such that there were either 10 or 11 participants of each gender in each group (personal communication, I. Cherney, 11/25/2013). The 3D group played *Antz Extreme Racing*, the 2D group played *Tetrus*, and the remaining control participants played paper-andpencil logic games. Thirty-six participants completed the training during 2 weeks and 25 participants completed the training during 1 week. The participants all completed mental rotation and card rotation spatial ability tests in pre- and post-test sessions before and after the training period, respectively. Cherney (2008) concluded that "the results suggest that even a very brief practice (4 h) in computer game play does improve performance on mental rotation measures" (p. 783), and specifically, "practice with the Antz game, a 3-D computer game, seemed particularly beneficial for women's Vandenberg mental rotation test (VMRT) scores" (p. 785).

**Figure 2** displays the results for each transfer test as a function of group and gender (mean and standard deviations provided by I. Cherney, personal communication, 10/17/13). Collapsing across training and control groups, Cherney (2008) reported that there were significant improvements on the mental rotation test scores for women (*p* < 0.001) but not for men, and that card rotation test scores improved for both men and women (*p* < 0.001). Unfortunately, the necessary statistical comparisons testing whether the male and female training and control groups differentially improved from pre- to post-test on each dependent variable were not reported in Cherney (2008). As seen in **Figure 2**, the results are not clear. For the mental rotation test, men playing the 2D Tetrus game showed almost the same amount of decrease in scores from pre- to post-test (2.1 items) that the women increased in scores from pre- to post-test (2.6 items). In addition, for the mental rotation test, the effect of 3D and 2D training for the female groups was to bring their post-test scores up to the level of the pre-test mental rotation score for the female control group. For card rotation, Cherney reported significant pre- to post-test increases for each of the six groups tested, but no comparison of differences in the gain scores across groups. Given the small sample sizes, between-groups comparisons of the mental rotation and card rotation pre-test scores were not significant (all *p*'s > 0.07), yet the effect sizes for the pre-test scores indicated smallto-large differences (Cohen's *d* = 0.11–0.81). Again, we do not find the pattern of results presented in **Figure 2** as providing compelling evidence for the efficacy of videogame training to improve spatial abilities, for either men or women.

## **CONCLUSION**

The studies reviewed here do not represent all of the published evidence in support of the efficacy of videogame training to improve spatial abilities (see Powers et al., 2013, for review). However, the highlighted studies do illustrate the numerous challenges both scientists and laypersons face when trying to interpret the available research. In the three studies reviewed here, the sample sizes are smaller than the recommended minimum number of observations per cell for at least one of the comparison groups (Simmons et al., 2011), and the use of small samples leads to many problems. First, and most obvious, the use of small samples leaves the study underpowered to quantify the true effect of videogame training, and even more problematically leads to an increased likelihood of producing an inflated effect size (Button et al., 2013). Although meta-analyses are more informative than any individual study in quantifying the exact magnitude of the effect of videogame training, the meta-analysis is not a panacea if studies with small sample sizes produce large effect sizes that are then included in the meta-analysis. For example, Powers et al. (2013) provided Cohen's *d* estimates of the effect sizes obtained in Maillot et al. (2012): *d* = 2.056 and *d* = 0.810 for matrix reasoning and directional headings, respectively. As reviewed above, the pattern of results for both the training and control groups were not straightforward, and inclusion of these large effect sizes in the meta-analysis will influence the overall meta-analytic estimate of the training effect size.

Second, although random sampling and assignment should eliminate pre-existing differences between the training and control groups, smaller samples provide less accurate estimates of the population values and will be more strongly influenced by an outlier value, and as such pre-test differences between training

and control groups may be more likely. However, the use of small sample sizes also means that statistical tests of pre-test values are likely to be non-significant, allowing researchers to declare the training and control groups did not differ at pre-test ("*p* > 0.05") despite the numerical differences between the groups. Note also that researchers are relying on failure to reject the null hypothesis as evidence for no difference between the training and control groups at pre-test (see Redick et al., 2013, for related discussion). As seen in the studies reviewed here (**Figures 1** and **2**), the control groups' pre-test values were numerically but not significantly higher across several of the transfer tests, complicating interpretation of results when compared with the videogame training groups.

In closing, we offer a few suggestions for future videogame training studies. First, as noted in other recent reviews (Boot et al., 2011, 2013; Shipstead et al., 2012; Green et al., 2013), we strongly encourage the use of appropriate experimental control procedures that measure and counteract placebo effects. Second, for the reasons outlined above, we advocate the use of much larger sample sizes for training and control groups. Third, we strongly recommend presenting the pre- and post-test values for transfer tests, instead of or in addition to only reporting pre- to posttest change scores. Presenting the pre- and post-test values will allow the reader to determine if the pattern of significant results allows a strong conclusion or if there is ambiguity in the transfer results. A further suggestion is to provide the full data for each participant as supplemental material. Having the full data will be beneficial for interested readers and researchers conducting future meta-analyses, as they will be able to conduct both between- and within-subject analyses and additionally make comparisons not presented in the article by the authors. Finally, we note there is considerable debate about how best to statistically assess change in such intervention studies, with various authors pointing out limitations with independent-samples *t*-tests of gain scores, group by session interactions in ANOVA, and using pre-test scores as covariates in an ANCOVA (Lord, 1967; Huck and McLean, 1975; Miller and Chapman, 2001; Wright, 2006). Sampling techniques that lead to similar pre-test values for training and control groups will help minimize differences among the statistical analyses, whether that is achieved via random assignment with larger samples or through some sort of matching technique (for pros and cons of matching, see Green et al., 2013). In addition, incorporation of Bayesian analyses to either supplement or replace null-hypothesis significance-testing may more accurately quantify the intervention effect in a particular study (for a recent example of the use of Bayes Factors in cognitive training research, see Sprenger et al., 2013).

Above all, we hope that researchers will not focus so much on obtaining a significant *p*-value that they fail to examine the pattern of results to understand the cause of the significant result.

## **AUTHOR CONTRIBUTIONS**

Thomas S. Redick and Sean B. Webster contributed to the literature review. Sean B. Webster contacted authors for necessary additional information. Thomas S. Redick and Sean B. Webster created the figures, drafted the manuscript, and approved the final version for submission.

## **ACKNOWLEDGMENTS**

While writing this article, Thomas S. Redick was supported by the Office of Naval Research (Award # N00014-12-1-1011). The authors thank Tyler Harrison for constructive feedback on an earlier draft.

## **REFERENCES**

**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 13 January 2014; accepted: 12 March 2014; published online: 26 March 2014. Citation: Redick TS and Webster SB (2014) Videogame interventions and spatial ability interactions. Front. Hum. Neurosci. 8:183. doi: 10.3389/fnhum.2014.00183 This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Redick and Webster. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Evaluating the relationship between change in performance on training tasks and on untrained outcomes

#### *Elizabeth M. Zelinski 1 \*, Kelly D. Peters 2, Shoshana Hindin1, Kevin T. Petway II 2 and Robert F. Kennison1,3*

*1 Zelinski Laboratory, Center for Digital Aging, Davis School of Gerontology, University of Southern California, Los Angeles, CA, USA*

*2 Psychology Department, University of Southern California, Los Angeles, CA, USA*

*3 Psychology Department, California State University, Los Angeles, CA, USA*

#### *Edited by:*

*Michelle W. Voss, University of Iowa, USA*

#### *Reviewed by:*

*Erika K. Hussey, University of Illinois, USA Sobanawartiny Wijeakumar, University of Iowa, USA*

#### *\*Correspondence:*

*Elizabeth M. Zelinski, Zelinski Laboratory, Center for Digital Aging, Davis School of Gerontology, University of Southern California, 3715 S. McClintock St., Los Angeles, CA 90089-0191, USA e-mail<EMAIL_ADDRESS>
Training interventions for older adults are designed to remediate performance on trained tasks and to generalize, or transfer, to untrained tasks. Evidence for transfer is typically based on the trained group showing greater improvement than controls on untrained tasks, or on a correlation between gains in training and in transfer tasks. However, this ignores potential correlational relationships between trained and untrained tasks that exist before training. By accounting for crossed (trained and untrained) and lagged (pre-training and post-training) and cross-lagged relationships between trained and untrained scores in structural equation models, the training-transfer gain relationship can be independently estimated. Transfer is confirmed if only the trained but not control participants' gain correlation is significant. Modeling data from the Improvement in Memory with Plasticity-based Adaptive Cognitive Training (IMPACT) study (Smith et al., 2009), transfer from speeded auditory discrimination and syllable span to list and text memory and to working memory was demonstrated in 487 adults aged 65–93. Evaluation of age, sex, and education on pretest scores and on change did not alter this. The overlap of the training with transfer measures was also investigated to evaluate the hypothesis that performance gains in a non-verbal speeded auditory discrimination task may be associated with gains on fewer tasks than gains in a verbal working memory task. Gains in speeded processing were associated with gains on one list memory measure. Syllable span gains were associated with improvement in difficult list recall, story recall, and working memory factor scores. Findings confirmed that more overlap with task demands was associated with gains to more of the tasks assessed, suggesting that transfer effects are related to task overlap in multimodal training.

**Keywords: cognitive training, aging, untrained outcomes, multimodal training**

## **INTRODUCTION**

Longitudinal declines in many cognitive processes, including memory, attention, working memory, and speed of processing, are normative in aging (e.g., Zelinski et al., 2011a). This has led to concerns that declines may negatively impact quality of life and increase the risk of losing independence, as cognition plays an important role in many activities of daily living including financial management (e.g., Jobe et al., 2001). At the same time, it has become increasingly clear that individual differences in healthy older adults' cognitive performance is associated with a wide range of potentially enriching experiences, including education, healthy lifestyle practices, engagement in cognitively challenging activities, social involvement, avoidance of stress, and positive attitudes that promote psychological well-being (Hertzog et al., 2009). Interventions to enhance cognition have also shown benefits; many of these involve training on tasks thought to benefit processes that decline with aging. An important indicator of the effectiveness of interventions designed to improve cognitive performance in older adults is whether training benefits generalize to tasks or cognitive activities that were not trained (e.g., Jobe et al., 2001). It is well established that training of specific strategies, such as mnemonics, does not produce transfer in older adults (e.g., Park et al., 2007). This approach to training holds little promise for reducing risk of decline or even supporting the maintenance of cognitive ability, possibly because older adults often do not apply strategies to new tasks. This may occur because older people experience difficulties in engaging such strategies (Zelinski, 2009), have greater willingness to use suboptimal strategies (Hertzog et al., 2007), or have poor memory self-concept (West et al., 2008).

However, extended practice of tasks such as dual-tasking or N-back, can transfer to untrained tasks (Zelinski, 2009). Game play that involves repetitive practice of cognitive skills that involve multitasking also can produce transfer (e.g., Basak et al., 2008; Anguera et al., 2013). A recent meta-analysis directly evaluated effects of extended practice cognitive training on untrained tasks. These interventions significantly improved older adults' performance on untrained cognitive tasks, with an estimated mean effect size of 0.32 after accounting for practice in the experimental and control groups (Hindin and Zelinski, 2012). All of the 25 extended practice studies in the meta-analysis evaluated improvements in untrained outcomes by comparing pre-post differences between experimental and control groups. None examined how individuals' performance was affected by training. Yet if transfer has occurred, those in the experimental group who gain more on the training task should improve more on the untrained task because the training should generalize to other tasks with common components (e.g., Persson et al., 2007; Lövdén et al., 2010) or at least the same task-specific demands (e.g., Buschkuehl et al., 2012). Several studies published subsequently to the Hindin and Zelinski meta-analysis have examined correlations between improvements on trained and untrained tasks in older adults, reporting significant correlations in the experimental group (e.g., Anguera et al., 2013; Stepankova et al., 2014).

McArdle and Prindle (2008) suggested that it is necessary to test for transfer with a more sophisticated modeling approach than the use of *t*-tests, ANOVA, or bivariate correlations. They argued that if trained and untrained tasks invoke similar constructs, these should be correlated at baseline as well as after training. This suggests that in order to assess transfer, existing relationships between performance on trained and untrained tasks at baseline should be accounted for, so that the independent relationship between baseline and posttest training and transfer task performance relationships can be ascertained. Relationships between the initial baseline and posttraining scores should also be accounted for, as individual differences in the construct measured may be related to performance gains (see also von Bastian et al., 2013). Therefore, the strongest test of whether training produces transfer is that those who received the training intervention show a significantly stronger relationship between changes in trained and untrained task performance after training than those in the control group after all other possible relationships between trained and untrained tasks prior to, as well as subsequent to, training in each group have been accounted for. It would also be expected that demographic covariates should not affect transfer if a clear interpretation of training benefits is to be made. Otherwise, interactions between the characteristics of participants and training might confound transfer.

McArdle and Prindle (2008) evaluated a series of structural equation models accounting for relationships between near (trained) and far (untrained) cognitive tasks that compared 699 participants trained over 10 h to improve reasoning with 698 members of a no-contact control group. Data were from the initial phase of the Advanced Cognitive Training for Independent and Vital Elderly (ACTIVE) trial (Ball et al., 2002), a randomized controlled single-blind study of three interventions examining whether older adults' cognitive abilities and everyday functioning could be improved over 2 years. The trained group had a higher latent change mean than the untrained group on the reasoning measures, as they had in the study, showing that training improved performance on the trained measure. The models also indicated that at baseline, relationships were significant and positive between the trained and untrained measures. There was also a significant and positive relationship between the trained and untrained latent change measures, but this relationship did not vary differentially for the trained and control group participants. Thus, this study showed no relationship between change in training and in transfer in the experimental group participants. However, no group effects of transfer had been observed in the main study (Ball et al., 2002), and the elegant structural analysis of McArdle and Prindle did not produce any new findings to support the existence of training-related transfer in the trained group. The present analyses extended the modeling approach of McArdle and Prindle to a different dataset that had produced transfer effects at the group level for the trained participants.

## **HYPOTHESES**

Data were from The Improvement in Memory with Plasticitybased Adaptive Cognitive Training (IMPACT) study (Smith et al., 2009). The training protocol of the IMPACT study is based on a conceptualization of age declines in memory that are associated with negative neuroplasticity. Mahncke et al. (2006) suggested that deficits associated with cognitive aging are due to reduced frequency of engaging in cognitively demanding activities with age, declines in the integrity of perceptual experience due to sensory deficits that lead to reduced signal to noise ratios in information processing, reduced neuromodulation of the attention-reward system due to reduced cognitive stimulation, negative learning, and coping with reduced stimulation by reducing cognitively engaging behaviors further, creating a negative spiral of increasing decline in cognitive functioning. This can be reversed by undoing the activities that cause negative neuroplasticity and engaging in activities that cause positive neuroplasticity: frequent intense practice of cognitively challenging tasks requiring fine sensory discrimination, rapid processing of sensory information, deep attention, and novelty (Mahncke et al., 2006). The training program, described below, was adaptive, so as to remain cognitively demanding, it improved the signal to noise ratio by training discrimination of increasingly finer differences between stimuli while reducing the stimulus presentation rate with sound compression, and included feedback and rewards to maintain deep attention. Stimuli ranged from sound sweeps, non-word syllables (phonemes), syllables, and verbal instructions, to stories. The primary training measure was performance on the simplest training task, time-ordered sound sweep discrimination, measured as the duration of the sound sweeps needed for high accuracy in performance.

The training program was multimodal in that multiple processes involving rapid auditory discrimination were trained. For example, the training tasks included discriminating easily confused phonemes, remembering them in order, remembering their locations in a matrix, remembering and following increasingly complex sets of instructions to move objects in particular sequences (e.g., *move the dog next to the girl with the black hat, then move the police officer to the front of the bank*), and remembering facts from stories. It was possible that the primary training measure of sound sweep discrimination might be differentially associated with outcome changes than another measure that had also been collected, syllable span. By assessing relationships of change in the two trained tasks in the IMPACT study, the issue of what changes are measured comes to the forefront. Most multimodal training studies do not include pre and post training measures of all aspects of the training, so it is difficult to determine what aspects of training gain are associated with transfer gains. In the present analysis, the transfer measures were not only different from the training tasks in terms of the specific materials used (e.g., numbers, letters) but tested recall where only recognition had been trained, using subtests from widely used clinical neuropsychological tests. They involved episodic memory recall or reorganization of material in working memory. That is, transfer tasks were not closely related to training tasks. The training tasks also differed substantially in terms of overlap with transfer tasks.

It was hypothesized that the complete IMPACT training program would produce transfer because the underlying neuroplastic mechanisms would have been improved, producing perceptual and memory representations with greater fidelity, so that there would be better performance on a range of untrained auditory memory tasks. Gains in neural timing and accuracy of auditory perception with the training used in IMPACT have been confirmed in an independent study of older adults (Anderson et al., 2013). The use of the speeded sound sweep discrimination task as the training measure in the published study (Smith et al., 2009) has relatively few components in common with more complex memory tasks. The speed task has a constant memory load of two sound samples, it is non-verbal, and requires emphasis on perception of the sweeps, which are presented in increasingly shorter durations. In the IMPACT study, data from another training task, the reproduction of sequences of easily confusable syllables (pat and m ˇ at), were collected. This task used a ˇ span measure, whereby sequences of syllables increased in length as individuals improved in their ability to discriminate and recognize them. Performance was measured as the maximal syllable span at pretest and posttest in the task and can be considered an index of training effect in the expansion of working memory span. This measure was not analyzed in the IMPACT publications, but its analysis allows for a comparison of transfer effects on the outcome measures with transfer associated it and with speed training. The syllable span task is a measure of working memory. It has been suggested that interventions that may be most effective for older adults are those retraining working memory or executive control processes (Lövdén et al., 2010). Training cognitive control such as coordination of information in working memory produces transfer in older adults to similar tasks (e.g., Buschkuehl et al., 2008; Karbach and Kray, 2009). The task in this study required discrimination of easily confused syllables presented for increasingly shorter durations, storing them, and remembering them in order. The number of syllables increased as performance improved. The phonemes are verbalizable, can be rehearsed, and the memory demands increase. Though these tasks were learned in the multimodal context, hypotheses about the relative amount of demand can be derived. In contrast to gains in the speeded time ordered auditory discrimination task, transfer may be more easily observed because of the mapping of relatively similar task demands to the untrained tasks (e.g., Buschkuehl et al., 2012).

Testing transfer from change in syllable span to change in the outcome measures of list and story memory and to working memory would provide an important test of the relationship of assessed training gain to transfer task gain based on task demand overlap. If similarity of demands is the critical predictor of transfer (e.g., Buschkuehl et al., 2012), training change in syllable span would show the strongest relationship with change in the working memory outcome tasks of backwards digit span and letter-number sequencing. Because working memory is also implicated in verbal memory, it was also hypothesized that transfer would also be observed in the other measures of the IMPACT study, though it was expected that story memory measures would show stronger transfer because reconstructing a story is more closely associated with working memory than is list memory (e.g., Lewis and Zelinski, 2010).

Individual differences that affect baseline performance, such as participants' age, should not be expected to affect transfer (see McArdle and Prindle, 2008). However, surprisingly few aging studies have examined how characteristics like age, sex, or education affect training gains. McArdle and Prindle (2008) found that age had a negative effect on baseline and change scores, that gender had small effects on pretest scores and that education affected only pretest scores. These relationships, however, did not affect transfer. In the present study, effects of age, sex, and education were included as covariates in the final set of analyses. Baseline memory outcome scores were expected to be more negatively affected by age, but positively by female gender and education as seen in other studies of memory in large samples (e.g., Zelinski and Gilewski, 2003). It was expected that being older would reduce training gains because of age-related limits on plasticity (e.g., Hertzog et al., 2009), but not the relationship between gains in training and transfer, following McArdle and Prindle (2008). Effects of gender and education on training gains were exploratory, as little was known about how these differences would affect training outcomes. It was also not clear whether transfer would be affected by those individual differences.

## **MATERIALS AND METHODS**

The IMPACT study had tested the efficacy of a commercially available computerized cognitive training program on the speeded auditory discrimination task and on untrained clinical neuropsychological measures of memory and attention (Smith et al., 2009). The study design was a double blinded randomized controlled trial comparing those who participated in the training, which used principles of brain plasticity, that is, was repetitive, adaptive, and trained perceptual discrimination, with active controls who watched DVDs of "usual treatment" educational television programs. Analyses were intent-to-treat. Participants were 487 healthy, cognitively normal men and women aged 65–93 recruited from communities in northern and southern California, and Minnesota. They were randomized into the training (*N* = 242) or active control (*N* = 245) conditions and given computers to use at home for the trial. Trained participants completed a series of six exercises focused on improving speed and accuracy of auditory memory. Exercises used computer-adaptive algorithms to maintain challenge. The specific exercises were:

*High or Low*: pairs of frequency-modulated sound sweeps. Participants indicated whether the direction of the sweeps is upward (from low to more high pitched) or downward (from high to more low pitched).

*Tell Us Apart:* pairs of confusable syllables, such as *bo¯* and *do¯*, are presented on the screen. One syllable was spoken and participants indicated which they heard.

*Match it:* a matrix of buttons was presented on the screen. Clicking a button revealed a written syllable that was spoken aloud. There were two buttons with the identical syllables in the matrix. Participants found the matched pairs; as they identified them correctly, the buttons disappeared until all were gone.

*Sound Replay*: Sequences of two, three, or more confusable syllables were presented auditorily. Participants listened to the syllables, then clicked buttons identifying the syllables in the order in which they were presented. There were more buttons on the screen than there were syllables, so the task involved recognition of the syllables as well as memory for their ordering. *Listen and Do*: A set of spoken instructions was presented. Participants saw a scene with various characters and structures on it, with instructions to click particular characters or structures or to move the characters. Participants followed the instructions in the order given.

*Story Teller:* Participants listened to segments of stories and answered multiple-choice questions about them.

Active controls watched educational television program series on their computers and answered questions about the content afterwards. Both groups completed their activities 1 h a day, 5 days a week for 8 week, totaling 40 h of exposure. Computers were removed from participants' homes after they completed their training. The top panel of **Table 1** shows demographic information for the experimental and control groups.

Performance was evaluated at baseline before randomization, within 3 weeks of training completion, and 3 months later. The primary outcome was a composite index score of performance on the auditory tests of the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Randolph, 1998), a test relatively insensitive to age declines before 65. The RBANS test was developed to detect dementia in older adults but is also used to screen younger adults for impairments in cognitive status. The subtests have two alternate forms. Alternate forms were administered at each test occasion. The subtests included in the analyses are:

*List learning:* A 10 word list is read to the participant for study/recall over 4 trials.

*Immediate List Recall:* The total number of words recalled correctly over the trials.

*Standard deviations are in parentheses.*

*Delayed List Recall:* recall of the list after completion of seven other tests.

*List Recognition:* selection of the 10 study words from a list of 20 read by the examiner.

*Story memory:* A short story is read aloud and recalled over two trials.

*Immediate Story Recall:* total number of ideas recalled over the two trials.

*Delayed Story Recall:* recall of the story after 7 other tests.

*Digit Span:* digit span forwards.

The primary outcome consisted of a normed index score based on the six subtests. Secondary outcomes included performance on the trained speeded sound sweep discrimination task, and on untrained tasks: an auditory memory and attention index composite of list learning scores from the Rey Auditory and Verbal Learning Test (RAVLT) an age-sensitive and more difficult test than the RBANS (Schmidt, 1996), story memory from the Rivermead Behavioral Memory Test (Wilson et al., 2003), and letter number sequencing and digit span backwards from the Wechsler Memory Scale (Wechsler, 1997). Published findings of the IMPACT study revealed significant Group × Time interactions shortly after the training ended on the primary outcome, on the secondary composite scores, on the trained task, and on individual test scores including RAVLT list memory and delayed list recall, WMS digits backwards, and letter-number sequencing, with larger posttest gains for the experimental group (Smith et al., 2009). Means and standard deviations of the individual tests for the experimental and control groups are published in Smith et al. (2009). Three months after training was discontinued, gains of the plasticity training group were somewhat reduced, but significant Group × Time interactions for the trained auditory discrimination task, the secondary composite, and for RAVLT word list recall and WMS letter-number sequencing indicated retention of gains in the trained group (Zelinski et al., 2011b).

### **MEASUREMENT MODEL OF UNTRAINED OUTCOMES**

Data from the pretest and immediate post-training assessments of the IMPACT study were analyzed. The published analyses included primary and secondary experimenter-determined outcome measures that had not been evaluated empirically for their psychometric characteristics. Initial analyses of all subtests administered were conducted to confirm the structure of the two outcomes of the IMPACT study as latent variables so that transfer to the common construct they represented rather than to specific test scores could be appropriately assessed (see Lövdén et al., 2010; Schmiedek et al., 2010). The data were from all participants at pretest, including those who dropped out during the training phase of the study. Confirmatory factor analyses indicated very poor fit of the individual baseline tests to the published experimenter-defined measurement structure of RBANS auditory memory and to the secondary measures of the auditory memory and attention index measure. A psychometrically sound structural model of the untrained outcomes had to be developed in order to test transfer. Individual test scores were evaluated for their intercorrelations, and those with non-significant correlations with all other tests were dropped, leaving 11 scores for further analysis.

Measurement models of the outcome variables were next assessed using *R* (R Project Homepage: http://www.R-project. org). To identify the model that best characterized the structure of the data, exploratory maximum likelihood factor analyses (*R*: psych, version <IP_ADDRESS>), extracted 2, 3, 4, and 5 factors, with each indicator (test score) constrained to load only on one factor. A Promax rotation was used to allow factors to correlate, and no equality constraints were imposed on factor loadings. Each model was compared to an independence null model, in which covariances among all observed variables were constrained to zero. For this analysis, four fit indices to determine goodness of fit were used: *RMSEA* (root mean square error of approximation; Steiger, 1990) with a value <0.08 (Browne and Cudeck, 1992), *SRMR* (standardized root mean square residual; Joreskog and Sorbom, 1988), *TLI* (Tucker-Lewis Index; Tucker and Lewis, 1973), and *BIC* (Bayesian Information Criterion; Schwarz, 1978). Results are shown in the top panel of **Table 2**. For the 2-, 3-, and 4-factor models, fit indices were relatively poor (*RMSEA* > 0.1, *SRMR* ≥ 0.1 for 2- and 3-factor models, TLI < 0.9). Fit indices for the 5-factor model were acceptable, and this model produced the lowest BIC value out of all the models examined.

Confirmatory factor analysis (*R*: lavaan, version 0.5–15) on both the 4- and 5-factor models was next conducted. Each indicator was constrained to load only on the factor it measured and factor covariances were freely estimated. All available data were included in the maximum likelihood estimation. Four fit indices were used to determine goodness of fit: *RMSEA, SRMR, TLI,* and *CFI* (Comparative Fit Index; Bentler, 1990). Like the *TLI*, the *CFI* takes into account the χ2 and df of hypothesized model and null model, with values ≥0.95 indicating good fit (Hu and Bentler, 1999).The χ2 test itself was not used because the sample size of 487 was relatively large, inflating its values so that it would differ significantly from zero under most circumstances (Marsh et al., 1988).

Results of the confirmatory factor analyses supported a 5 factor model. Fit indices for the 5-factor model indicated acceptable fit, whereas fit indices for the 4-factor model were not as strong (see **Table 2**, lower panel). The 5-factor model consisted of: RBANS list memory, the RBANS list learning, list recall, and list recognition scores; RAVLT list memory, immediate recall and delayed recall measures of the Rey Auditory and Verbal Learning Test; RBANS story memory, the story memory and story recall

measures from the RBANS; RBMT story memory, the immediate and delayed story recall from the Rivermead Behavioral Memory Test, and WMS Working Memory, the Wechsler Memory Scale letter-number sequencing and backwards digit span scores.

The next step was to assess invariance of the 5-factor measurement model between the experimental and control groups at pretest to ascertain that the variables measured the same construct and therefore had the same meaning in both groups (see McArdle and Prindle, 2008). Increasingly stringent measurement invariance was assessed using four models in *R*: lavaan (version 0.5–12): configural, metric, scalar, and structural invariance. Configural invariance indicates that the variables load on the same factors across groups, but the value of the factor loadings may vary. Metric invariance indicates that the factor loadings are identical across groups. Scalar invariance indicates that the item intercepts are identical across groups, and structural invariance indicates that the factor means are identical across groups (see Horn and McArdle, 1992). Indices used to evaluate overall model fit included the normed χ2 (χ2/*df;* Wheaton et al., 1977). A χ2/*df* ratio of 3:1 or less indicates good fit (Carmines and McIver, 1981). *RMSEA* was also included in the invariance analyses. Fit statistics are shown in **Table 4**.

Results from the invariance analyses of the 5-factor model across the experimental and control groups supported the strictest measurement invariance and structural invariance, as fit did not worsen with increasing stringency of invariance tests. The models did not vary in *CFI* (.97) but the structural model resulted in a smaller *RMSEA* of 0.06 compared to the 0.07 of all other models. The χ2*/df* ratio for the structural model (1.96) also indicated the best fit relative to metric (2.06), scalar (2.06), and configural (2.26) models.

This indicated that any observed differences between experimental and control groups on the factors could be interpreted as representing differences in the same constructs. **Table 3** shows the factor loadings and communalities for the tests in the five-factor model.

The five between-group invariant factors identified in the measurement model seen in **Table 3** were represented by unit weight factor scores of the tasks that loaded on each factor, that is, the sum of the scores on each of the factors. Factor scores was used instead of latent variable models of each factor because analyses estimating latent factors either did not converge or produced non-positive definite covariance matrices.

## **STRUCTURAL EQUATION MODELS**

Multigroup structural equation models were used to test the hypothesis that latent change in each trained task was associated with latent change in each untrained variable after controlling for crossed, lagged, and cross-lagged relationships between the trained and untrained scores assessed at pre and at posttest in the experimental but not in the active control group. The model is shown in **Figure 1**. Rectangles represent manifest variables and circles latent ones. The triangle is an indicator of the latent change means. Indicators of training effects were the time order judgment sound sweep discrimination task, referred to in the tables as *speed* and the recognition of sequences of confusable syllables,

### **Table 3 | Standardized factor loadings of the structural invariance model for outcomes.**

*RBANS, Repeatable Battery for the Assessment of Neuropsychological Status; RAVLT, Rey Auditory Verbal Learning Test; RBMT, Rivermead Behavioral Memory Test; WMS, Wechsler Memory Scale.*

referred to as *syllable span*. Analyses were conducted separately for each training effect indicator and for each of the five outcome factor scores.

The modeling approach involved estimating the maximum likelihood parameters for the illustrated bivariate change score model and testing whether selected parameters differed between the experimental and control groups. Analyses were conducted separately for each of the five outcomes and the two trained indicators in an intent-to-treat design, so that all available data, including those of the dropouts, were included. For all models, it was assumed that random assignment to groups eliminated baseline differences in test scores so that baseline intercepts for

the trained and untrained variables were set to be equal for both groups. Model 1 was set to be completely invariant over groups, with all parameters constrained to be equal. Model 2 freed the intercepts for the latent change of the training and the outcome indicators across groups with all other parameters constrained to be equal. This tested the hypothesis that training affected the means of the trained and untrained outcomes. Model 3 included the freed intercepts and the regression parameters of the crossed and lagged relationships between pretest and latent change of trained and untrained outcomes across groups. Model 4 additionally freed the variances of the latent changes for trained and untrained outcomes.

## **RESULTS**

**Table 4** shows the observed means and standard deviations for the trained and control groups on the pretest and posttest trained measures and untrained factor scores. Latent difference scores, however, were analyzed in the structural equation models.

**Table 5** shows the model fit results. Fit indices included the nested -*2 Log Likelihood (-2LL*)/number of *df* test, which subtracts the value of *-2LL* and *df* from each successive model, with the - *-2LL*/*df* tested using the χ2 distribution to determine a significant improvement in fit from the prior model, with a significant χ2/*df* indicating improvement in fit. This, together with the smallest *AIC*, and smallest *RMSEA*, was used to select the best fitting model to characterize the trained and control groups.

Results for sound sweep discrimination training are seen in **Table 6** and **Figure 1** for the training effects of speed on RBANS list memory. The models that best fit the experimental and control groups for each outcome factor score with the training indicator of speed were the ones that freed all tested parameters, indicating that those parameters in the structural model differed across groups. For all outcomes, the fit indices for Model 4 were the smallest of all four models and there were significant reductions in *-2LL*. The critical regression parameter for this study was the path from the latent change in the speed training measure to the latent change in each outcome.

**Table 6** shows the unstandardized and standardized parameters for the analyses. The covariance at pretest between speed and each outcome (Speed Pre ↔ Outcome Pre) in the first row of the middle panel of **Table 6** was significant, indicating a relationship between the two measures before training. The standardized values are their correlations, which were low, ranging from −0.16 to −0.21 for the four memory factor scores and with a moderate value of −0.38 for WMS Working Memory.

Intercepts for latent changes on all of the outcomes (1→ -Outcome) differed significantly from zero for the experimental and control groups, suggesting that practice effects were observed in both groups. Pretest speed and pretest outcome performance were negatively associated with their respective latent changes (Speed Pre → -Speed; Outcome Pre→ -Outcome), indicating greater change in those with lower baseline scores and possibly regression to the mean. This was the case for both the experimental and control groups. Crossed and lagged relationships between speed and outcome measures were significant,

**Table 4 | Means and Standard Deviations (in parentheses) for the pretest and posttest scores on the trained tasks and untrained task factor scores for the experimental and control groups.**

**Table 5 | Nested tests of fit for models with speed (top panel) or syllable span (bottom panel) and each of the outcome factor scores testing parameter differences between experimental and active control groups.**

*(Continued)*

#### **Table 5 | Continued**

*Model 1, Fully invariant; Model 2, Model 1* + *different latent intercepts; Model 3, Model 2* + *different regressions; Model 4, Model 3* + *different posttest variances.*

*aModel selected as the best-fitting model. CFI* = *1 for all best-fitting models.*

confirming the need to control for them in assessing training effects.

Most critically, the test of transfer as the independent relationship between latent speed and latent outcome change was significant only for the experimental group on the RBANS List Memory factor score. Transfer was not observed in the RBANS Story Memory, RAVLT List Memory, RBMT Story Memory, or WMS Working Memory factor scores.

The next series of analyses evaluated model fit with syllable span as the training measure with results seen in the lower panel of **Table 5**. Unlike for the speed training task, the model testing syllable span task parameters less consistently differentiated between parameters for the experimental and control groups. Selecting the best fitting (or least misfitting model) required consideration of the relative weight of the fit indexes because of contraindications across them. For example, the -*2LL/ df t*est was significant for Models 3 and 4, indicating no fit improvements beyond those of Model 2. However, *AIC* was smaller for Model 3 than for Model 2 for RAVLT List memory, RBMT Story Memory and WMS Working Memory, and smaller than for Model 4 for all of those outcomes. *RMSEA* was generally smaller for Model 3 than Model 2, but it was decided that Model 2 would be considered best fitting if it had the lowest *AIC* and an *RMSEA 90% CI* that did not differ from that of Model 3. Otherwise, Model 3 was selected as the best-fitting. Thus Model 2 was considered the best-fitting model for the two RBANS factor scores. Model 3 was considered best-fitting for RAVLT List Memory, RBMT Story Memory, and WMS Working memory.

The pretest standardized covariances, shown in **Table 7**, that is, the correlations between syllable span and each outcome were moderate for the memory factor scores, with the smallest values of 0.23 for the correlation with RBMT Story Memory, and from 0.32 to 0.36 for the other measures. The correlation was 0.64 for syllable span training with WMS Working memory. These pretest relationships were larger than those observed for the relationships of speed with the outcomes, suggesting more overlap. The intercepts for latent changes in syllable span were significantly greater than zero for both groups, suggesting the presence of a practice effect, as they were for speed. Negative relationships between pretest and latent change in syllable span indicated more gains in those with poorer baseline scores, implying regression to the mean in both groups.

*\*p* < *0.05. Equals signs indicates that the parameter was constrained to be equal for the experimental and control groups.*

**Table 6 | Maximum**

**experimental**

1 1

 **and control groups.**

 **likelihood estimates**

 **and**

**standardized**

**parameters**

 **(in**

**parentheses)**

 **of the best-fitting**

 **bivariate models for effects of sound sweep**

**discrimination**

 **training in the**

1 1

The differences in parameters for the two RBANS factor scores in Model 2 suggested that the trained group only differed from the control group in the amount of improvement in the model intercepts but not the regression parameters. These did differ for the remaining outcome factor scores for which Model 3 was the best fit. The critical test of the relationship between latent change in the trained and in the untrained scores was significant for RAVLT List Memory, RBMT Story Memory, and WMS Working Memory for the experimental but not the control group, indicating evidence of transfer. In addition, the relationship between the two latent change variables was significant for syllable span and RBANS Story Memory but because the path was constrained to be equal for experimental and control groups in that model, it did not demonstrate transfer of training as defined in the analysis.

The final set of analyses tested whether transfer was associated with individual differences. They included the covariates of age, sex, and education, all of which were associated with baseline training task performance. Bivariate change models tested the baseline and latent change trained and outcome variables regressed on the covariates, with covariate effects fixed across experimental and control groups, because of random assignment. The critical relationships of latent changes in training and outcomes were free to vary. **Table 8** shows the standardized estimates for speed and syllable span, which were identical across outcomes, and **Table 9** the standardized estimates for each of the five outcomes, which were identical across training task analyses, and for the latent change- to- latent change regression coefficients for each training task.

For Speed, the pretest scores only were associated with the covariates. Being younger and male were associated with lower (faster) speed. Paradoxically, having more years of education was associated with slower performance. No correlations were observed for the latent change of speed. Age was negatively associated with syllable span at baseline, with worse performance, and more education was associated with higher scores. For latent change in syllable span, being older was associated with less gain and more highly educated with more gain. There were no sex differences in associations with baseline or latent change syllable span.

The covariates, as expected, had significant relationships with the baseline outcome factor scores, as seen in **Table 9**. Older people had lower baseline scores on all of the outcomes. Women were

**Table 8 | Standardized regression parameters for the analyses of the regression of training task variables on age, sex, and education.**

*Parameters were constrained to be identical across training groups. \*p* < *0.05.*

**experimental**

 **and**

**Table 7 | Maximum**

 **likelihood estimates**

 **and**

**standardized**

**parameters**

 **(in**

**parentheses)**

 **of the best-fitting**

 **bivariate models for effects of syllable span training in the**

*\*p* < *0.05. Equals signs indicates that the parameter was constrained to be equal for the experimental and control groups.*

better on baseline list memory factor scores, for both the RBANS and RAVLT. More education was associated with better baseline performance on all five factor scores. Age was associated with latent changes in the outcome variables, with less gain for older individuals. Female gender was associated with larger gains on RBANS List Memory, and more education with greater gains on WMS Working Memory.

Despite the relationships of covariates with the outcomes at pretest and for their latent changes, all of the significant latent change training-latent change outcome relationships observed in the main bivariate analyses for the experimental but not the control group remained significant after accounting for covariates. Transfer was therefore independent of the covariates.

## **DISCUSSION**

The goal of cognitive training of older adults is to support them in either maintaining or improving their functioning. Critical to this is the effectiveness of training in producing transfer. It has been suggested that multimodal cognitive training will produce transfer to multiple outcomes (e.g., Basak et al., 2008). However, it is not clear whether transfer is more likely to be observed, in the context of multimodal training, in training tasks that have greater demand overlaps with outcomes, and this was a focus of the present study.

Data modeling included controlling for relationships in performance between trained and untrained tasks not only at baseline, but subsequent to training, in a study dataset that showed improvement in untrained task performance after training at the group level. The data source was the IMPACT study, which involved a design with many strengths, including being the largest multisite randomized controlled double-blind trial of a commercially available cognitive training program with 487 participants over age 65 in experimental and control groups. It included an active control group and was conducted at three different sites. Published results showed interactions between experimental/control group participation and assessment visit, with the trained participants showing better performance, and Cohen's *d* effect sizes for the interaction ranging from 0.20 to 0.33 (Smith et al., 2009). However, like most studies in the cognitive training literature, data analyses were only conducted at the group level and only one training effect was reported.

Transfer from a task assessing the speed of discriminating time-ordered sound sweeps was assumed to reflect relatively less task demand overlap with the outcome constructs than transfer from a task assessing expansion of syllable span. Results suggested that transfer to a relatively easy list memory outcome was associated with improvement in the training indicator of speed, and that transfer to relatively difficult list memory, story memory, and working memory outcomes were associated with improvement in the training indicator of syllable span.

Because change in the speeded non-verbal training task was associated only with latent change of one memory task factor score, its utility in the measurement of transfer in this study was limited. Processing speed has long been characterized as a cognitive primitive (e.g., Salthouse, 1996) that underlies age related performance declines in many cognitive tasks, including memory. Perceptual speed was significantly associated with memory for word lists but not for text memory in cross-sectional research (Lewis and Zelinski, 2010). However, perceptual speed training gain in the present study showed transfer only to one factor score from a neuropsychological test that does not differentiate performance at ages under 65 (Randolph, 1998). The task demand explanation would suggest that rapid processing of non-verbal auditory information overlaps only somewhat with skills involved with rapid processing of the relatively low-retrieval demand material of the RBANS list memory factor scores. That score is based on a 10-item 4-trial free recall + delayed recall of the same list. In comparison, the RAVLT list memory factor score is based on a 15-item 5-trial free recall + free recall of an interference list followed by initial list recall, + delayed interference list recall. A lack of transfer was also observed for training on a perceptual speed task and list recall in the ACTIVE trial (Ball et al., 2002) as well. This suggests that improving on a non-verbal training task with a fixed and low memory load has only limited value as an indicator of transfer to gains in verbal memory.

On the other hand, improvement in syllable span was associated with transfer to the more difficult RAVLT list memory, RBMT Story Memory, and WMS Working Memory factor scores. Age declines in working memory performance are well documented, and working memory has been considered to be an important mechanism in word list recall and text recall, as coordinating to-be-remembered information in working memory contributes to retrieval of both item- and discourse-level information (e.g., Lewis and Zelinski, 2010). The largest standardized parameter was observed for the effect of gains in syllable span on gains in the factor score derived from two re-sequencing span measures. It was predicted that the transfer relationship would be stronger for working memory outcomes that for recall outcomes because of similarities in span task demands. This was confirmed. The standardized coefficients for list and story memory transfer, on the other hand, were similar. Pretest correlations were greater for syllable span and the outcomes than for speed and outcomes, suggesting more commonalities of syllable span with transfer measures at baseline. Because those relationships before and after training were covaried in the structural equation model, the relationship of latent changes in training and in transfer was independent of those influences.

If the present analysis had only included the targeted sound sweep discrimination measure, the argument of transfer from the training program would be only weakly supported. By analyzing gains on another training task, the transfer findings suggests extension to more outcomes that tap into similar constructs as those trained. Thus, in general, the findings support an overlapping task demand model of transfer not due to confounding of crossed, lagged, or cross-lagged relationships.

The findings of task-specific transfer are confirmed by several studies reporting limited transfer between different working memory/cognitive control tasks and untrained working memory tasks (Buschkuehl et al., 2008; Li et al., 2008; Karbach and Kray, 2009; Schmiedek et al., 2010). Dahlin et al. (2008) found that, after working memory training, brain activations in young adults increased in the striatum during working memory updating training as well as during transfer tasks. Older adults showed activation during the trained but not the transfer task and showed no evidence of behavioral transfer. Thus, transfer may suggest similarity of functional neural activation patterns between the trained and transfer tasks, but this is not consistently observed (see Buschkuehl et al., 2012).

In the present study, individual differences among participants affected latent change independent of baseline functioning. Increasing age was associated with reduced latent change in all measures except for speed, female sex was associated with more latent change in RBANS List memory, and more years of education with more latent change in syllable span and in WMS Working Memory. This suggests that, as found elsewhere, very elderly adults gain less from training than younger ones, but they do show some benefit (see Buschkuehl et al., 2008; Hertzog et al., 2009; von Bastian et al., 2013). Female gender and more education were associated with better baseline cognitive performance, as is often observed, but this is the first study to demonstrate a benefit for women in list recall and for more years of schooling in training and transfer gains in working memory span tasks. Most critically, significant transfer in the experimental group only from latent trained change to latent outcome change remained significant.

## **METHODOLOGICAL IMPLICATIONS**

The findings confirm the value of assessing relationships between trained and untrained scores in evaluating transfer. In all cases, there were significant pretraining relationships between the trained task and outcome factor scores for both experimental and control groups. The findings of significant intercepts for latent change in the models for both trained and control participants showed that practice effects were present in both groups. Practice may inflate the apparent training effect size considerably if only the data of experimental groups are included in transfer task effect size computation (see Hindin and Zelinski, 2012). Many training studies only use repeated measures ANOVA of untrained tasks to assess transfer, which accounts for practice, but this study suggests that such findings may be compromised by the complex of pretraining and postraining relationships between trained and transfer measures.

Recently, theoretical concerns about the interpretation of correlational relationships of gains in trained and transfer variables based on observed strong relationships between baseline task performance measures have been raised (e.g., Redick et al., 2013; Tidwell et al., 2014). It has been assumed that strong baseline relationships indicate that gain score relationships in the trained group reflect a causal change. In the working memory literature, the very strong baseline relationship between working memory and intelligence has been suggested by some as evidence that working memory training can improve intelligence. This has led to the use of analyses that produce misleading results.

Several recent studies that did not report training group differences in transfer used responder analyses to test for training effects (e.g., Jaeggi et al., 2011; Redick et al., 2013; Novick et al., 2014). The idea is that because not all participants improve with training, they should be categorized based on training outcomes, with correlations of change scores for trained and untrained tasks within successful and unsuccessful outcome groups computed. As Tidwell et al. (2014) have shown, this categorization is problematic because of lack of inclusion of control participants, a restriction of range for correlations, and spurious relationships between changes in training and transfer.

In addition, Moreau and Conway (2014) showed that even if training did produce transfer, strong pretest correlations do not guarantee strong gain correlations. Gains on both tasks may be negligibly related, for a number of reasons, but especially if the gain score correlations are computed for manifest variables, which contain error. Negative relationships between pretest trained and untrained scores and their respective changes, possibly because of regression to the mean, have also been observed in training studies (e.g., Whitlock et al., 2012). Shipstead et al. (2010) note that this problem affects outcomes, but is generally ignored. Because of these measurement problems, it is crucial to assess the relationship between training and transfer change independent of all major confounding relationships and to assess latent change, which is free of error. Another issue is that studies in the training literature rarely use intent-to-treat analyses, which include all pretested participants, and any training data, even of dropouts, to represent all available data, not just that of those selfor experimenter-selected to participate. When maximum likelihood algorithms are used in modeling with all available data, this reduces the possibility that systematic individual differences in dropout characteristics leads to biased findings. One of the serious problems in the training literature is that most published experiments do not include sample sizes adequate for the sophisticated modeling of effects that account for possible confounds as presented here. Many studies are additionally underpowered in terms of sample size and duration of training, thus limiting exposure to the intervention (see Basak et al., 2008 for an example).

We therefore agree that correlational modeling, as practiced in the literature, suffers from interpretive problems, and that unless the complex of interrelationships between trained and transfer measures is assessed and covaried in all participants within the experimental and control groups, latent variables are evaluated, and all available data are modeled, the problems described here lead to interpretive difficulties.

Suggestions have also been made that biases in interpretation of training effects exist because, in effect, competing hypotheses rather than the null hypothesis, are being evaluated. In the working memory training literature, the hypothesis that training transfers to abilities like intelligence assumes that the null is simply the absence of transfer. However, an alternative hypothesis is implied by the intelligence literature, which suggests that the abilities cannot be improved by training (see Tidwell et al., 2014). The Bayesian approach evaluates the likelihood that findings support the null vs. a transfer hypothesis. Following Sprenger et al. (2013), we computed Bayes-factor analysis of the Group x Time interaction effects observed in the Smith et al. (2009) paper, transforming them to two-sample *t*'s because there was 1 *df* in the numerator of the F ratio. We found that one of the seven previously significant interactions on untrained tasks was shown instead to support the null hypothesis with a Bayes factor value of 1.59. A total of 9 untrained task scores (including those that were not significant) was analyzed to compute the median Bayes Factor, which, for all reported outcomes, was 0.79, thus in favor of modest transfer effects.

Hindin and Zelinski (2012) assessed quality of extended practice training studies in their meta-analysis and found that studies with higher quality (measured with respect to random assignment to conditions, reports of attrition, sample size, etc.) had larger effect sizes for transfer tasks. The mean estimated effect size of *d* = 0.32, equivalent to *r* = 0.16, associated with transfer in older adults (Hindin and Zelinski, 2012) may seem inconsequential relative to effect sizes for pre-post change in a trained task. However, many medical interventions become clinical practice with much smaller effect sizes, for example *r* = 0.02 for the effect of aspirin and reduced risk of death by heart attack (Meyer et al., 2001). Provigil (Modafinil), a narcolepsy drug, used off-label to improve working memory and attention, has an estimated mean effect size on working memory and similar tasks of *r* = 0.11 or *d* = 0.23 in young adults (Hindin and Zelinski, 2012). Although expectation of substantial transfer effects, that is, those as large as effects for improvements in pre- to post-task training, may be unrealistic, we note that transfer effects for working memory interventions, largely in children, as shown by Melby-Lervag and Hulme (2013) are smaller and not different from zero. Older adults may show more transfer from training than young adults on average because their baseline performance is worse due to reduced neuroplasticity, which is re-engaged with training (see Mahncke et al., 2006).

## **LIMITATIONS**

Tidwell et al. (2014) suggest that computation of correlations between trained and transfer tasks are uninformative because it is likely that measurement characteristics of the training task are not invariant as a result of exposure. This is a concern for the current study, but individual item scores were unavailable for differential item functioning analyses before and after training.

Concerns raised in the literature include the observation that training is adaptive whereas active control conditions generally are not, and this was true of the present study. Though this could bias findings because adaptive training promotes performance improvements to a greater extent than standardized training, and because there may be different levels of motivation and strategy use that may affect outcomes in experimental and control groups, the evidence for this potential source of spurious training and transfer effects is quite weak (see Redick et al., 2013).

In the present study, there was a trained group and an active control condition with double blinding. A concern in clinical trials, even with double blinding, is whether the trained group gets more attention from study staff and whether there is an implicit message because of unchallenging sham material that control participants are not getting the experimental treatment, so that they experience less social interaction and expect less improvement, both of which dampen performance. In the present study, there were no differences in the amount of interaction with trainers for the two treatment groups. Participants had been told that after the study was completed, they would receive upon request copies of the training materials that produced better outcomes on the untrained tasks. Some of the control participants requested copies of the DVDs they had watched. This suggests that expectancies of cognitive benefits, which could affect performance, were present in some control participants (see Boot et al., 2013), but this was not systematically assessed so it is unknown whether the majority of those in the control group did expect to improve and to the same degree as those in the training condition on the outcomes.

The study was not informative regarding change in underlying processes compared to overlap in similarities in task characteristics. This could not be evaluated for three reasons. First, the neural basis of overlap was not tested. Second, the multimodal training design could not rule out complex sources of transfer. Third, the speeded auditory discrimination and syllable span tasks differed with respect to whether they were non-verbal or verbal, as well as on their measurement characteristics. Though the findings would suggest that syllable span was more effective for transfer to recall memory than time-ordered sound sweep discrimination, we note that training effects from the four other trained tasks in the program used in the IMPACT study could not be assessed. We also note that all training tasks involved adaptive speeded processing and difficult auditory discrimination training, and that with the extant design, the specific benefits to transfer within the training program could not be isolated.

We note that what constitutes near and far transfer has not yet been objectively defined, and varies from study to study, so prediction of the amount of transfer that should be observed for a given outcome is difficult. In the present study, the most parsimonious explanation for performance improvements on untrained tasks in older adults is that of overlap in task demands, because training was multimodal. This is an important limitation. However, improvement in untrained tasks rather than broad abilities in older adults may have important implications for public health. The ACTIVE trial showed that training of reasoning and of speed was associated with reductions in risk of dependency 10 years after the study was initiated (Rebok et al., 2014). We agree, though, that elucidating the mechanisms of transfer is a critical goal for the cognitive training literature. Promising approaches for understanding the basis of transfer include testing neural activation patterns during task performance (e.g., Dahlin et al., 2008) and developing targeted tasks that clearly vary process engagement (e.g., Persson et al., 2007).

Other limitations to this study are those of the IMPACT study inclusion and exclusion criteria. This resulted in a convenience sample of very healthy participants, with high fluency in English, and low participation rates by members of ethnic minorities. Participants had committed to engage in the study for a minimum of 6 months. These characteristics suggest that the findings may not be generalizable to the population of older adults.

## **CONCLUSIONS**

The findings have positive implications for the cognitive training of older adults who are healthy and willing to engage in challenging and extensive multimodal training such as that provided in the IMPACT study. The current set of findings suggest that even when individual differences including age are incorporated into models that test transfer independent of other possible within-study influences, the relationship between latent changes in trained and untrained tasks generally remains significant.

## **ACKNOWLEDGMENTS**

This research was supported in part by the Center for Digital Aging of the USC Davis School of Gerontology, grants R01AG10569, P50AG005142, and T32AG00037 from the National Institute on Aging and H133E080024 from the National Institute on Disability and Rehabilitation Research to the University of Southern California, and a grant from Posit Science Corporation.

## **REFERENCES**

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 17 January 2014; accepted: 23 July 2014; published online: 13 August 2014. Citation: Zelinski EM, Peters KD, Hindin S, Petway KT II and Kennison RF (2014) Evaluating the relationship between change in performance on training tasks and on untrained outcomes. Front. Hum. Neurosci. 8:617. doi: 10.3389/fnhum.2014.00617 This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Zelinski, Peters, Hindin, Petway and Kennison. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Harnessing the neuroplastic potential of the human brain & the future of cognitive rehabilitation

## *Jyoti Mishra\* and Adam Gazzaley\**

*Department of Neurology, Physiology and Psychiatry, University of California, San Francisco, San Francisco, CA, USA \*Correspondence<EMAIL_ADDRESS><EMAIL_ADDRESS>
#### *Edited by:*

*Guido P. H. Band, Leiden University, Netherlands*

#### *Reviewed by:*

*Geert Van Boxtel, Tilburg University, Netherlands*

**Keywords: cognitive training, neurotherapeutics, cognitive control, neuroplasticity, closed loop**

Neuroplasticity is the remarkable ability of the brain that allows us to learn and adapt to our environment. Many studies have now shown that plasticity is retained throughout the lifespan from infancy to very old age (Merzenich et al., 1991; Merzenich and DeCharms, 1996; Greenwood and Parasuraman, 2010; May, 2011; Bavelier et al., 2012). Enriching life experiences, including literacy, prolonged engagement in the arts, sciences and music, meditation and aerobic physical activities have all been shown to engender positive neuroplasticity that boosts cognitive function and/or prevents cognitive loss (Vance et al., 2010; Hayes et al., 2013; Matta Mello Portugal et al., 2013; Newberg et al., 2013; Zatorre, 2013). Unfortunately, just as enriching experiences generate positive plasticity, negative plasticity ensues in impoverished settings. For instance, many studies now show that low socioeconomic, resource-poor environments, which are associated with stress, violence and abuse within families and communities, have detrimental effects on cognition and neural function (D'Angiulli et al., 2012; McEwen and Morrison, 2013). As cognitive neuroscientists we observe both positive and negative aspects of plasticity in neural systems, in functional changes of neural activations, neural oscillations and strength of connectivity between brain regions, in structural changes in gray matter volume and white matter integrity, and importantly in the relationship between such neuroplastic changes and concomitant cognitive/behavioral changes. As we come to understand various facets of plasticity, it drives further the quest to develop new activities/interventions that engender maximal positive plasticity in selectively targeted neural systems; we envision such activities will in turn generate "far transfer of benefit" to generalized cognition and thereby improve the human condition.

In today's modern technological and internet-connected era, individuals are increasingly engaging with cognitive training software to improve cognitive function. In fact over the past 10–15 years, several companies have become established proponents and marketers of such software, transforming it into a multimillion dollar industry with exponential projected future growth. The fact that this technology is easily accessible over the internet to the home-setting, and at low-cost, has facilitated it's mass adoption. Scientifically, however, not all "brain training" is made equal. All too often, basic cognitive neuroscience experimental paradigms are embedded in commercial "brain training" approaches with add-on visual graphic skins that attempt to maximize user-engagement; a process known as gamification. Although these experimental paradigms had been originally developed to understand cognition, that does not mean that they are also the best tools to engender positive neuroplasticity. It is no surprise then that some scientific investigations have uncovered that generic brain training approaches yield no positive cognitive outcomes (Owen et al., 2010). However, a blanket statement that all cognitive training is ineffective is also unfair. In recent years, development and evaluation of cognitive training approaches in many labs, including our own, has revealed evidence for positive neuroplasticity, as well as for transfer of benefit to untrained cognitive abilities (Tallal et al., 1996; Temple et al., 2003; Stevens et al., 2008; Smith et al., 2009; Ball et al., 2010; Berry et al., 2010; Anderson et al., 2013; Anguera et al., 2013; Mishra et al., 2013; Wolinsky et al., 2013). Furthermore, in two of our training studies we find neurobehavioral correlations that relate on-task neuroplasticity to broader improvements in untrained aspects of cognition. Other researchers have also reported positive findings and transfer of training effects to untrained cognitive abilities in the context of custom-designed working memory exercises (Klingberg, 2010; Rutledge et al., 2012), task-switching training (Karbach and Kray, 2009), as well as for a specific genre of commercially available games, i.e., action video games (Bavelier et al., 2012) (although it is difficult to make strong recommendations about many off-the-shelf games given concerns over violent content). From these studies we are coming to understand some of the design principles that may govern the development of effective neuroplasticity-targeted training, as well as the scientific evaluation methods that can be used to provide convincing proof of the efficacy of the training intervention. Here, we summarize some of these principles that have emerged from two of our published training studies that now inform the development and evaluation of our next generation of training tools.

In our first training study in older adults, we simply trained visual perceptual discrimination of Gabor patches that had built-in directed motion animation (Berry et al., 2010). Ten hours of training improved on-task perception relative to performance changes in a non-training (no-contact) control group. Interestingly, the training also benefitted delayed-recognition working memory of an untrained motion direction task. Not only was working memory performance improved, electroencephalography (EEG) neural recordings showed that training evoked more efficient sensory encoding of the stimuli, which correlated with the working memory performance gains. This finding that 10 h of simple perceptual training engendered transfer of benefits to working memory aligns with recent understanding that perceptual training improves signal to noise contrast, which then leads to refined encoding at multiple neural scales and hence, at least some degree of generalized cognitive benefits (Vinogradov et al., 2012).

We are now gaining an appreciation that the observed gains in our perceptual training study, and in similar studies performed by other labs, some of which have shown long-lasting cognitive benefits (Willis et al., 2006; Rebok et al., 2014), may be mediated by two fundamental design elements that drive neuroplasticity. 1) Training incorporated continuous performance feedback at multiple levels of game play providing repeated cycles of reward to the user 2) Training was adaptive to the trainee's in-the-moment game performance; i.e., adaptivity was incorporated using psychophysical staircase functions that enhance training challenge in response to accurate performance and reduce it for inaccurate performance. The up-down step ratio in such staircases is often chosen to maintain overall task challenge at 75–85%, at which point the user is optimally engaged but not frustrated. Thus, continuous performance feedback rewards and adaptive task challenge uniquely personalize the training to the cognitive capacity of each individual, and allows abilities to improve over time. Overall we have found these features to be critically important in generating positive neuroplasticity and cognitive benefit. Note, it is important to realize that casual game software is often not designed to provide the optimal dose of repetitive rewards nor incorporate adaptive progressions specifically targeted to the cognitive domains that may be deficient in a given population cohort. These factors, along with the heterogeneity of tested populations, very small training doses on multiple cognitive exercises, and the use of assessment measures that are insensitive to detect training related benefits in the tested population, all may contribute to a failure to observe positive impact (e.g., Owen et al., 2010).

While reward cycles and adaptive progressions are key components of software design, it is equally important to tailor these game mechanics toward improving specific deficits observed in a population cohort. For instance, Anguera et al. (2013) showed that deficient cognitive control abilities, such as working memory and sustained attention, in healthy older adults can be enhanced by specifically training on a multitasking performance-adaptive and rewarding video game, "*Neuroracer."* "*Neuroracer*" implements visual discrimination training in a go-no-go task for colored shape targets, with the added demand of simultaneously driving on a virtual road. "*Neuroracer*" evidenced extensive gains such that healthy older adults who multi-tasked 175% worse than younger adults on a first assessment, achieved significant post-training performance levels on the game itself that surpassed those of young adults by +44%. Importantly, training on "*Neuroracer*" transferred to untrained measures of sustained attention and working memory in the setting of interference, with EEG-based neural recordings showing that plasticity of midline frontal theta (mf theta) neural oscillations may be a mediator of these cognitive improvements.

While we have tested some aspects of sound game design, as described, other aspects of high-level video games may contribute to their success and we look ahead to assessing these empirically. For example, immersion, fun, real-world features, continuous performance, 3D environments, virtual reality, high-levels of art, story, and music facilitate sustained performance and better compliance, and also deeper engagement that we suspect maximally harnesses plasticity. Evaluation of the influence of these features on training effectiveness requires careful scientific study design. For this, the "*Neuroracer*" study adopted a rigorous three-armed randomized controlled design. In addition to the multitasking training group, the study included an active single-task training control, as well as a no-contact control group. The singletask training control performed the exact same tasks as the training group of visual discrimination and driving, except that task engagement was not concurrent. This active control directly tested our hypothesis that only training in a setting that stresses cognitive control via a high interference environment would show significant cognitive gains. Outcomes of the "Neuroracer" multitasking training were not achieved in the active control group or in the no-contact group, the latter being critical for assessing practice effects due to repeated evaluations. Thus, the "*Neuroracer*" study highlighted that rigorous scientific evaluation of a cognitive training approach requires appropriate control groups, and often more than one control group, especially if we want to understand the underlying mechanisms of training effectiveness. Indeed longitudinal data collection is arduous, but without randomized, controlled and single/double blinded enrollments, we cannot convince ourselves of the significance of the results of new interventions. This is especially appropriate for healthy populations, while single-arm feasibility trials do remain informative as a first pass in cognitively impaired populations. In addition, we should also implement expectation bias measures for all participants, which confirm that all study groups anticipate the same level of influence of their assigned intervention on the outcome measures, thus assuring appropriate placebo control (Boot et al., 2013). Finally, adequately powered large sample size studies and investigations that measure sustainability of the cognitive gains and underlying neuroplasticity in yearly follow-ups are rare and need to be performed more often to address the long-term efficacy of cognitive interventions. Such rigor is convention in pharmaceutical clinical trials, and its adoption for video game testing, along with safety evaluations that detect potential side effects such as gameaddiction, would promote a path toward FDA approvals and medical prescription of such technologies.

Equipped with our growing understanding of how to design cognitive training approaches to target plasticity in specific neural circuits, we are now embarking on the development of the next generation training technologies. We envision these advances to include combining behavior-digital closed loops that link behavioral performance metrics to adaptive modulations of a training task on a digital platform, with neuro-digital closed loops that link neural performance measures to adaptive game mechanics. For example, the "*Neuroracer*" training study discovered that neuroplasticity of midline frontal theta (enhanced mf theta posttraining) is a key neural factor that correlates with transferred cognitive gains. In order to test whether mf theta plasticity is truly causal in enabling improved cognition, we are now developing neurodigital closed loops that directly target mf theta activity.  The goal is for these measurements to be event-locked to task stimuli, account for ocular and muscle-related artifacts, and use source localization algorithms (Mullen et al., 2013) so that they can be directly integrated in the game environment to guide reward feedback to the user and adaptivity of task challenge in real-time. We hypothesize that using neural performance as the driver for task-adaptivity will generate more rapid, efficient and specific circuit plasticity than is currently obtained using behavior-adaptive cognitive training approaches. This hypothesis is borne out of data, which shows that single-trial behavioral performance is predicted by neural measures such as mf-theta oscillations preceding the behavior. Thus the neuro-digital closed loop offers the potential to selectively train and refine the bottleneck neural processes that govern the final behavioral outcome. Importantly, by directly embedding taskrelated neural activity in a closed loop, this approach can provide missing causal evidence between neuroplasticity and cognitive benefits. This line of investigation is especially promising in the light of accumulating scientific evidence of the value of conventional neurofeedback approaches (Gruzelier, 2013; Wang and Hsieh, 2013; Arns et al., 2014), which also creates a neuro-digital closed loop, albeit driven by ongoing scalp EEG oscillations as opposed to task-related neural processes as we envision.

We are aware that unlike traditional cognitive training, a neuro-digital closed loop approach is not feasible as a mainstay in the home setting at present. Yet, with rapid developments of mobile EEG technology (Stopczynski et al., 2013), as well as advances in the real-time computational power available on consumer devices such as laptops and tablets, we expect that deployment in the home environment will be a reality within a few years. Neurodigital closed loops are also an exciting way to achieve personalized therapeutics, as each feedback loop is customized to the individual user's neural capacities in the moment. While here we have provided a simplistic example of a closed loop tied to task-related mf theta activation, one can conceive of more sophisticated neural targets, including frontal-posterior effective connectivity based on task interaction dynamics. Further advances in this field are expected as neuroscientists collaborate with neural engineers, who have predominantly focused related efforts on neuroprosthetic development (Borton et al., 2013). Neuro-engineers have designed efficient closed loop decoding algorithms for brain-machine interfaces in animal model systems, and these techniques are now ripe for adoption in humans (Carmena, 2013). Finally, especially beneficial for clinical populations that exhibit weakened neural responsivity, another intriguing step will be the integration of neuro-digital closed loop systems with transcranial electrical current stimulation or even deep brain stimulation technologies (Coffman et al., 2013), which may provide a needed plasticity boost to impaired brain regions.

To achieve the goals of our field and fully harness the potential of neuroplasticity for cognitive benefit, we look forward to continued technological development, such as neuro-digital closed loops, and their integration with emerging design principles of cognitive training games. These technologies validated using randomized, controlled scientific evaluation methodologies will generate new understanding of how to translate cognitive neuroscience discoveries into new educational tools for healthy populations and mental healthcare interventions for neuropsychiatric populations in need of cognitive remediation.

## **ACKNOWLEDGMENTS**

This work was supported by the National Institute of Health grants 5R01AG040333 (Adam Gazzaley), 5R24TW007988-05 subaward VUMC38412 (Jyoti Mishra). We would like to thank Joaquin Anguera for his feedback on this opinion article.

## **REFERENCES**

EEGLAB, SIFT, NFT, BCILAB, and ERICA: new tools for advanced EEG processing. *Comput. Intell. Neurosci.* 2011, 130714. doi: 10.1155/2011/130714

in the brain. *Prog. Brain Res.* 207, 351–377. doi: 10.1016/B978-0-444-63327-9.00013-8

**Conflict of Interest Statement:** Jyoti Mishra is a part-time scientist at the Brain Plasticity Institute, PositScience, a company that develops cognitive training software. Adam Gazzaley is co-founder and chief science advisor of Akili Interactive Labs, a company that develops cognitive training software. Jyoti Mishra and Adam Gazzaley have a patent pending for "Methods of Suppressing Irrelevant Stimuli." Adam Gazzaley has a patent pending for a gamebased cognitive training intervention: "Enhancing cognition in the presence of distraction and/or interruption."

*Received: 30 December 2013; accepted: 27 March 2014; published online: 11 April 2014.*

*Citation: Mishra J and Gazzaley A (2014) Harnessing the neuroplastic potential of the human brain & the future of cognitive rehabilitation. Front. Hum. Neurosci. 8:218. doi: 10.3389/fnhum.2014.00218*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Mishra and Gazzaley. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Spaced cognitive training promotes training transfer

#### *Zuowei Wang1 \*, Renlai Zhou2,3 and Priti Shah1*

*1 Department of Psychology, Combined Program in Education and Psychology, University of Michigan, Ann Arbor, MI, USA*

*2 Beijing Key Laboratory of Experimental Psychology, School of Psychology, Beijing Normal University, Beijing, China*

*3 State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China*

#### *Edited by:*

*Guido P. H. Band, Leiden University, Netherlands*

### *Reviewed by:*

*Dietsje Jolles, Stanford University, USA Pier Prins, University of Amsterdam, Netherlands*

#### *\*Correspondence:*

*Zuowei Wang, Combined Program in Education and Psychology, University of Michigan, 610 E. University Ave., Room 1400-J, Ann Arbor, MI 48109, USA e-mail<EMAIL_ADDRESS>
Cognitive training studies yield wildly inconsistent results. One dimension on which studies vary is the scheduling of training sessions (Morrison and Chein, 2011). In this study, we systematically address whether or not spacing of practice influences training and transfer. We randomly assigned 115 fifth grade children to an active control group or one of four training groups who received working memory training based on a "running span" task (Zhao et al., 2011). All groups received the same total amount of training: 20 sessions of training with 60 trials for an average of 20 min per session. The training was spread across 2, 5, 10, or 20 days. The active control group received 20-min sessions of math instruction for 20 sessions. Before and after training participants in all five groups performed a single transfer test that assessed fluid intelligence, the Raven's Progressive Matrices Test. Overall, participants in all four training groups improved significantly on the training task (at least partially), as reflected by increased speed. More importantly, the only training group to show significant improvement on the Raven's was the group who had the greatest amount of spacing (20 days group) during training and improvement in this group was significantly higher than that of the control group.

**Keywords: cognitive training, working memory, transfer, schedule, spacing, primary school, children, fluid intelligence**

## **INTRODUCTION**

Working memory is the cognitive system that actively maintains and processes information for human problem solving (Miyake and Shah, 1999). Working memory training has been extensively studied in recent years (e.g., Klingberg, 2010; Morrison and Chein, 2011). Studies have shown that working memory training is beneficial to various subject populations, varying from young children to old adults, including healthy subjects as well as those with special needs. For example, working memory training has been found to reduce the symptoms of ADHD children (Klingberg et al., 2002, 2005), facilitate the recovery of cognitive functions in patients after stroke (Westerberg et al., 2007), enhance "old-old" adults' memory performance (80 years old; Buschkuehl et al., 2008), and improve fluid intelligence in pupils (Jaeggi et al., 2011; Zhao et al., 2011) as well as college students (Jaeggi et al., 2008, 2014).

The significance of working memory training is largely dependent on the potential transfer effects to other untrained situations. Due to the various transfer effects identified by previous studies, some researchers view working memory training as promising for general cognition enhancement (see the review by Morrison and Chein, 2011). However, while researchers almost always find performance improvement on trained tasks, not all studies find transfer. In fact, some researchers argue that there is little to no solid evidence that true far transfer effects arise from working memory training (Shipstead et al., 2012; Melby-Lervåg and Hulme, 2013; Rapport et al., 2013; Redick et al., 2013).

Efforts have been taken to investigate factors that affect successful training and transfer. Three broad classes of factors are likely relevant: individual characteristics of participants receiving training, the nature of the training task, and conditions of training. Individual characteristics that affect training and transfer may include initial ability of participants, the underlying source of any deficits in working memory performance, and motivational factors. Research in our laboratory has found, for example, that individuals who believe that intelligence is a malleable construct are more likely to benefit from training than those who believe intelligence is fixed (Jaeggi et al., 2014). Individuals who report that a training task is "too difficult" seem to disengage and show less improvement than individuals who enjoy the challenge of a task (Jaeggi et al., 2011). Other individual motivational factors may include the degree to which individuals are intrinsically motivated (Katz et al., in press; Shah et al., 2012).

The nature of the training task(s), of course, will also influence what types of transfer might be found. In a classic review of training and transfer, Schmidt and Bjork (1992) emphasize the importance of using a variety of tasks to increase the likelihood that training effects are not task-specific. Likewise, tasks that are adaptive and cognitively challenging may also limit the extent to which task-specific strategies that can be developed. In the original Jaeggi et al. (2008) study, the use of the dual n-back task that required maintaining and updating information in both visual and auditory modalities may have limited the ability of individuals to develop specific rehearsal strategies. More generally, Morrison and Chein (2011) found in their review of working memory training that studies that focus on training strategies were not as effective as those that focused on core working memory capacity. Some characteristics of the training task may also interact with motivation or perceived difficulty of a task. We found, for example, that in a brief, 3-day cognitive training intervention that some game-like features may in fact serve as a distraction making cognitive training more difficult (Katz et al., in press, this issue).

A third class of factors that affect the effectiveness of cognitive training is the dosage or sheer amount of training (Jaeggi et al., 2008) as well as the training schedule (Schmidt and Bjork, 1992). It has been known for over 100 years that individuals remember information better under spaced practice conditions compared to massed practice (Ebbinghaus, 1885). This spacing effect has been extensively studied under numerous contexts since then. Most of these studies showed that spaced learning led to better learning outcomes (for a review, see Cepeda et al., 2006). For example, spaced learning is beneficial to foreign word learning (Bahrick et al., 1993), inductive learning (Kornell and Bjork, 2008; Vlach et al., 2008), and professional training (Hagman, 1980). The spacing effect is also found in cognitive skill acquisition tasks for both human and animal subjects (Shebilske et al., 1994, 1999; Sisti et al., 2007). For example, Sisti et al. (2007) trained rats using different schedules on a water maze task and they found that the rats that received more spaced training outperformed rats who received massed training. More interestingly, performance was correlated with the survival of new neurons in the hippocampus, suggesting that spaced training elicited more neural changes.