# ADVANCEMENTS IN TECHNOLOGY-BASED ASSESSMENT: EMERGING ITEM FORMATS, TEST DESIGNS, AND DATA SOURCES

EDITED BY : Fank Goldhammer, Ronny Scherer and Samuel Greiff PUBLISHED IN : Frontiers in Psychology and Frontiers in Education

#### Frontiers eBook Copyright Statement

The copyright in the text of individual articles in this eBook is the property of their respective authors or their respective institutions or funders. The copyright in graphics and images within each article may be subject to copyright of other parties. In both cases this is subject to a license granted to Frontiers. The compilation of articles constituting this eBook is the property of Frontiers.

Each article within this eBook, and the eBook itself, are published under the most recent version of the Creative Commons CC-BY licence. The version current at the date of publication of this eBook is CC-BY 4.0. If the CC-BY licence is updated, the licence granted by Frontiers is automatically updated to the new version.

When exercising any right under the CC-BY licence, Frontiers must be attributed as the original publisher of the article or eBook, as applicable.

Authors have the responsibility of ensuring that any graphics or other materials which are the property of others may be included in the CC-BY licence, but this should be checked before relying on the CC-BY licence to reproduce those materials. Any copyright notices relating to those materials must be complied with.

Copyright and source acknowledgement notices may not be removed and must be displayed in any copy, derivative work or partial copy which includes the elements in question.

All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For further information please read Frontiers' Conditions for Website Use and Copyright Statement, and the applicable CC-BY licence.

ISSN 1664-8714 ISBN 978-2-88963-506-1 DOI 10.3389/978-2-88963-506-1

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office<EMAIL_ADDRESS>
# ADVANCEMENTS IN TECHNOLOGY-BASED ASSESSMENT: EMERGING ITEM FORMATS, TEST DESIGNS, AND DATA SOURCES

#### Topic Editors:

Frank Goldhammer, Leibniz Institute for Research and Information in Education (DIPF), Germany Ronny Scherer, Centre for Educational Measurement, Faculty of Educational Sciences, University of Oslo, Norway Samuel Greiff, University of Luxembourg, Luxembourg

Citation: Goldhammer, F., Scherer, R., Greiff, S., eds. (2020). Advancements in Technology-Based Assessment: Emerging Item Formats, Test Designs, and Data Sources. Lausanne: Frontiers Media SA. doi: 10.3389/978-2-88963-506-1

# Table of Contents

Pietro Cipresso, Elisa Pedroli, Silvia Serino, Michelle Semonella, Cosimo Tuena, Desirée Colombo, Federica Pallavicini and Giuseppe Riva

Andrea Horbach and Torsten Zesch

*56 Multiple-Choice Item Distractor Development Using Topic Modeling Approaches*

Jinnie Shin, Qi Guo and Mark J. Gierl

*70 The Expanded Evidence-Centered Design (e-ECD) for Learning and Assessment Systems: A Framework for Incorporating Learning Goals and Processes Within Assessment Design*

Meirav Arieli-Attali, Sue Ward, Jay Thomas, Benjamin Deonovic and Alina A. von Davier

Andreas Rausch, Kristina Kögler and Jürgen Seifried

*115 Evaluating Different Equating Setups in the Continuous Item Pool Calibration for Computerized Adaptive Testing*

Sebastian Born, Aron Fink, Christian Spoden and Andreas Frey

*129 Collaborative Problem Solving: Processing Actions, Time, and Performance*

Paul De Boeck and Kathleen Scalise

*138 Making the Psychological Dimension of Learning Visible: Using Technology-Based Assessment to Monitor Students' Cognitive Development*

Gyöngyvér Molnár and Benő Csapó

*154 The Skilled, the Knowledgeable, and the Motivated: Investigating the Strategic Allocation of Time on Task in a Computer-Based Assessment* Johannes Naumann

Vanessa R. Simmering, Lu Ou and Maria Bolsinova

*247 Combining Text Mining of Long Constructed Responses and Item-Based Measures: A Hybrid Test Design to Screen for Posttraumatic Stress Disorder (PTSD)*

Qiwei He, Bernard P. Veldkamp, Cees A. W. Glas and Stéphanie M. van den Berg

*259 Predictive Feature Generation and Selection Using Process Data From PISA Interactive Problem-Solving Items: An Application of Random Forests*

Zhuangzhuang Han, Qiwei He and Matthias von Davier

# Editorial: Advancements in Technology-Based Assessment: Emerging Item Formats, Test Designs, and Data Sources

#### Frank Goldhammer 1,2 \*, Ronny Scherer 3 and Samuel Greiff 4

*1 Educational Quality and Evaluation, DIPF - Leibniz Institute for Research and Information in Education, Frankfurt, Germany, 2 Centre for International Student Assessment (ZIB), Frankfurt, Germany, 3 Centre for Educational Measurement (CEMO), University of Oslo, Oslo, Norway, 4 Cognitive Science & Assessment, University of Luxembourg, Esch-sur-Alzette, Luxembourg*

Keywords: technology-based assessment, item design, test design, automatic scoring, process data, assessment of/for learning

#### **Editorial on the Research Topic**

#### **Advancements in Technology-Based Assessment: Emerging Item Formats, Test Designs, and Data Sources**

Technology has become an indispensable tool for educational and psychological assessment in today's world. Individual researchers and large-scale assessment programs alike are increasingly using digital technology (e.g., laptops, tablets, and smartphones) to collect behavioral data beyond the mere correctness of item responses. Along these lines, technology innovates and enhances assessments in terms of item and test design, methods of test delivery, data collection and analysis, and the reporting of test results.

The aim of this Research Topic is to present recent developments in technology-based assessment and in the advancements of knowledge associated with it. Our focus is on cognitive assessments, including the measurement of abilities, competences, knowledge, and skills, but also includes non-cognitive aspects of assessment (Rausch et al.; Simmering et al.). In the area of (cognitive) assessments, the innovations driven by technology are manifold, and the topics covered in this collection are, accordingly, wide and comprehensive: Digital assessments facilitate the creation of new types of stimuli and response formats that were out of reach for assessments using paper; for instance, interactive simulations may include multimedia elements, as well as virtual or augmented realities (Cipresso et al.; de-Juan-Ripoll et al.). These types of assessments also allow for the widening of the construct coverage in an assessment; for instance, through stimulating and making visible certain problem-solving strategies that represent new forms of problem solving (Han et al.; Kroeze et al.). Moreover, technology allows for the automated generation of items based on specific item models (Shin et al.). Such items can be assembled into tests in a more flexible way than what is possible in paper-and-pencil tests and can even be created on the fly; for instance, tailoring item difficulty to individual ability (adaptive testing) while assuring that multiple content constraints are met (Born et al.; Zhang et al.). As a requirement for adaptive testing, or to lower the burden of raters who code item responses manually, computers enable the automatic scoring of constructed responses; for instance, text responses can be coded automatically by using natural language processing and text mining (He et al.; Horbach and Zesch).

Technology-based assessments provide not only response data (e.g., correct vs. incorrect responses) but also process data (e.g., frequencies and sequences of test-taking strategies, including navigation behavior) that reflect the course of solving a test item and gives information on the

Edited and reviewed by:

*Yenchun Jim Wu, National Taiwan Normal University, Taiwan*

> \*Correspondence: *Frank Goldhammer<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Educational Psychology, a section of the journal Frontiers in Psychology*

Received: *19 December 2019* Accepted: *23 December 2019* Published: *20 January 2020*

#### Citation:

*Goldhammer F, Scherer R and Greiff S (2020) Editorial: Advancements in Technology-Based Assessment: Emerging Item Formats, Test Designs, and Data Sources. Front. Psychol. 10:3047. doi: 10.3389/fpsyg.2019.03047*

**5**

#### TABLE 1 | Overview of the papers.

*(Continued)*

#### TABLE 1 | Continued

path toward the solution (Han et al.). Process data, among others, have been used successfully to evaluate and explain data quality (Lindner et al.), to define process-oriented latent variables (De Boeck and Scalise), to improve measurement precision, and to address substantial research questions (Naumann). Large-scale result and process data also call for data-driven computational approaches in addition to traditional psychometrics and new concepts for storing and managing data (von Davier et al.).

The contributions of this Research Topic address how technology can further improve and enhance educational and psychological assessment from various perspectives. Regarding educational testing, not only is research presented on the assessment of learning, that is, the summative assessment of learning outcomes (Molnár and Csapó), but a number of studies on this topic also focus conceptually and empirically on the assessment for learning, that is, the formative assessment providing feedback to support the learning process (Arieli-Attali et al.; Blaauw et al.; Csapó and Molnár; den Ouden et al.; Kroeze et al.).

**Table 1** gives an overview of all the papers included in this Research Topic and summarizes them with respect to their key features. Reflecting the scope of the Research Topic, we used four major categories to classify the papers: (1) papers focusing on the use of new data types and sources, (2) innovative item designs, (3) innovative test designs, and (4) statistical approaches. We refrained from multiple category assignments of papers, which was possible, and focused on their core contribution. The papers' key findings and advancements impressively represent the current state-of-the-art in the field of technology-based assessment in (standardized) educational testing, and, as topic editors, we were happy to receive such a great collection of papers with various foci.

Regarding the future of technology-based assessment, we assume that inferences about the individual's or learner's knowledge, skills, or other attributes will increasingly be based on empirical (multimodal) data from less- or nonstandardized testing situations. Typical examples are stealth assessments in digital games (Shute and Ventura, 2013; Shute, 2015), digital learning environments (Nguyen et al., 2018), or online activities (Kosinski et al., 2013). Such new kinds of unobtrusive, continuous assessments will further extend the traditional assessment paradigm and enhance our understanding of what an item, a test, and the empirical evidence for inferring

### REFERENCES

attributes can be (Mislevy, 2019). Major challenges lie in the identification and synthesis of evidence from the situations the individual encounters in these non-standardized settings, as well as in validating the interpretation of derived measures. This Research Topic provides much input for these questions. We hope that you will enjoy reading the contributions as much as we did.

### AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

### ACKNOWLEDGMENTS

This work was funded by the Centre for International Student Assessment (ZIB) in Germany. We thank all authors who have contributed to this Research Topic and the reviewers for their valuable feedback on the manuscript.

Shute, V. J., and Ventura, M. (2013). Stealth Assessment: Measuring and Supporting Learning in Video Games. Cambridge, MA: MIT Press.

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Goldhammer, Scherer and Greiff. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Assessment of Unilateral Spatial Neglect Using a Free Mobile Application for Italian Clinicians

Pietro Cipresso1,2† , Elisa Pedroli1 \* † , Silvia Serino1,2, Michelle Semonella1 , Cosimo Tuena1 , Desirée Colombo3 , Federica Pallavicini4 and Giuseppe Riva1,2

1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy, 2 Department of Psychology, Università Cattolica del Sacro Cuore, Milan, Italy, 3 Department of Basic Psychology, Clinic and Psychobiology, Universitat Jaume I, Castellón de la Plana, Spain, 4 "Riccardo Massa" Department of Human Sciences for Education, University of Milano-Bicocca, Milan, Italy

#### Edited by:

Ronny Scherer, University of Oslo, Norway

#### Reviewed by:

Timothy R. Brick, Pennsylvania State University, United States Nelson Silva Filho, Universidade Estadual Paulista Júlio de Mesquita Filho (UNESP), Brazil

\*Correspondence:

Elisa Pedroli<EMAIL_ADDRESS>†These authors have contributed equally to this work

#### Specialty section:

This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology

Received: 30 April 2018 Accepted: 29 October 2018 Published: 22 November 2018

#### Citation:

Cipresso P, Pedroli E, Serino S, Semonella M, Tuena C, Colombo D, Pallavicini F and Riva G (2018) Assessment of Unilateral Spatial Neglect Using a Free Mobile Application for Italian Clinicians. Front. Psychol. 9:2241. doi: 10.3389/fpsyg.2018.02241 Background: Unilateral Spatial Neglect (USN) is traditionally assessed with paper-andpencil tests or computer-based tests. Thanks to the wide-spreading of mobile devices, and the extensive capabilities that they have in dealing complex elements, it is possible to provide clinicians with tools for cognitive assessment. Contemporary 3D engine is, in general generally, able to deploy complex 3D environments for iOS, Android and Windows mobile, i.e., most of the mobile phone and tablet operative systems.

Results: This brand-new scenario and pressing requests from professionals, pushed us to build an application for the assessment of USN. Our first attempt was to replicate the classic cognitive tests, traditionally used at this purpose. Ecological assessment is difficult in real scenarios so we implemented virtual environments to assess patients' abilities in realistic situations. At the moment, the application is available only for iPad and iPhone for free, from the Apple Store, under the name of "Neglect App." The App contains traditional tests (e.g., barrage with and without distractors) and ecological tests (e.g., to distribute the tea in a table to close people). Scoring of each test is available to the clinicians through a database with the executed ecological tasks, that are stored locally.

Conclusion: In conclusion, Neglect App is an advanced mobile platform for the assessment of Neglect.

Keywords: neglect, psychometrics, computational psychometrics, ecological assessment, mobile virtual reality, mHealth, pervasive computing, mental health

### INTRODUCTION

The Unilateral Spatial Neglect (USN) or Neglect manifests in about 2/3 of patients during the acute phase following a stroke. Stroke is an occurrence of cerebral vascular disease resulting in acute disruption of the focal or generalized brain function. Every year, there are approximately 500,000 stroke patients in Europe. This is the third leading cause of death in Western countries after cardiovascular diseases and malignancies (Sudlow and Warlow, 1997; Pendlebury et al., 2009; Roger et al., 2012; Go et al., 2014; Mozaffarian et al., 2015).

**9**

A stroke is a catastrophic and often unexpected event with a wide range of physical and psychological consequences in the long term for both patients and their families.

The long-term effects of stroke depend on the type, severity, and location of the occlusion: it is important to identify as soon as possible which part of the brain and how severely it has been affected. In general, two basic categories of impairments or disabilities can be identified: cognitive disability, which includes memory problems, difficulty in executive functions and aphasia, and motor disabilities, which includes the inability to walk and problems with coordination and balance (ataxia), mobility difficulties with arms, hemiparesis or hemiplegia, spasticity and contractures.

In particular USN can be defined as a disorder because the patient has difficulties to explore, pay attention, perceive, and act within the space opposite the region of the brain lesion. Often, there is also a difficulty in elaborating mental images in the opposite side of the damaged one. It is important to underline that the problems shown by patients are not caused by primary sensory or motor deficit, although they are often associated with hemiplegia and hemianopia (Ducros, 2012; Vocat et al., 2013; Heilman, 2014; Saj et al., 2014). These problems occur mainly following a damage to the right brain hemisphere, but there are patients in which the syndrome arose after a left-sided lesion; right neglect is considered less severe and less enduring (Stone et al., 1991; Halligan and Robertson, 2014). Regardless the side of the lesion, this disorder can be caused by the damage of several areas; the most typical one is the parietal lobe, specifically the inferior parietal lobule, followed by the frontal lobe and other sub-cortical structures such as the thalamus and the basal ganglia (Moretti et al., 2012; Saj et al., 2012; Antal et al., 2014).

In the acute phase and in the more severe form, the patient appears with the head and the gaze turned to the ipsilesional side, insensitive to any stimulation coming from the contralesional side. Over time, symptoms may ease, although more and more studies are showing that the disorder can last even for years (Kerkhoff and Schenk, 2012).

The neglect can be accompanied by several phenomena:

(Treccani et al., 2012; Antoniello and Gottesman, 2013; Marshall et al., 2013; Bartolomeo, 2014).

The standard neuropsychological tests for the analysis of extra personal neglect can be divided into:

What may be of great help to clinical placement is real exploration of space such as the room, where the patient is hospitalized to or the one where tests are conducted to have more complete picture of the patient's spatial abilities. Unfortunately, it is difficult to make these tests in a clinical setting because of the higher requested time and human resources. Finally, it is difficult to standardize these tests due to the heterogeneity of the experimental situations.

The purpose of this App was to include the described tests for a portable and electronic use, including also an automated score recording, that can also help in simplifying the difficult process of neuropsychological assessment. In one hand several tests have been included, as it is shown in the following sections. On the other hand, we made the effort of including new paradigms and tests that are difficult to be made in paper and pencil mode. In particular, navigation tasks and ecological tests represent our effort integrating current paradigms for neuropsychological assessment. At the moment, a plaint of possible features and indexes are probably still missing, however, the App represent the first effort ever in integrating many tests and tasks in a mobile application. This could be the first step toward future integrations.

### IMPLEMENTATION

Neglect App is the first application for mobile devices which makes use of the huge potential of virtual environments for the assessment of the USN for which an evaluation as effective and prompt as possible are crucial (Pallavicini et al., 2015; Pedroli et al., 2015a,b, 2016).

During the process of the design of the App, we also exploited the potential of 3D interactive applications for preventing and/or improving cognitive impairments related to USN, on the basis of a series of advantages amply documented by scientific literature:

### Neuroplasticity

Neuroplasticity: the App permits to use scenarios specifically designed following principles that regulate and facilitate neuroplasticity (the neurobiological process basis of recovery of cognitive and motor functions), such as exercise intensity, exercise frequency, "enriched stimulation" (Cheung et al., 2014; Ekman et al., 2018).

### Personalized Training

fpsyg-09-02241 November 20, 2018 Time: 19:27 # 3

Personalized training: the App is based on highly automated functioning mechanisms that requires a minimal contribution by the clinical therapists, who have the possibility to customize the intensity and the difficulty of the training based on the specific needs of the patients; Engaging tasks: in the App, the content of training exercises are based on defining some tasks to retrain specific abilities (for example, increasing complexity time by time), and in the same time integrating in the scenario some recreational elements to maintain a high level of engagement and compliance of the older participant. Specifically, ecological simulations can be particularly engaging by supporting a process known as "transformation of flow," defined as a person's ability to exploit an optimal (flow) experience to identify and use new and unexpected psychological resources as sources of involvement (Riva et al., 2006; Pedroli et al., 2018). Also, presence is a key point of the engagement in the use of technology. Presence is usually defined as the "sense of being there" or the "feeling of being in a world that exists outside the self." The ability to interact actively with the environment greatly improves the possibility of experiencing presence (Riva et al., 2007; Villani et al., 2012b).

### Tracking and Objective/Quantitative Measure

Tracking and objective/quantitative measure: it is possible to record a high quantity of data and use them to create some indexes of performance in order to measure in a quantitative and objective way the improvement of the performances observable in the course of possible rehabilitative process.

### Transferring of the Training in Activity of Daily Living (ADL)

Transferring of the training in activity of daily living (ADL): many studies suggested the potential offered by ecological tasks to transfer the results of re-learning of cognitive and motor abilities that were damaged in ADL. Positive impact of ecological tasks on ADL is documented by many studies (Laver et al., 2015; Chiang et al., 2017).

A previous pilot study investigating the correlation between Neglect App test and classic test in order to understand the usability and ecologicity of our app. Results showed that the cancelation tests of Neglect App were equally effective to the traditional tests in the screening of symptoms between patients with and without neglect. Moreover, the Neglect App Card Dealing task was more sensitive in detecting neglect symptoms than traditional functional task (Pallavicini et al., 2015).

Neglect App contains a series of trials for neglect evaluation through classic tests and virtually interactive environments with the double advantage of automating and making more ecological the evaluation of neglect patients, who thus show a difficulty and/or incapacity to explore, pay attention, perceive, and act in the space region opposite to the area of the brain lesion. Thanks to Neglect App, it is possible to evaluate the explorative behavior of the patient in a fast and simple way, inside ecological environments and receiving all the data, from the performed sessions, included in a database. Neglect App can be downloaded for free at: https://itunes.apple.com/it/app/neglectapp/id788480837?mt=8.

### RESULTS AND DISCUSSION

Evaluation is composed of nine exercises divided in two groups: ecological tasks and barrage. The first group comprises ecological tasks, some of which inspired by the ecological battery by Zoccolotti et al., 1994), some others created by starting from real life situations and tasks used clinically but lacking standardization. The app always provides all the score dividing the results in left, right, center, and total areas. Moreover a screenshot with the results is always recorded and generated in the report.

## Ecological Tasks

#### Serve Tea

The patient is required to distribute tea to himself and people sitting at the table with him using objects placed in the center of the table (**Figure 1**).

The task is commonly used by clinicians in real settings, however, the experience has been replicated in the tablet to be more usable, keeping its own ecological validity.

The patients can be used their finger to drag and drop the single objects for taking the task as requested by the App.

The App already contains the instructions that have to be followed, so the clinicians have just to give the tablet to the patient observing the correct use while executing the task.

Clinicians are not required to take note of the performed actions since the App is able to record every significant action consequently calculating the standard scores that can be used and integrated in a clinical protocol.

FIGURE 1 | Laying the table tapping on the iPad using the Neglect App.

The score is assigned on the basis of proper and wrong objects placed to the right, in the center and to the left. Time employed and unconsidered objects are also signaled (**Figure 2**).

In any time, clinicians are able to access to the patients' score directly from the App, visualizing each score in each task assigned at any time.

#### Card Dealing

The second exercise requires the patient to hand out playing cards to himself and people sitting at the table with him (**Figure 3**). The score is assigned on the basis of correctly given cards, omitted cards, wrong cards (i.e., those in excess) to the right, to the left and in the middle and the time employed to compete the exercise.

#### Controlling an Orders List

In this task, the patient is required to check an orders list to verify if the dishes noted herein are on the shelves; if they are, he/she will have to select the dish on the shelf and the note on the list (**Figure 4**).

Score is assigned on the basis of: the dishes selected correctly; those selected wrongly; the correct dishes omitted; the correct selections and omissions on the list; and the time taken (**Figure 5**).

#### Exploration

Within this environment the patient finds him/herself in a room in which he/she can move freely to left or right describing all the objects that are in the room and touching them accordingly (**Figure 6**). The app calculates automatically, as the patient

FIGURE 3 | Distributing cards tapping on the iPad using the Neglect App.

moves, if the selected object was on the right or the left. The report indicates selected objects on the left, the ones on the right, repetitions, time employed, and omitted elements.

#### Apples Pursuit

Within this environment the patient finds him/herself in an office in which he/she can move freely to left or right to identifying and touching all the apples inside (**Figure 7**). The app calculates automatically, as the patient moves, if the selected apple was on the right or the left. The report indicates selected apples on the left, the ones on the right, repetitions, time employed, and omitted apples.

### Barrage Tasks

Barrage tests take the cue from classical cancelation tasks commonly used clinically (Zoccolotti et al., 1994) and comprehend four exercises, described below.

#### Simple Barrage

Patient is required to select all objects (hammers) in the room. There are no distractors (**Figure 8**). The number of selected objects, repetitions, objects omitted on the left and on the right and time employed are considered (**Figure 9**).

#### Simple Barrage With Distractors

Patient is required to select all target objects (screwdrivers) in the room, which are mixed with distractors (**Figure 10**). The number

FIGURE 7 | Apples pursuit task.

FIGURE 8 | Simple barrage task.

of selected target objects, repetitions, target objects omitted and the distractors selected on the left and on the right and time employed are considered (**Figure 11**).

#### Dynamic Barrage

Patient is required to select all objects (balloons) in the sky. There are no distractors. The peculiarity here is that the objects are moving (**Figure 12**). The number of selected objects, repetitions, objects omitted on the left and on the right and time employed are considered (**Figure 13**).

#### Dynamic Barrage With Distractors

Patient is required to select all target objects (kites) in the room, which are mixed with distractors (**Figure 14**). The number of selected target objects, repetitions, target objects omitted and the distractors selected on the left and on the right and time employed are considered (**Figure 15**).

A qualitative analysis of the barrage tasks may give information about dysexecutive behaviors because it is possible

to select multiple times every single item and the target in the simple version of both barrage tasks (simple and dynamic) is in the environment of the barrage with distractions tasks.

### Data Management

FIGURE 10 | Simple barrage with distractors task.

All data can be downloaded in a unique file by connecting the iPad to a Computer or a Mac equipped with iTunes software. Once downloaded, the file can be easily read with a client

software able to interact with SQL Databases (**Figure 16**). All data, including images, are exportable to be computed for the statistical analysis.

FIGURE 14 | Dynamic barrage with distractors task (all the elements are in a continuous movement).

### CONCLUSION

Neglect may influence the behaviors of the patient in everyday life activity: they can constantly hit the objects placed on his left, not paying attention to the left side of the road when he crosses. In severe cases he can ignore the food in the left half of the plate. So, it has a sufficiently serious framework that allows the patient to cope independently.

The functions such as memory, speech, or attention in neuropsychological research were traditionally assessed through program of standardized tests, which have clear psychometric advantages, but often measure behaviors that are very different from those of everyday life (Chaytor and Schmitter-Edgecombe, 2003).

In recent years, there has been a growing interest in the development of tools that allows ecological and functional assessment above all by using mobile device (Villani et al., 2012a, 2013; Carbonaro et al., 2014; Pedroli et al., 2015b). The results of a meta-analytic review of Negu¸t et al. (2016) support the sensitivity of virtual reality tools in detecting cognitive deficit. One of the areas where emerges this need is the assessment of neglect. We decided to diffuse the application in the Italian market with a future intention to extend worldwide a possible English version. The Neglect App temporal cycle concern from the moment of the patients into the Clinique to the continuous assessment at the patient's home and back to the Clinique in a closed loop for the continuous assessment (**Figure 17**).

Assessment by using a mobile tool and virtual environments might represent a great challenge for very sophisticated methods able to assess in a way before unthinkable and sometimes impossible in real settings. In particular, navigation tasks allow the system to identifying if an object in the space is located in left

FIGURE 16 | Database management to brows and analyze the data collected.

or right side when selected. On the other hand, in real settings to do this navigation task is too expensive, requiring eye-tracking glasses. Moreover a computational approach can be easily used to provide more feedback to the patients and to model behaviors (Cipresso, 2015; Cipresso et al., 2015).

One of the limitations of the App is the screen dimension, that does not provide any direct advantage compared to paper and pencil test. Actually, this limitation has been recently overcome by the iPad Pro 12,900 that can be effectively used with our App, being totally compatible. Another limitation is the lack of normative data available for a quantitative analysis of the results; only a qualitative analysis is recommended. A future study could be able to fill this gap.

At the moment we have not implement some additional information and indexes that could help the clinicians to better understand the characteristics of their explorative behaviors in order to program a more personalized rehabilitations. In particular, it could be interesting to report the starting point and the path of the exploration made by patients or some other indexes like the ones reported in the Chung et al.'s (2016) article.

Additionally, to create some tasks for rehabilitations could make our application completer and more interesting. Provide some tasks for make exercises in a virtual environment could help patients and clinicians to improve clinical practice.

The future development will have directed to fill these limitations with the addition of some specific tasks both for assessment and rehabilitation. A manipulation of the cognitive complexity of the barrage tasks according to the criteria proposed by Ricci et al. (2016) and Sarri et al. (2009) could help to have a more precise assessment process. To aim this scope the introduction of a 3D version of line bisection task are also consider because some patients may show neglect symptoms in this kind of task and not in the barrage one.

Also, a new version developed to take advantage of the immersive technology could be designed in order to reach a higher degree of ecologicity.

After all these modifications a validation study will be necessary in order to prove the validity of our system. Also, a clinical trial for the rehabilitation session could be done in order to prove the usefulness of a computerizing protocol.

#### REFERENCES

Both, convergent and discriminant validity, need to be verified comparing current tools accordingly. At this purpose can be used current neuropsychological battery and specific test, such as barrage test, front assessment battery, real task (e.g., lay the table in real context), and so on.

We are so providing the scientific and clinical communities a free advanced tool able to be a practical and flexible way for the assessment directly in the patients' place but also a brand-new way for the assessment of Neglect.

#### Availability and Requirements

### AUTHOR CONTRIBUTIONS

PC and EP wrote the manuscript. PC, EP, SS, MS, CT, and DC collected the literature materials. GR supervised the study. PC, EP, SS, FP, and GR conceived the idea of the study, established the software requirements, and supervised the technological, clinical, and scientific aspects. All authors read and approved the final manuscript.

### FUNDING

This study was partially supported by the Italian funded project "VRehab. Virtual Reality in the Assessment and TeleRehabilitation of Parkinson's Disease and Post-Stroke Disabilities" – RF-2009-1472190.

the potentiality of mobile virtual reality. Technol. Health Care 23, 795–807. doi: 10.3233/THC-151039

cancer patients: a preliminary controlled study. Stud. Health Technol. Inform. 173, 524–528.

mapping. J. Neurol. Neurosur. Psychiatry 82, 862–868. doi: 10.1136/jnnp.2010. 224261

Zoccolotti, P., Pizzamiglio, L., Pittau, P., and Galati, G. (1994). Batteria di Test Per L'esame dell'Attenzione. Roma: Psytest.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Cipresso, Pedroli, Serino, Semonella, Tuena, Colombo, Pallavicini and Riva. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Virtual Reality as a New Approach for Risk Taking Assessment

Carla de-Juan-Ripoll\*, José L. Soler-Domínguez, Jaime Guixeres, Manuel Contero, Noemi Álvarez Gutiérrez and Mariano Alcañiz

Instituto de Investigación e Innovación en Bioingeniería (i3B), Universitat Politècnica de València, Valencia, Spain

Understanding how people behave when facing hazardous situations, how intrinsic and extrinsic factors influence the risk taking (RT) decision making process and to what extent it is possible to modify their reactions externally, are questions that have long interested academics and society in general. In the spheres, among others, of Occupational Safety and Health (OSH), the military, finance and sociology, this topic has multidisciplinary implications because we all constantly face RT situations. Researchers have hitherto assessed RT profiles by conducting questionnaires prior to and after the presentation of stimuli; however, this can lead to the production of biased, nonrealistic, RT profiles. This is due to the reflexive nature of choosing an answer in a questionnaire, which is remote from the reactive, emotional and impulsive decision making processes inherent to real, risky situations. One way to address this question is to exploit VR capabilities to generate immersive environments that recreate realistic seeming but simulated hazardous situations. We propose VR as the next-generation tool to study RT processes, taking advantage of the big four families of metrics which can provide objective assessment methods with high ecological validity: the realworld risks approach (high presence VR environments triggering real-world reactions), embodied interactions (more natural interactions eliciting more natural behaviors), stealth assessment (unnoticed real-time assessments offering efficient behavioral metrics) and physiological real-time measurement (physiological signals avoiding subjective bias). Additionally, VR can provide an invaluable tool, after the assessment phase, to train in skills related to RT due to its transferability to real-world situations.

Keywords: virtual reality, risk taking, occupational risks, risk attitude, risk perception, stealth assessment, psychophysiological assessment, embodiment

## INTRODUCTION

Each year, deficient Occupational Safety and Health (OSH) practices cause a global cost of approximately 2680 billion euros (Elsler et al., 2017). Although OSH training has shown positive impacts in the workplace, its effectiveness is below expectations (Robson et al., 2012). It has been demonstrated that the natural differences between individuals can appreciably influence this low effectiveness at several levels, cognitive, motivational and functional, among others (Motowildo et al., 1997). Risk propensity, defined as the "willingness to take risks" (MacCrimmon and Wehrung, 1990) and risk perception, defined as the individual's assessment of how risky a

#### Edited by:

Samuel Greiff, University of Luxembourg, Luxembourg

#### Reviewed by:

Pedro Gamito, Universidade Lusófona, Portugal Valerie Shute, Florida State University, United States

> \*Correspondence: Carla de-Juan-Ripoll<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology

Received: 01 September 2018 Accepted: 27 November 2018 Published: 12 December 2018

#### Citation:

de-Juan-Ripoll C, Soler-Domínguez JL, Guixeres J, Contero M, Álvarez Gutiérrez N and Alcañiz M (2018) Virtual Reality as a New Approach for Risk Taking Assessment. Front. Psychol. 9:2532. doi: 10.3389/fpsyg.2018.02532

**20**

situation is (Baird and Thomas, 1985), have been shown to have strong influence on risky decision making behaviors (Sitkin and Weingart, 1995). The measurement of risk taking (RT) attitudes is a recognized challenge for researchers and practitioners. Researchers have mostly employed self-report instruments to assess individual constructs based on theoretical psychological models (Brockhaus Sr, 1980; Ford et al., 1990; Gullone et al., 2000; Portell and Solé, 2001; Steinberg, 2004; Gardner and Steinberg, 2005; Sneddon et al., 2013; Rodríguez-Garzón et al., 2015). We have not found any one model that defines RT, thus its measurement requires further investigation. Lejuez et al. (2002) developed and validated a laboratory-based behavioral measure of RT (Balloon Analog Risk Task – BART). While this is a validated tool that has been used in several studies, we believe that it is desirable to develop a more ecological system to measure RT. VR provides the capability of creating interactive environments in which users can perform while their behavioral responses are recorded (Parsons, 2015). Accordingly, we propose that virtual environment based assessments are tools that can enhance the ecological validity of the evaluation of the responses evoked (Parsey and Schmitter-Edgecombe, 2013).

In this article we focus on the measurement of RT using physiological and behavioral metrics, with VR being employed as a tool to create immersive situations. We propose to use VR to assess RT attitudes under the paradigm of stealth assessment. VR can provide engaging virtual worlds which will allow real time measurement of RT behaviors.

This paper is comprised of four sections. In the first we review the theoretical framework of RT in the previous literature. In the second we summarize the extant instruments for the measurement of RT behaviors and discuss the current issues that make us believe that there is a need to establish a new approach. In the third we propose VR as a step forward in the assessment of RT. The fourth section briefly discusses the substantial implications raised by the article and our proposals for future research in this field.

### RESEARCH INTO RISK TAKING

RT research can be said to have started with the nuclear debate of the sixties. It was focused on risk acceptance and dealt with factors such as benefits and voluntariness. Since then, several more factors have been proposed for the explanation of RT: trust, trustworthiness and trust propensity (Colquitt et al., 2007); supportive supervision, job autonomy and communication quality (Parker et al., 2001); problem framing and outcome history (Sitkin and Weingart, 1995); expected utility (Kahneman and Tversky, 1986); genre (Byrnes et al., 1999) and boredom (Schroeter et al., 2014).

While these factors have been demonstrated to influence RT, individual differences constitute a key element in decision making processes (see **Figure 1**). According to Rundmo, 1996, a biased perception of risk – understood as the subjective evaluation of a risk - can lead to misjudgements of potentially hazardous risk sources. Therefore, if the subjective evaluation of a risk differs from the objective risk, this should be corrected (Risk Research Committee, 1980). Personality traits influence attitude toward risk, prompting risk seeking or risk aversion behaviors. This set of personal, innate, basic characteristics associated with risk were named Intrinsic Risk Attitude (IRA) by Schoemaker (1993) and have been shown to be consistent in various situations and contexts (Dohmen et al., 2011). Additionally, cognitive and affective states are also considered to be key influencers in the decision making process. We highlight mood and cognitive load as two main representative factors in this category. Mood has a strong influence on RT. People in a positive mood tend to focus on the benefits of a risky situation, much more so than those in neutral mood, making them more susceptible to undertake risky behaviors (Forgas, 1982, 1995; Forgas and Bower, 1987; Yuen and Lee, 2003). On the other hand, people in a negative mood overestimate risks and try to avoid potential loss and, therefore, think and act more carefully (Jorgensen, 1996). Cognitive load, the amount of mental activity involved in working memory, might also play a role in risk perception, since some kind of decisions, based on utilitarian judgments, require additional cognitive resources (Greene et al., 2008).

### RISK TAKING MEASURES: CURRENT ISSUES

RT measurement is a non-deterministic and non-standardized process based on different perspectives. Traditionally, most theories of human behavior are based on a model of the human mind that assumes that humans can think and verbalize accurately about their attitudes, emotions and behaviors (Simon, 1976; Brief, 1998). To date, most of the theoretical constructs used in RT assessment are based on explicit measures such as self-reports. However, recent advances in neuroscience have demonstrated that most of the brain processes that regulate our emotions, attitudes and behaviors are not conscious. That is, they are implicit processes that, in contrast to explicit processes, humans cannot verbalize (Barsade et al., 2009; George, 2009; Becker et al., 2011).

Several explicit measures of RT, oriented to evaluate attitude to risk, deferred risk perception or expected risk behavior, have been proposed in the last fifty years. Some authors have employed self-report measures based on questionnaires on compliance with safety practices in the workplace (Parker et al., 2001; Mohamed et al., 2009; Seo et al., 2015), attitude toward risk and organizational commitment (Kivimäki and Kalimo, 1993) and in studies into decision making (Sitkin and Weingart, 1995). On the other hand, some works have drawn on theoretical multidimensional models based on psychological constructs, such as personality (Lejuez et al., 2002; Skeel et al., 2007), impulsivity (Lejuez et al., 2002), sensation seeking (Horvath and Zuckerman, 1993; Lejuez et al., 2002) and situational awareness (Lejuez et al., 2002).

However, as in many other disciplines, pre- and postexperiment questionnaires have an important intrinsic bias

TABLE 1 | VR features and benefits of risk taking measurement.

since individuals' cognitive and psychological states will be different when they answer the questionnaires to when they actually underwent the experiences that the researchers wish to analyse (Kivikangas et al., 2011). As stated in (Wang et al., 2015), this tendency is primarily due to "social desirability effects," which can lead to untrue accounts of behavior, attitudes and beliefs (Paulhus, 1991). In addition, there may be different interpretations of specific self-report items, resulting in unreliability and poorer validity (Lanyon and Goodstein, 1997). Lastly, some self-reporting questions need people to possess overt knowledge of their dispositions (Schmitt, 1994) and this does not always run true.

To our knowledge, the BART (Lejuez et al., 2002) constitutes, to date, the only tool for RT measurement using implicit measures. The authors developed and validated a laboratorybased behavioral measure of risky behaviors. In this task, a balloon was presented in the middle of the screen. Subjects were asked to pump it as much as possible, knowing that it could exploit at any time. Participants were told that they would obtain a financial reward the more they could inflate the balloon without breaking it. Although the reliability of this tool has been retested (White et al., 2008), extensive investigations have demonstrated that the correspondence between performance in neuropsychological tests and real-life behaviors is very weak (Manchester et al., 2004; Sbordone, 2008; Bottari et al., 2009).

In the BART validation study, researchers employed measures of impulsivity, sensation seeking and behavioral constraint. We consider this a good basis to build on, since each of these constructs has been investigated independently and associated with RT. Firstly, impulsivity has been associated with RT in terms of drug use, drink driving and seatbelt use (de Wit, 2009; Stanford et al., 1996). Some authors have also demonstrated its connection with emotional self-control, inhibition and, especially, the management of frustrating situations (Cooper et al., 2000; Boyer, 2006). In addition, researchers have studied the relationship between the sensation seeking trait and RT in several domains, such as recreation, health, career, finance, safety and social life (Nicholson et al., 2005). Donohew et al. (1999) concluded that sensation seeking is an important factor in sexual RT. According to Tellegen's (1985), model behavioral constraint is one of the dimensions that composes personality. The behavioral constraint factor encompasses control, harm avoidance and traditionalism facets. In the same way, there is empirical evidence of the influence of personality traits on RT attitudes, in particular punishment avoidance (Paulus et al., 2003). We can find an interesting study from Wills et al. (2006) supporting this idea in the substance abuse field.

### LIMITATIONS OF CURRENT RISK TAKING MEASURES

As mentioned previously, to date the majority of RT assessment tools has been based on explicit measures and the use of questionnaires.

BART, with its multi-dimensional set of psycho-cognitive influences, represents the only alternative to explicit measures of RT behavior, but its design has some intrinsic limitations that current technologies could help to overcome.

In this regard, we believe that the existing measurement instruments do not reflect real situations, in which the subjects can perform as in real life, which leads to skewed results. In the laboratory the controlled stimuli given to subjects often do not include variables that are present in real life situations. Thus, the ecological validity of these methodologies, such as BART, is quite limited. Furthermore, these measurement tools do not involve any strong physical interaction, but require only simple actions, such as clicking a mouse, ignoring the

de-Juan-Ripoll et al. VR for Risk Taking Assessment

influence of the reactions of the rest of the body. In addition, when an individual is submitted to the currently available tests, (s)he is aware that (s)he is being assessed and can alter the outcomes; so we propose stealth assessment as a means of obtaining reliable results about real behaviors unnoticed by the subject. Lastly, we suggest that physiological processes must be considered as important measures of RT, as these measurements are uncontaminated by the participant's answering style, social desirability, interpretations of questionnaire item wording, the limits of his or her memory or by observer bias (Kivikangas et al., 2011). Thus, we propose an alternative measurement method which aims to advance in four specific aspects:

et al., 2012) that argues that taking a meaningful action enhances learning in comparison to passively perceiving that action. This idea has been strongly supported for decades by classical learning theorists such as Piaget and Cook (1952) and Vygotsky (1978). We propose to take advantage of the ideas underlying embodied learning theory and use high level cognitive experiences, involving sensing, acting and thinking, to measure and change attitudes in a deeper, more effective way.

### VIRTUAL REALITY AND RISK TAKING ASSESSMENT

Virtual Reality is a 3D synthetic environment able to simulate real experiences in which subjects can interact as if they were in the real world (Alcañiz et al., 2003). VR provides

greater immersion, fidelity and higher level of active user involvement than traditional methods of assessment and training (Hedberg and Alexander, 1994). In our view, VR constitutes a suitable tool for behavioral measurement, since it complies with the requirements (see **Table 1**) of the four specific aspects discussed in the previous section: (1) the real-world risks approach, (2) embodied learning, (3) stealth assessment and (4) physiological real-time measurement.

(1) According to Slater (2009), the result of immersion through technology is the psychological state of "being there," where the subject essentially forgets that (s)he is in a virtual reality setting. This produces a sense of presence and a "plausibility illusion" which evoke the perception that what is happening in the VR is actual and allows subjects to interact and behave as they might in real life. VR is being used increasingly for natural phenomena and social interactions simulation, since it has been demonstrated that neural mechanisms in humans when they are immersed in a virtual environment are similar to those in real life (Alcañiz et al., 2009). When we talk about training and learning, failure is a necessary ingredient. There is evidence that people who have faced real hazards have a more cautious attitude toward OSH (Cavalcanti and Soares, 2012). Hazards in real life can involve serious danger. This is why VR emerges as a potential medium for RT assessment and training, allowing users to operate, without risks, in a quasi-real environment (Amokrane et al., 2008). VR allows the exposure of a person to a risky situation and the activation of high fidelity cognitive processes and behaviors due to the plausibility of the immersion. (2) VR environments allow users to take part in an embodied learning experience, mainly through physical interactions (Kilteni et al., 2012). Going further with this concept (Dourish, 1999, unpublished), we consider a virtual interaction to be fully embodied when it is believable, in the sense of using our body coherently as we do in the real world. The dual-process theory of moral judgment, when it refers to moral dilemmas, makes a distinction between personal and impersonal dilemmas (Greene et al., 2001; Greene, 2009): personal dilemmas are conflicts in which the subject experiences the situation in the first person and actions are carried out physically – e.g., pushing. Conversely, impersonal dilemmas are seen from the outside, and the subjects do not take overt physical actions, but make only minor responses, such as pressing switches or levers. Based on this distinction, it has been demonstrated that when actions are based on the first person perspective and involve physical acts, the subjects tend to make more emotional decisions (Greene et al., 2001; Amit et al., 2014). (3) Stealth assessment can be also defined as a performancebased method, in which what is evaluated is latent (Rupp et al., 2010). Under this paradigm, embedding assessments in immersive virtual worlds is an innovative approach (Shute and Spector, 2008) that, in our view, is an improvement from the standpoint of ecological validity. (4) Regarding physiological real-time measurement, VR provides interactive and multimodal sensorial stimuli that provide unique advantages over other methodologies in neuroscientific investigation (Bohil et al., 2011). Thus, due to technological advances, researchers can now use accurate, affordable devices to obtain physiological measures which have been found to be more effective than self-reported measures as they (a) are not intrusive, (b) do no rely on participants' self-assessment of their emotional or cognitive experience, and (c) can detect changes in participants in real time. We have previous experience in combining VR technology with brain activity measures, and these results have shown that interactive virtual environments allow the measurement of emotional responses (Marín-Morales et al., 2018).

For these reasons, customizable, domain independent VR environments, in which individuals can, to a certain extent, act freely and react naturally to different risks or hazards, open to researchers an uncharted field of information about RT attitudes and behaviors. The set of these requirements may result in an application that includes a virtual environment, with a specific narrative that face the users with risky situations. This should be designed following stealth assessment methodology, and would allow physiological and behavioral measurement to provide information about individual decision making in the field of RT. We will show an example of how this tool might perform: the user could be in a virtual environment that consists in a path which (s)he must cover from start to finish, within the shortest possible time. Suddenly, (s)he meets a bifurcation, where (s)he has to choose whether a safe but log way – less risk, less potential benefit -, or a dangerous but short path – higher risk, higher potential benefit -. During this decision making process, we could take measures of galvanic skin response to assess emotional activation, and behavioral measures such as reaction time and the decision made by the user. As a result, we could obtain information about specific weight of emotional processes in RT, and its influence on behavior.

Our future research aims to study to what extent a VR tool is able to measure the cognitive and affective processes that influence RT. Furthermore, we would focus on how virtual interactions and narratives weight on the decision making process.

### CONCLUSION

RT measurement is a major challenge for companies and researchers. Investigations into behavioral measurement are at a turning point as, due to the potential of technological advances, we can generate virtual worlds to evaluate and, going further, train people in certain skills and competences. We suggest that virtual reality is the most appropriate medium for assessing attitudes to risk and risk perception, conditioning factors in the RT process, due to their immersive capabilities. We propose to undertake future investigations into realworld risks, embodied interactions, stealth assessment and physiological real-time measurement as differentiating elements in RT assessment. If we can study and measure the real, unbiased reactions of people facing risky or hazardous situations, it will be possible to create customized training programs to fit their individual characteristics. This can be expected to contribute to the improvement of OSH training programs, reducing work-related incidents and, consequently, costs for companies.

### AUTHOR CONTRIBUTIONS

fpsyg-09-02532 December 11, 2018 Time: 17:57 # 6

MA, CdJ, and NÁ were responsible for the general idea of the paper. CdJ and JS participated in drafting the work, while JG and MC revised it in-depth and provided new ideas thanks to their previous experience. MA supervised the entire work, revised the manuscript and approved the final version to be submitted. All

### REFERENCES

authors made substantial contributions to the conception and development of the work.

### FUNDING

This work was supported by the Spanish Ministry of Economy, Industry and Competitiveness funded projects "Advanced Therapeutic Tools for Mental Health" (DPI2016-77396-R), and "Assessment and Training on Decision Making in Risk Environments" (RTC-2017-6523-6) (MINECO/AEI/FEDER,UE).

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 de-Juan-Ripoll, Soler-Domínguez, Guixeres, Contero, Álvarez Gutiérrez and Alcañiz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Automated Feedback Can Improve Hypothesis Quality

Karel A. Kroeze1,2 \*, Stéphanie M. van den Berg2 , Ard W. Lazonder 3 , Bernard P. Veldkamp2 and Ton de Jong1

*1 Department of Instructional Technology, University of Twente, Enschede, Netherlands, 2 Department of Research Methodology, Measurement and Data Analysis, University of Twente, Enschede, Netherlands, 3 Behavioural Science Institute, Radboud University, Nijmegen, Netherlands*

Stating a hypothesis is one of the central processes in inquiry learning, and often forms the starting point of the inquiry process. We designed, implemented, and evaluated an automated parsing and feedback system that informed students about the quality of hypotheses they had created in an online tool, the hypothesis scratchpad. In two pilot studies in different domains ("supply and demand" from economics and "electrical circuits" from physics) we determined the parser's accuracy by comparing its judgments with those of human experts. A satisfactory to high accuracy was reached. In the main study (in the "electrical circuits" domain), students were assigned to one of two conditions: no feedback (control) and automated feedback. We found that the subset of students in the experimental condition who asked for automated feedback on their hypotheses were much more likely to create a syntactically correct hypothesis than students in either condition who did not ask for feedback.

#### Edited by:

*Samuel Greiff, University of Luxembourg, Luxembourg*

#### Reviewed by:

*Trude Nilsen, University of Oslo, Norway Jessica Andrews-Todd,*

*Educational Testing Service, United States*

\*Correspondence:

*Karel A. Kroeze<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Educational Psychology, a section of the journal Frontiers in Education*

Received: *14 September 2018* Accepted: *11 December 2018* Published: *04 January 2019*

#### Citation:

*Kroeze KA, van den Berg SM, Lazonder AW, Veldkamp BP and de Jong T (2019) Automated Feedback Can Improve Hypothesis Quality. Front. Educ. 3:116. doi: 10.3389/feduc.2018.00116* Keywords: automated feedback, hypotheses, inquiry learning, context-free grammars, online learning environment

### INTRODUCTION

Active forms of learning are seen as key to acquiring deep conceptual knowledge, especially in science domains (Hake, 1998; Freeman et al., 2014). One of the active forms of learning is inquiry learning. Inquiry learning has been defined in many different ways with as its kernel that the method starts from questions for which students need to find answers [see e.g., (Prince and Felder, 2007)]. In the current work, we focus on one of the ways inquiry is used in instruction, namely "learning science by doing science": students are expected to form and test hypotheses by performing experiments and analyzing data. In following an inquiry cycle, students learn both science content and the scientific method. In this study, we focus on the practice of the scientific method, and in particular on the creation of hypotheses.

Most models of inquiry-based learning encompass an orientation and conceptualization phase that enables students to familiarize themselves with the topic of investigation. Common activities during orientation are studying background information and conducting a few explorative experiments with the equipment at hand. The intended outcome of these initial explorations is the formation of theories and ideas, formalized in hypotheses (Pedaste et al., 2015). Hypotheses are integral to the inquiry cycle: they direct students' attention to specific aspects of the research problem and, hence, facilitate experimental design and data interpretation (Klahr and Dunbar, 1988; Zimmerman, 2007). In a classic study, Tschirgi (1980) found that both children and adults design more conclusive experiments when trying to test a hypothesis that contradicts prior

**28**

evidence. Hypothesis testing also increases the amount of domain knowledge students gain from an inquiry (Burns and Vollmeyer, 2002; Brod et al., 2018), which is probably due to the fact that hypotheses, regardless of their specificity and truth value, provide direction to students' inquiry process (Lazonder et al., 2009).

The importance of hypothesizing nevertheless stands in marked contrast with its occurrence in high school science classes. Research has consistently shown that inquiry is a complex process in which students make mistakes (Mulder et al., 2010). Specifically, students of all ages have problems in formulating hypotheses, particularly when they are unfamiliar with the topic of inquiry (Gijlers and de Jong, 2005; Mulder et al., 2010), and when experimental data is anomalous (Lazonder, 2014). As a consequence, few students generate hypotheses on their own account, and when they do, they often stick to a single hypothesis that is known to be true (i.e., confirmation bias) or formulate imprecise statements that cannot be tested in research. These natural tendencies demonstrate that unguided inquiry learning is likely to be ineffective (Mayer, 2004; Kirschner et al., 2006; de Jong and Lazonder, 2014). However, guided inquiry learning has been shown to compare favorably to both direct instruction (D'Angelo et al., 2014) and unguided inquiry learning (Furtak et al., 2012), and helps foster a deeper conceptual understanding (Alfieri et al., 2011).

Inspired by these positive findings we set out to design and evaluate a software scaffold that presented students with automatically generated feedback on the quality of their hypotheses.

### THEORETICAL FRAMEWORK

### Adaptive and Automated Scaffolding

Inquiry learning often takes place in virtual or remote laboratories and, to be successful, should be supplemented with guidance (de Jong and Lazonder, 2014). Furthermore, de Jong and Lazonder (2014) postulated that different types of students require different types of guidance. Recent work on differentiated guidance lends credence to this argument, finding a moderating effect of students' age (Lazonder and Harmsen, 2016) and prior knowledge (van Riesen et al., 2018) on learning activities and knowledge gains. Moreover, Furtak et al. (2012) showed teacherled inquiry activities to be more effective than student-led inquiry, implying that teachers are effective suppliers of guidance. However, given that teachers' time is an increasingly valuable resource, several adaptive software agents have recently been developed to support teachers on specific tasks and that adapt the guidance to students' characteristics. While Belland et al. (2016) found no added effect of limited adaptive scaffolding over static scaffolding, intelligent tutoring systems (Nye et al., 2014), adaptive environments (Durlach and Ray, 2011; Vandewaetere et al., 2011), and automated feedback (Gerard et al., 2015, 2016) have all shown promising results. The common-sense conclusion appears to be that the more guidance is adapted to the individual student, the better the guidance—and thus the student—performs. Indeed, Pedaste et al. (2015) recently identified the development of "virtual teacher assistants that analyse and respond to individual learners to create meaningful learning activities" as one of the main challenges in the field.

Although adaptive and automated elements are increasingly common in online learning environments (e.g., Aleven et al., 2010; Lukasenko et al., 2010; Vandewaetere et al., 2011; Gerard et al., 2015, 2016; Ryoo and Linn, 2016), they have typically been designed and implemented for a single learning activity in a specific domain. The reason for this is simple; even adaptive guidance for a single well-defined learning task generally requires years of research and development. Data must be gathered and coded, models have to be trained and fitted, appropriate feedback has to be fine-tuned and a digital environment has to be developed. Each of these steps involves the input of experts from different fields; teachers, statisticians, educational researchers, and computer scientists. As a result, scaffolds in multi-domain environments such as Go-Lab (de Jong et al., 2014) and WISE (Linn et al., 2003) generally do not adapt to the individual student, nor can they automatically assess products or provide context-sensitive feedback. The hypothesis scaffold we describe and test in this paper aims to fill this gap.

We have been unable to find any existing literature on the automated scoring of and feedback on free-text hypotheses. In contrast, a variety of increasingly sophisticated natural language processing (NLP) techniques have been employed for automated essay scoring. However, the techniques applied to scoring essays typically require a large amount of training data, and even when training data is available they are unlikely to provide the level of detail on the underlying structure of hypotheses required to give meaningful feedback. Training data is not readily available for hypotheses, and would be expensive to gather (Shermis and Burstein, 2013).

Anjewierden et al. (2015) noted that the "language" of hypotheses is a subset of natural language with a specific structure. They suggested using a domain-specific list of variables and categorical values (the lexicon), in conjunction with a grammar of hypotheses. Together, the lexicon and grammar could be used to create a hypothesis parses that is robust, and can be adapted to different domains with relative ease. The work reported here attempts to implement such a context-free grammar.

### Feedback

The informative tutoring feedback model [ITF, (Narciss, 2006, 2008)] distinguishes between internal feedback and external feedback, and a wide variety of feedback types. Internal feedback is provided by individual cognitive monitoring processes (Ifenthaler, 2011), external feedback can be provided by for example; teachers, peers, or automated scaffolds. Both types of feedback may conflict with or reinforce an internal reference value. Careful feedback design can help students regulate their learning process, particularly when internal and external feedback conflict (Narciss, 2008).

The function of feedback may be cognitive, meta-cognitive, or motivational, and a distinction can be made between simple (e.g., knowledge of performance, correct result) and elaborated (e.g., knowledge about task constraints, mistakes, and concepts) forms of feedback. These components broadly overlap with outcome, corrective and explanatory feedback types (e.g., Johnson and Priest, 2014). In a second-order meta-analysis on the effects of feedback, Hattie and Timperley (2007) prescribed that good feedback should set clear goals (feed up), inform the student of their progress (feed back), and provide steps to improve (feed forward). Finally, immediate feedback has been shown to give larger benefits than delayed feedback (Van der Kleij et al., 2015).

### Research Goal and Context

This project is performed in the Go-Lab ecosystem (de Jong et al., 2014). Go-Lab is an online environment where teachers and authors can share online and remote laboratories (Labs) and scaffolding applications (Apps). Apps and Labs can, together with multimedia material, be combined to create Inquiry Learning Spaces (ILS), which can also be shared on the Go-Lab environment. **Figure 1** shows a screenshot of a typical ILS. This ILS is organized in six phases that follow an inquiry cycle (in this case; Orientation, Conceptualization, Investigation, Interpretation, Conclusion, and Discussion), and can be navigated freely.

The hypothesis scratchpad app [**Figure 2**; (Bollen and Sikken, 2018)] is used to support students with hypothesis generation. This study aimed to create an adaptive version of the hypothesis scratchpad that can scaffold the individual student in hypothesizing in any domain, with a minimum of set-up time for teachers. This new version will need to (1) identify mistakes in students' hypotheses, and (2) provide students with appropriate feedback to correct these mistakes. If the app achieves both of these goals, it will be a considerable step toward "empowering science teachers using technology-enhanced scaffolding to improve inquiry learning" (Pedaste et al., 2015).

### DESIGN

For this project the hypothesis scratchpad currently available in Go-Lab has been extended. An automated feedback system was developed that can identify flaws in students' hypotheses and provide tailored feedback that enables students to correct their mistakes. The aim is to improve the quality of students' hypotheses.

The following sections will (1) describe the main components of hypotheses and the criteria used to assess them, (2) introduce the process of parsing hypotheses and applying criteria, (3) present the feedback given to students, and (4) formalize the outcome measures and statistical analyses used.

### Criteria

Quinn and George (1975) were the first to formally define a set of criteria for evaluating hypotheses: (1) it makes sense; (2) it is empirical, a (partial) scientific relation; (3) it is adequate, a scientific relation between at least two variables; (4) it is precise a qualified and/or quantified relation; and (5) it states a test, an explicit statement of a test. Subsequent research on hypothesis generation has broadly followed the same criteria, or a subset thereof. Van Joolingen and De Jong (1991, 1993) used a "syntax" and a "precision" measure, that correspond roughly with the "it makes sense" and "precise' criteria of Quinn and George. Mulder et al. (2010) used a "specificity" scale, using criteria comparable to those of Quinn and George.

Based on the criteria used by Quinn and George, and the measures used by Van Joolingen and de Jong, we developed a set of criteria that could be implemented in automated feedback. **Table 1** lists these criteria, providing a short explanation and examples from the electrical circuits domain for each criterion. In the automated feedback, the first two criteria are straightforward in that they rely on the presence of certain words. The remaining criteria are established using a context-free grammar parser, which is described in the next section.

### Parser

To detect mistakes, the automated system needs to interpret hypotheses on the criteria listed in **Table 1**. Given the observation that hypotheses are a relatively structured subset of natural language (Anjewierden et al., 2015), we can define a context-free grammar [CFG, (Chomsky, 1956)] that covers all well-structured hypotheses.

CFGs can be used to define natural languages, and are ideally suited to define heavily structured languages [e.g., programming languages, (Chomsky, 1956)]. A CFG is comprised of a set of production rules. All the sentences that can be produced by the repeated application of these rules are the formal language of that grammar.

The grammar that defines hypotheses looks something like the following1 ;

```
HYPOTHESIS -> if ACTION then ACTION
HYPOTHESIS -> ACTION if ACTION
ACTION -> VAR INTERACTOR VAR
ACTION -> VAR MODIFIER
ACTION -> MODIFIER VAR
ACTION -> ACTION and ACTION
VAR -> PROPERTY VAR
VAR -> bulbs
VAR -> voltage
VAR -> brightness
INTERACTOR -> is greater than
INTERACTOR -> is smaller than
INTERACTOR -> is equal to
MODIFIER -> increases
MODIFIER -> decreases
QUALIFIER -> series circuit
QUALIFIER -> parallel circuit
```
Each line is a production rule, the left-hand side of the rule can be replaced by the right-hand side. Uppercase words refer to further rules (they are non-terminal) and lowercase words refer to tokens (they are terminal). A token can be anything, but in our case, they are (sets of) words, e.g., "voltage" or "is greater than."

Consider the following hypothesis; "if the number of bulbs in a series circuit increases, the brightness of the bulbs decreases." If we were to apply our grammar, we can decompose this hypothesis

1For the complete grammar, see https://github.com/Karel-Kroeze/adaptivehypothesis-grammars.

as per **Figure 3**. Although this decomposition provides the structure of the hypothesis, it still does not contain the semantic information necessary to evaluate the criteria.

If we add semantic information to each of the tokens, and rules on how to unify this information to each of the production rules, we can extract all relevant information from the hypothesis (Knuth, 1968; Shieber, 2003). **Figure 4** shows an example of the final parse result2 which contains all the information needed to evaluate the criteria discussed.

2The parser was created using the Nearley.js package (Hardmath123., 2017), which implements the Earley context-free parsing algorithm (Earley, 1970). The source code of the parser is available on GitHub; https://github.com/Karel-Kroeze/ adaptive-hypothesis-utils/.

TABLE 1 | Scoring criteria.

### Feedback

The automated hypothesis scratchpad gives students the opportunity to request feedback. **Figure 5** shows an example of the automated hypothesis scratchpad, with the feedback button highlighted (the highlight is not part of the interface).

**Table 2** gives an overview of the feedback used. The feedback follows the guidelines set by Hattie and Timperley (2007) in that it informs students of their progress, is specific about the mistakes made, and—where relevant—suggests modes of improvement. The first three criteria from **Table 2** are required conditions; if a hypothesis does not have variables, a modifier or cannot be parsed, the other criteria are not shown. Conversely, if these criteria are met, feedback is presented only on the other relevant criteria.

Feedback was presented to the student in textual form in a pop-up window and was shown immediately after a student

requested it by clicking the feedback button. Feedback was never presented automatically. After receiving feedback, students could revise their hypothesis, and ask for feedback again. No explicit limits were placed on the amount of times students could ask for feedback.

### Measures

Three outcome measures are of interest; (1) do students use the feedback tool, (2) does the parser correctly classify mistakes, and (3) do students' hypotheses improve after receiving feedback.

All student actions within a Go-Lab inquiry learning space are logged to a database. Specifically, the history of all hypotheses is tracked, including requests for feedback, and the feedback received. Feedback counts can thus be readily determined from the log files. A snapshot of a hypothesis is made whenever a student asks for feedback, and of the final state of the hypothesis. The collection of snapshots for a hypothesis creates a "story" for that hypothesis, tracking it over time.

The validity of classifications made by the parser is evaluated by calculating an inter-rater reliability between the results of the parser and human coders. The human coders were instructed to code as a teacher, ignoring small mistakes in spelling and syntax if the intention of a hypothesis was clear. To train the human coders, a sample of snapshots was coded, and any disagreements were discussed. After reaching agreement, each coder independently coded the remaining snapshots. Agreement is calculated using Cohens' κ, and interpreted using rules of thumb Landis and Koch (1977) .

Each snapshot is given a score based on the number of criteria passed, resulting in a score in a 0−k range, where k is the number of criteria used (three in the first pilot, six in the second pilot and final experiment). Improvement of hypotheses is evaluated by comparing the score for a snapshot to the score for the previous snapshot. The quality of a hypothesis is the quality of the final snapshot of that hypothesis.

If feedback is effective, we expect to see that students who have feedback available create higher quality hypotheses, and that hypothesis quality increases after students ask for feedback: each consecutive snapshot should have a higher quality than the last.

During the study, it became apparent that the aggregate score does not follow a parametric distribution, and therefore could not be used as an outcome measure. The variables and modifier criteria were satisfied by almost all students in our samples. The syntax criterion was often indicative for success on the manipulation, CVS and qualified criteria. Thus, even though the variables, modifier and CVS criteria might be important from a science education perspective, the syntactically correct criterion was used as an indicator for hypothesis quality.

Multilevel logistic models (i.e., generalized linear mixed models) were used to account for the inherent group structure in the data, controlling for student and class effects where appropriate. The models used were comprised of two levels, students and classes. All reported effects are on the student level. To perform the models, we used R (R Core Team, 2018) and the package "lme4" (Bates et al., 2015). The scripts used in analyses are deposited along with the raw and generated datasets at DANS (Kroeze, 2018).

### FIELD STUDIES

Three field studies were conducted. An initial pilot study was conducted with an early version of the hypothesis parser to assess the feasibility of automated parsing of hypotheses using a context-free grammar. Following that, a second pilot study was conducted with the complete version of the parser to identify any remaining issues with the parser and ILS before moving on to the final experiment. The final experiment used a quasi-experimental design to assess the benefit of the tool in improving students' hypotheses. Each of these studies is described in more detail in the following sections.

### First Pilot Study

#### Participants

Four classes of 13- to 14-year-old secondary education students (n = 99), spread over three HAVO classes (preparing for a university of applied science, n = 76) and one VMBO class (preparing for vocational education, n = 23) at a local high school participated in the pilot. Students had already studied the subject matter (supply and demand) as part of their regular curriculum and had previously participated in studies using Go-Lab ILSs and a version of the hypothesis scratchpad that did not provide feedback.

#### Materials and Procedure

The pilot revolved around a short ILS set in the supply & demand domain, where students were introduced to the interactions between price, supply, and demand. The ILS was created in collaboration with a participating economics teacher. Each class performed the study in a single 50-min session. At the beginning of a session, students were given an oral introduction detailing how to use the environment and refreshing them on what a hypothesis is. They were then asked to open the inquiry learning space, where they were first presented with information on the domain. They were then asked to create as many hypotheses about this domain as possible in the automated hypothesis scratchpad, and to use the feedback mechanism when they were stuck or wanted to check their hypothesis. An initial version of the parser was used that could detect the first three criteria: it has two variables, it has a modifier, and it is a syntactically correct sentence. Students were regularly encouraged and reminded to create as many hypotheses as

#### TABLE 2 | Feedback for each criterion.

*[HYPOTHESIS], [INDEPENDENT], [DEPENDENT], and [VARIABLE] will be dynamically replaced with the actual hypothesis and variables used by the student and recognized by the parser. The feedback has been translated from the Dutch original used in the experiments.*

*aUsed when a hypothesis starts valid but is incomplete (partial parse).*

*bUsed when a hypothesis cannot be parsed (nonsense, or syntax error).*

*cUsed when the independent variable is not manipulated.*

*dUsed when multiple independent variables are manipulated.*

possible3 , but no attempt was made to force the creation of hypotheses or the use of the feedback tool. The session was concluded with a small user satisfaction questionnaire. During each session, the researcher and the classroom teacher monitored the class, answering process-related questions, and eliciting feedback if any out of the ordinary situations or interactions were encountered.

#### Results

A total of 979 hypotheses were collected from 96 students. Most students created three to five hypotheses and asked for feedback multiple times over the course of the experiment. One student asked for feedback 84 times and was removed as an outlier.

Inter-rater reliability between the parser and two human experts was almost perfect on all three criteria (Cohen's κ = 0.81 − 0.96), showing high parser accuracy. Hypotheses for which students requested and received feedback at least once were more likely to be correct on all criteria. This relation is visible in **Figure 6**, and statistically significant using a multilevel logistic model estimating the probability of a syntactically correct

3Unfortunately, during one of the HAVO sessions the teacher instructed students to create 'at least 4' hypotheses, which was immediately interpreted as 'create 4 hypotheses'.

hypothesis by the number of feedback requests, corrected for student and class effects, gender, and age (βfeedbackCount = 1.00, SEβ = 0.17, CIOR = 1.93 − 3.83, p < 0.001), where βfeedbackCount is the effect of each additional feedback request, and CIOR the confidence interval of the Odds Ratio.

#### Discussion

The first pilot took place under test conditions; students were told to create as many hypotheses as possible, and the learning space was only there to provide a setting for hypotheses to be created. Such conditions are different from usual educational practice. Nevertheless, high parser accuracy and significantly increased quality of hypotheses showed that a parser is feasible, and that a hypothesis scratchpad enhanced with automated scoring and feedback is promising.

Therefore, a second pilot study was conducted using an expanded version of the context-free grammar that included all criteria listed in **Table 1**. In addition, the automated scratchpad was embedded in a full ILS, aligning much closer to how the tool is likely to be used in practice.

## Second Pilot Study

#### Participants

Participants came from one HAVO class of 13 to 14-year-old secondary educations students (n = 27), at a local high school. The students had recently been introduced to electrical circuits as part of their regular curriculum but were familiar with neither Go-Lab environments nor the hypothesis scratchpad prior to the experiment.

#### Materials and Procedure

A short ILS in the electrical circuits domain that could be completed in a single 50-min session was created in collaboration with participating teachers. At the beginning of a session, students were given an oral introduction detailing how to use the tools in the ILS and refreshing them on what a hypothesis is. They were then asked to open the ILS, where they were presented with a short pre-test, followed by some information on the domain. To guide students' hypothesis construction, they were asked to enter two predictions about the change in brightness of lightbulbs in series and parallel circuits after adding another bulb. In the next steps, students were asked to turn these predictions into hypotheses in the automated hypothesis scratchpad, and design an experiment in the Experiment Design app [see e.g., (van Riesen et al., 2018)] to test their hypotheses. Finally, students were given time to create an experimental setup in the Circuit Lab virtual laboratory, test their hypotheses, and enter their conclusions.

All student actions took place in the ILS, which encompassed a full inquiry cycle, from orientation to conclusion. This created an environment more likely to occur in real educational settings. An expanded version of the automated hypothesis scratchpad was used, designed to be able to classify and give feedback to all the relevant criteria.

During the session, the researcher and the classroom teacher monitored the class, answering process-related questions and eliciting feedback if any out of the ordinary situations or interactions were encountered.

#### Results

Both the researcher and the classroom teacher noticed that students had problems working with the ILS and staying on-task. These problems were process related (e.g., students got distracted, skipped steps) and tool related (i.e., students did not know how to work with the tool). Attempts to provide instructions during the experiment were largely ineffective because students were at different stages of the ILS (making group instructions difficult), and there were too many students to provide individual instructions.

In addition, some of the written instructions were too long. For example, upon seeing the instructions, one student immediately uttered: "too long, won't read." It seems likely that his sentiments were shared by other students, highlighting the need for verbal (or at least more interactive) instructions.

A total of 50 hypotheses were collected from 27 students. The plurality (13) of students created two hypotheses each, 7 students did not create any hypotheses. Most (16) students asked for feedback at least once, 11 students did not ask for feedback. One student asked for feedback 23 times and was removed as an outlier.

Parser accuracy was below expectations, achieving a Cohens' κ of 0.91, 0.90, and 0.40 on the contains at least two variables, contains a modifier, and is a syntactically correct sentence criterion, respectively. Accuracy for the manipulates exactly one variable and is qualified criteria is not reported, as the parser failed to recognize 30 out of 46 syntactically correct snapshots, leaving only 16 parsed snapshots.

Although there does appear to be a positive effect of feedback on hypothesis quality (see **Figure 7**), this effect was not statistically significant, as shown by a multilevel logistic model estimating the probability of a syntactically correct hypothesis by the number of feedback requests, correcting for student effects, gender and age (βfeedbackCount = 0.46, SEβ = 0.24, CIOR = 0.98 − 2.57, p = .058).

#### Discussion

The number of collected hypotheses per student was lower than in the first pilot. In part, that was by design: the first pilot was specifically set up to encourage students to create as many hypotheses as possible, whereas, in this pilot students were guided to create two hypotheses. The participants in this pilot also had less experience working in an ILS, which caused several processrelated issues during the session that likely influenced the number of hypotheses created. A more structured lesson plan where students start and end each step in the inquiry cycle at the same time will allow for verbal instructions to be given before starting each section.

Many students failed to distinguish between series and parallel circuits in their hypotheses, even when their predictions did show they understood the differences between the types of circuits. This does seem to indicate the need for supporting the creation of hypotheses while at the same time highlighting that the currently implemented support is insufficient.

Poor parser accuracy can be attributed to students' difficulties in working with the ILS, additional criteria introducing more complexity to the grammar, and a lack of training data for the Electrical Circuits domain in the target language (Dutch) to calibrate the parser. Using the data gathered in the pilot, we were able to make improvements to the grammar used by the parser. When applying this new grammar to the gathered hypotheses, inter-rater agreement on the syntax criterion was raised to moderate (Cohens' κ = 0.53).

## Main Study

#### Participants

Six classes of 13- to 15-year-old secondary education students (n = 132), from two local high schools participated in the study. Six students used incorrect login credentials and were left out of the analyses. The remaining participants came from 4 HAVO classes (n = 78), and 2 VWO classes (n = 48). Students were randomly assigned to one of two conditions. Students in the experimental condition (n = 68) used the automated hypothesis scratchpad, while those in the control condition (n = 58) used a version of the hypothesis scratchpad that did not provide feedback. No significant differences were present in the distribution of age, gender, and current physics grade across conditions (**Table 3**).

#### Materials and procedure

A single 50-min session was used, covering the same material as that of the second pilot study. The ILS used in the second pilot study was used again, with some minor changes to ameliorate some of the process-related issues students encountered. In particular, written descriptions and instructions were shortened. Instead, at the outset of the session and each phase, students were given a short oral introduction.

Students received a link to a randomizer4 that assigned each student to one of two conditions and redirected them to the corresponding ILS. Students were instructed not to move to the next phase until told to do so.

At pre-set intervals during the sessions, the researcher gave an oral introduction to the next phase of the inquiry cycle, and the corresponding tools in the ILS. Students where then encouraged to start with that phase. In each session, the researcher and the class teacher monitored the students, answering process-related questions, and eliciting feedback if any extra-ordinary situations or interactions were encountered.

#### Results

Most students were already familiar with the GoLab environment and its tools and encountered no significant difficulties. Based on observations during the sessions, oral introductions prior to each phase of the ILS appeared to keep most students on task, most of the time.

Students in the experimental condition created 201 hypotheses, for 56 of which feedback was requested. Of the 68 students in the experimental condition, exactly half never asked for feedback.

Parser accuracy was moderate to almost perfect, achieving a Cohens' κ of 0.84, 0.70, and 0.59 on the contains at least two variables, contains a modifier, and is a syntactically correct sentence criterion, respectively, and > 0.80 for the manipulates exactly one variable and is qualified criteria.

**Figure 8** appears to show that on average the hypotheses generated in the experimental condition scored higher on all criteria. In addition, **Figure 9** suggests a positive relation between the number of feedback requests and the quality of hypotheses. In particular, hypotheses for which feedback was requested at least once appear to be of higher quality.

To test the effect of our tool on hypothesis quality, we fitted a multilevel logistic model, controlling for student and class effects, as well as gender, age, physics grade, and academic level. We found no significant effect from being assigned to the experimental condition (βcondition = 0.25, SEβ = 0.34, CIOR = 0.66 − 2.50, p = 0.472). Given that half of all participants in the experimental group never requested feedback, this outcome was not unexpected.

However, when we split the experimental group in two, based on whether students requested feedback or not (n = 34 in both groups, **Figure 10**), and contrast those who requested feedback against those who did not or could not, controlling for student and class effects, as well as gender, age, physics grade and academic level, the effect of requesting feedback is significant (βfeedbackCount = 1.47, SEβ = 0.42, CIOR = 1.92 − 9.89, p < 0.001).

It could be argued that students who did not request feedback when it was made available to them are less proficient students. However, a contrast analysis comparing students in the control condition (who could not ask for feedback) and those in the experimental condition who did not request feedback found no significant difference between the two groups on the syntactically correct criterion (βcondition = −0.30, SE = 0.39, CIOR = 0.34 − 1.60, p = 0.445). We thus found no evidence to suggest that there was a difference between students who could have asked for feedback but did not do so, and students who did not have the option to ask for feedback.

#### GENERAL DISCUSSION

The creation of hypotheses is a critical step in the inquiry cycle (Zimmerman, 2007), yet students of all ages experience difficulties creating informative hypotheses (Mulder et al., 2010). Automated scaffolds can help students create informative hypotheses, but their implementation in the regular curriculum is often cost-prohibitive, especially since they can typically only be used in one specific domain and language. This study set out to create a hypothesis scratchpad that can automatically evaluate and score hypotheses and provide students with immediate feedback. We use a flexible Context-Free Grammar approach that can relatively easily be adapted and extended for other languages and domains. We described the development process of this tool over two pilot studies and evaluated its instructional effectiveness in a controlled experiment.

Across three studies, we showed that a hypothesis parser based on a context-free-grammar is feasible, attaining moderate to almost perfect levels of agreement with human coders. The required complexity of the parser is directly linked to the syntactical complexity of the domain. For example, the electrical circuits domain requires a more complex parser than the supply and demand domain. Further development of the context-freegrammar used in the parser will contribute to higher reliability and may extend it to other languages and domains.

The second pilot study illustrated that a lack of familiarity of students with the online environment and the tools used can have a negative effect on their performance. Students were distracted by technical and process related issues, and had difficulty remaining on-task. In the final experiment, we used a largely identical learning environment, but students were verbally introduced to each phase. These introductions allowed students to focus on the content of the learning environment, rather than on how to use the learning environment itself.

Nevertheless, when using the automated hypothesis scratchpad in a "typical" ILS, students often did not request

4A separate ILS was created for each condition. The randomizer forwarded the students browser to one of these conditions. Randomization was weighted to ensure a roughly equal distribution across conditions in each session.

#### TABLE 3 | Participant characteristics, by condition.

feedback. Timmers et al. (2015) found a relation between gender and the willingness to ask for feedback, but such a relation was not present in our sample. In fact, none of the background variables collected (age, gender, physics grade and educational level) were significantly related to feedback requests or the quality of hypotheses.

If the goal was to obtain as many hypotheses as possible and assess the performance of the parser alone, we would have been better off following the approach taken in the first pilot. However, we deliberately chose to embed the automated hypothesis scratchpad in a typical ILS in the second pilot and main study, with the aim of replicating "real-world" conditions. In doing so, we can draw conclusions that are likely to be applicable to educational practice, rather than in laboratory conditions alone.

In the first pilot, the number of feedback requests was significantly related to the quality of hypotheses. This result was confirmed in a controlled experiment, where students who requested feedback were significantly more likely to create syntactically valid hypotheses than those who did not. The effects of feedback were immediate; hypotheses for which feedback was requested once where more likely to be correct.

To the best of our knowledge, no other tool exists that can reliably score hypotheses, can easily be adapted to different domains, and that allows students to create free-text hypotheses. The automated hypothesis scratchpad we present here can provide a clear and immediate benefit in science learning, provided students request feedback. By increasing the quality of students' hypotheses, we may assume that students are able to engage in more targeted inquiries, positively impacting their learning outcomes. How students can best be encouraged to request (and use) feedback is an open problem, and out of scope for this project. The automated hypothesis scratchpad could also be adapted to be a monitoring tool, highlighting students that may have difficulties creating hypotheses, allowing teachers to intervene directly.

The ability to reliably score hypotheses presents possibilities besides giving feedback. For example, hypothesis scores could serve as an indicator of inquiry skill. As such, they can be part of student models in adaptive inquiry learning environments. Crucially, obtaining an estimate from students' inquiry products

is less obtrusive than doing so with a pre-test, and likely to be more reliable than estimates obtained from students' inquiry processes.

The aggregate hypothesis score computed for students did not have a known parametric distribution. This represents a serious limitation, as the score could not be used in statistical analyses. As a result, we chose to only test statistical significance based on the syntax criterion. Investigating alternative modeling techniques to arrive at a statistically valid conclusion based on multiple interdependent criteria will be part of our future work.

An automated hypothesis scratchpad providing students with immediate feedback on the quality of their hypotheses was implemented using context-free grammars. The automated scratchpad was shown to be effective; students who used its feedback function created better hypotheses than those who did not. The use of context-free grammars makes it relatively straightforward to separate the basic syntax of hypotheses, language specific constructs, and domain specific implementations. This separation allows for the quick adaptation of the tool to new languages and domains, allowing configuration by teachers, and inclusion in a broad range of inquiry environments.

### ETHICAL STATEMENT

All participating schools have obtained written and informed consent from students' parents to perform research activities that fall within the regular curriculum. Parents were not asked to give consent for this study specifically. The experiments we performed were embedded in the students' curriculum, and the collected data was limited to learning processes and outcomes. Students were briefed that their activities in the online learning environment would be logged, and

### REFERENCES

that this data would be used in anonymized form. Both the research protocol and consent procedures followed were approved by the ethical board of the faculty of Behavioural, Management and Social Sciences of the University of Twente (ref # 17029).

### AUTHOR CONTRIBUTIONS

KK, TdJ, AL, and SvdB designed the experiment. KK and AL designed the intervention. KK, SvdB, and BV performed statistical analyses, TdJ and AL helped put experimental results into context. KK wrote the manuscript, aided by TdJ, SvdB, AL, and BV.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Kroeze, van den Berg, Lazonder, Veldkamp and de Jong. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Influence of Variance in Learner Answers on Automatic Content Scoring

Andrea Horbach\* and Torsten Zesch

Language Technology Lab, University Duisburg-Essen, Duisburg, Germany

Automatic content scoring is an important application in the area of automatic educational assessment. Short texts written by learners are scored based on their content while spelling and grammar mistakes are usually ignored. The difficulty of automatically scoring such texts varies according to the variance within the learner answers. In this paper, we first discuss factors that influence variance in learner answers, so that practitioners can better estimate if automatic scoring might be applicable to their usage scenario. We then compare the two main paradigms in content scoring: (i) similarity-based and (ii) instance-based methods, and discuss how well they can deal with each of the variance-inducing factors described before.

#### Edited by:

Ronny Scherer, Department of Teacher Education and School Research, Faculty of Educational Sciences, University of Oslo, Norway

#### Reviewed by:

Mark Gierl, University of Alberta, Canada Dirk Ifenthaler, Universität Mannheim, Germany

> \*Correspondence: Andrea Horbach<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Educational Psychology, a section of the journal Frontiers in Education

Received: 07 December 2018 Accepted: 12 March 2019 Published: 04 April 2019

#### Citation:

Horbach A and Zesch T (2019) The Influence of Variance in Learner Answers on Automatic Content Scoring. Front. Educ. 4:28. doi: 10.3389/feduc.2019.00028 Keywords: automatic content scoring, short-answer questions, natural language processing, linguistic variance, machine learning

## 1. INTRODUCTION

Automatic content scoring is a task from the field of educational natural language processing (NLP). In this task, a free-text answer written by students should be automatically assigned a score or correctness label in the same way as a human teacher would do. Content scoring tasks have been a popular exercise type for a variety of subjects and educational scenarios, such as listening or reading comprehension (in language learning) or definition questions (in science education). In a traditional classroom-setting, answers to such exercises are manually scored by a teacher, but in recent years, their automatic scoring has received growing attention as well (for an overview, see e.g., Ziai et al., 2012 and Burrows et al. (2014)). Automatic content scoring may decrease the manual scoring workload (Burstein et al., 2001) as well as offer more consistency in scoring (Haley et al., 2007). Additionally, automatic scoring provides the advantage that evaluation can happen in the absence of a teacher so that students may receive feedback immediately without having to wait for human scoring. With the increasing popularity of MOOCS and other online learning platforms, automatic scoring has become a topic of growing importance for educators in general.

In this paper, we restrict ourselves to short-answer questions as one instance of free-form assessment. While other test types, such as multiple choice items, are much easier to score, free-text items have an advantage from a testing perspective. They require active formulation instead of just selecting the correct answer from a set of alternatives, i.e., they test production instead of recognition.

Answers to short-answer questions have a typical length between a single phrase and two to three sentences. This places them in length between gap-filling exercises, which often ask for single words, and essays, where learners write longer texts. We do not cover automatic essay scoring in this article, even if it is related to short-answer scoring, and to some extent even the same methods might be applied. The main reason is that scoring essays usually takes into consideration the form

**42**

of the essay (style, grammar, spelling, etc.) in addition to content (Burstein et al., 2013), which introduces many additional factors of influence that are beyond our scope.

**Figure 1** shows examples from three different content scoring datasets (ASAP, POWERGRADING and SEMEVAL) and highlights the main components of a content scoring scenario: a prompt, a set of learner answers with scoring labels, and (one or several) reference answers.

Numeric or binary scoring labels, as we see in ASAP and POWERGRADING, can be easily summed up and compared. They are thus often used in summative feedback, where the goal is to inform teachers, e.g., about the performance of students in a homework assignment. For formative feedback, which is directed toward the learner, in contrast, a more informative categorical label might be preferable, e.g., to inform a student of their learning progress. The SEMEVAL data is an example for scoring labels aiming into that direction.

• In addition to learner answers, datasets often include teacherspecified **reference answers** for each label. A reference answer showcases a representative answer for a given score and can be used for (human or automatic) comparison with a learner answer. Alternatively, scoring guidelines describing properties of answers with a certain score can be provided. This is often the case when answers are so complex that just providing a small number of reference answers does not nearly cover the conceptual range of possible correct answers and misconceptions. This is for example the case for the ASAP dataset. When reference answers are given, many datasets only provide reference answers for correct answers and not for incorrect ones, e.g., POWERGRADING and SEMEVAL.

The content scoring scenario with its interrelated textual components – a prompt, learner answers, and a reference answer – render automatic content scoring a challenging application of Natural Language Processing which bears strong resemblances to various core NLP fields like paraphrasing (Bhagat and Hovy, 2013), textual entailment (Dagan et al., 2013), and textual similarity (Bär et al., 2012). In all those fields, the semantic relation between two texts is assessed, a method that directly transfers to the comparison between learner and reference answers, as we will see later.

During recent years, many approaches for automatic content scoring have been published on various datasets (see Burrows et al. (2014) for an overview). A practitioner who is considering using automatic scoring for their own educational data might easily feel overwhelmed. They might find it hard to compare approaches and draw conclusions for their applicability on their specific scoring scenario. In particular, approaches often apply various machine learning methods with a variety of features and are trained and evaluated using different datasets. Thus, comparing any two approaches from the literature can be difficult.

This paper aims to shed light on the individual factors influencing automatic content scoring and identifies the variance in the answers as one key factor that makes scoring difficult. We start in section 2 by discussing the nature of this variance, followed by a discussion of datasets and their parameters that influence variance. We discuss in section 3 properties of automatic scoring methods and review existing approaches, especially with respect to whether they score answers based on features extracted from the answers themselves or based on a comparison with a reference answer. We then discuss in section 4 how these factors can be isolated in scoring experiments. We either provide own experiments, discuss relevant studies from the literature, or formulate requirements for datasets that would make currently infeasible experiments possible.

### 2. VARIANCE IN LEARNER ANSWERS

Variance is the reasons why automatic scoring has to go beyond simply matching learner answers to reference answers. The more variance we find in the learner answers, the more complex the scoring model has to be and therefore the harder is the content scoring task (Padó, 2016). Thus, in this section, we discuss why variance increases the difficulty of automatic scoring and analyze publicly available datasets with respect to the variance-inducing properties.

### 2.1. Sources of Variance

From an NLP perspective, automating content scoring of freetext prompts is a challenging task, mainly due to the textual variance of answers given by the learner. Variance can occur on several levels, as highlighted in **Figure 2**. It can occur both on the conceptual level as well as on the realization level, whereas variance in realization can mean variance of the linguistic expression as well as orthographic variance.

### 2.1.1. Conceptual Variance

Conceptual variance occurs when a prompt asks for multiple aspects or has more than one correct solution. For example, in the prompt Name one state that borders Mexico from the POWERGRADING dataset, there are four different correct solutions: California, Arizona, New Mexico, and Texas. A scoring method needs to take all of them into account. However, conceptually different correct solutions are not the main problem, as their number is usually rather small. The

FIGURE 1 | Exemplary content scoring prompts from three different datasets with reference answers (if available) as well as several learner answers with their scoring labels.

much bigger problem is variance within incorrect answers, as there are usually many ways for a learner to get an answer wrong so that incorrect answers often correspond to several misconceptions. For the Powergrading example prompt in **Table 1** (asking What is the economic system in the United States?), frequent misconceptions center around democracy or US dollar, but there also is a long tail of infrequent other misconceptions.

### 2.1.2. Variance in Realization

In contrast to the conceptual variance we have just discussed, which covers different ways of conceptually answering a question, variance in realization means different ways of formulating the same conceptual answer. We consider variance in linguistic expression as well as variance on the orthographic level.

### **<IP_ADDRESS>. Variance of linguistic expression**

This refers to the fact that natural language provides many possibilities to express roughly the same meaning (Meecham and Rees-Miller, 2005; Bhagat and Hovy, 2013). This variance of expression makes it in most cases impossible to preemptively enumerate all correct solutions to a prompt and score new learner answers by string comparison alone. For example consider the

#### TABLE 1 | Dataset statistics.

Tokens per answer are counted individually across all answers for one prompt and the minimum, median1 , and maximum of these values reported. i.e., the prompt with the shortest answers in ASAP has on average 26.5 tokens.

following three sentences. They all come from the SEMEVAL prompt in **Figure 1**. The first is a reference answer, while the other two are learner answers.

While the first learner answer in the example above shares many words with the reference answer, the second learner answer has much lower overlap. The term difference is replaced by the related term measurement. For such cases of lexical variance, we need some form of external knowledge to decide that difference and measurement are similar.

#### **<IP_ADDRESS>. Orthographic variance**

A property of (especially non-native) learner data that also contributes toward high realization variance in the data is the orthographic variability and occurrence of linguistic deviations from the standard (Ellis and Barkhuizen, 2005), which can also make it hard for humans to understand what was intended (Reznicek et al., 2013). For example in the learner answer L1 above, the learner misspelled electrical state as electrial stat. The number of spelling errors – and thus how pronounced this deviation is – depends on a number of factors, such as whether answers have been written by language learners or native speakers or whether answers refer to a text visually available to the learner at the time of writing the answer or not.

### 2.2. Content Scoring Datasets

In the following, we introduce publicly available datasets for content scoring. Afterwards, we categorize all datasets in **Tables 1**, **2** according to various factors that influence variance. The datasets come from different research contexts, we present them here in alphabetical order:

1For the median, we report the lower median if there is an even number of items, so that the value corresponds to the average number of tokens per answer of a specific prompt.

2https://www.kaggle.com/c/asap-sas

TABLE 2 | Overview of content scoring datasets.

### 2.3. Dataset Properties Influencing Variance

We now discuss dataset-inherent properties that can help us to estimate the amount of variance to be expected in data.

#### 2.3.1. Prompt Type

The type of prompt has a strong influence on the expected answer variance. Imagine, for example, a factual question like Where was Mozart born? and a reading comprehension question such as What conclusion can you draw from the text? For the first question, there is no variance in the correct answers (Salzburg) and probably only little variance in the misconceptions (Vienna). For the second question, a very high variance is to be expected. In general, the more open-ended a question is, the harder it will be to automatize its scoring.

Different answer taxonomies have been proposed to classify questions in the classroom according to the cognitive processes involved for the student and they provide also clues about ease of automatic scoring. Anderson et al. (2001) provide a classification scheme according to the cognitive skills that are involved in solving an exercise: remembering, understanding, applying, analyzing, evaluating, and creating in ascending order of difficulty for the student. This taxonomy could of course also be applied to content scoring prompts. Padó (2017) annotates questions in the CSSAG dataset according to this taxonomy and finds that questions from the lower categories are not only easier for students, but produce also less variance and need less elaborate methods for automatic scoring. She also finds that the instructional context of a question needs to be considered when assigning a level (e.g., to differentiate between a real analyzing question and one that is actually a remembering question because the analysis has been explicitly made in the course). Therefore it is hard to apply such a taxonomy to a dataset where the instructional context is unknown.

A taxonomy specifically for reading comprehension questions has been developed by Day and Park (2005). It classifies questions by comprehension as literal, reorganization, inference, prediction, evaluation, and personal response (again ordered from easy to hard). Literal questions are the easiest because their answers can be found verbatim in the text. Such questions tend to have lower variance, especially when given to low-proficiency learners, as they often lift their answers from the text. Also for this taxonomy, it has been found that reading comprehension prompts for language learners focus on the lower comprehension types (Meurers et al., 2011b) and that among these literal questions are easier to score than reorganization and inference questions. We argue that questions with comprehension types higher in the taxonomy contain so much variance that they are difficult to handle automatically. An example for a personal response question from Day and Park (2005) is What do you like or dislike about this article? We argue that answers to such questions go beyond content-based evaluation and rather touch the area of essay scoring, as how an opinion is expressed it might be more important than its actual content.

The modality of a prompt also plays a role. By modality, we mean whether a question refers to a written or a spoken text. Especially for non-native speakers, listening comprehension exercises will yield a much higher variance as learners cannot copy material from the text based on the written form, but mostly write what they think they understood auditorily. This leads especially to a high orthographic variance and makes scoring harder compared to a similar prompt administered as reading comprehension exercise.

**Table 2** shows that existing datasets cover very diverse prompts from reading comprehension for language learning over science question to biology and literature questions, but that they do not nearly cover all possible prompt types.

#### 2.3.2. Answer Length

Answer length of course is strongly related to the type of question asked. Where or when questions usually require only a phrasal answer, whereas why questions are often answered with complete sentences. Shorter answers consisting of only a few words often correspond only to a single concept mentioned in the answer (see the example from the POWERGRADING dataset in **Table 1**), whereas longer answers (as we saw in the ASAP example) tend to be also conceptually more complex. It seems intuitive that this conceptual complexity is accompanied by a higher variance in the data. In a longer answer, there are more options how to phrase and order ideas in different ways.

Answer length is a measure that can be easily determined for a new dataset once the learner answers are collected, so it can serve as a quick indicator for the ease of scoring. In general, shorter answers can be scored better than longer answers. Of course, also datasets with answers of the same length can display different types of complexity and variance. Nevertheless, we consider answer length as a good and at the same time cheap indicator.

**Table 1** presents some core answer length statistics for each dataset. A dataset usually consists of several individual prompts and different prompts in a dataset might differ more or less from each other. To characterize the variance between prompts in a dataset better we give the average answer length in tokens, as well as the minimum, median, and maximum value across the different prompts. **Figure 3** visualizes for each dataset the distribution of the average answer length per prompt. We see that the individual datasets span a wide range of lengths from very short phrasal answers in POWERGRADING to long answers almost resembling short essays in ASAP. We also see that the number of different prompts and individual learner answers and thus also the number of learner answers for each prompt varies considerably, from datasets with only a very restricted number of answers for each question, such as in

CREE and CREG, to several thousand answers per prompt in ASAP.

#### 2.3.3. Language

The language that is used to answer a prompt, such as English, German, or Chinese, is also an important factor influencing the answer variance. Methods that work well for one language may not be directly transferable to other languages. This is due both to the linguistic properties of individual languages as well as to the availability of language-specific NLP resources used for scoring. By linguistic properties we mean especially the morphological richness of a language and the restrictiveness of word order. If an answer given in English talks about a red apple, it might be sufficient to look for the term red apple, while in German, depending on the grammatical context, terms such as (ein) roter Apfel, (der) rote Apfel, (einen) roten Apfel, or (des) roten Apfels might occur. Thus, a scoring approach based on token n-grams usually needs fewer training instances in English compared to German, as an English n-gram often corresponds to several German n-grams. For morphologically-richer languages such as Finnish or Turkish, approaches developed for English might completely fail.

Freeness of word order is related to morphological richness. Highly inflected languages, such as German, have usually a less restricted word order than English. Thus, n-gram models work well for the mainly linear grammatical structures in English, but less so for German with freer word-order and more long-distance dependencies (Andresen and Zinsmeister, 2017).