## **SpringerBriefs in Educational Communications and Technology**

**Michail Giannakos**

# **Experimental Studies in Learning Technology and Child–Computer Interaction**

## **SpringerBriefs in Educational Communications and Technology**

## **Series Editors**

J. Michael Spector, University of North Texas, Denton, TX, USA

M. J. Bishop, University System of Maryland, Adelphi, MD, USA

Dirk Ifenthaler, Learning, Design and Technology, University of Mannheim, Mannheim, Baden-Württemberg, Germany

Published in collaboration with the AECT (Association for Educational Communications and Technology), Springer Briefs in Educational Communications and Technology focuses on topics of keen current interest in the broad area of educational information science and technology. Each Brief is intended to provide an introduction to a focused area of educational information science and technology, giving an overview of theories, issues, core concepts and/or key literature in a particular feld. A brief could also provide:

Michail Giannakos

# Experimental Studies in Learning Technology and Child–Computer Interaction

Michail Giannakos Department of Computer Science Norwegian University of Science & Technology Trondheim, Norway

This book is an open access publication.

ISSN 2196-498X ISSN 2196-4998 (electronic) SpringerBriefs in Educational Communications and Technology. ISBN 978-3-031-14349-6 ISBN 978-3-031-14350-2 (eBook) https://doi.org/10.1007/978-3-031-14350-2

© Association for Educational Communications and Technology (AECT) 2022

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specifc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

*In loving memory of my grandfather, Nikitas Koveos (1925-2022), who taught me the value of hard work, respect and kindness.*

## **Prologue**

In the turn of the millennium I was making my frst steps in the feld of childcomputer interaction coming from a computer science background and thus with little training in empirical research methods. What was driving my efforts then and what still drives many researchers in the feld of child-computer interaction is to create new digital technologies for children, related concepts and interaction paradigms that will render the use of these technologies pleasurable and benefcial for children. Like others in this situation, I soon found myself overwhelmed by the vast literature on methodology, measurements, experiment design, and data analysis, quantitative or qualitative, and still having to make a substantial step to translate methods and practices developed in for social sciences, to address the challenges and choices that I was facing in the feld of child-computer interaction.

Many researchers like me have learnt to design their experiments and navigate these choices by trying and learning from failures and successes in their own research and those reported in the growing body of literature on child-computer interaction and interaction design and children. While these felds of inquiry continue to borrow and learn from neighboring disciplines such as human-factors, psychology, health, and the learning sciences, this approach is not the most effcient for starting researchers. The research approaches used have grown substantially over the years to include research through design, the use of analytics, and even artifcial intelligence techniques, making the challenges described above even more substantial for researchers making their frst steps in this feld now. To move beyond trial and error and an eclectic approach to research methods, the feld of child-computer interaction needs to develop its own resources and a common ground regarding methodologies, challenges, and best practices for conducting empirical research.

With this book Michalis Giannakos makes an important advance to address this need, with this tutorial introduction for graduate students and researchers. Giannakos is one of the protagonists in research in child-computer interaction and especially its overlap with the feld of Educational Technologies. Despite their neighborhood, these felds have different intellectual traditions and adopt diverse research paradigms, so researchers arriving at them from any direction are well served by this introduction to experimental research methods. This book provides a succinct

account of the challenges and common approaches for empirical research with children, especially in relation to experimental research.

Next to guiding beginning researchers in an effcient and focused way, the book contributes a valuable point of reference to the evolving multidisciplinary childcomputer interaction research community, where individual researchers unfamiliar with experimental methods need them in their own research or in evaluating the research of others.

The illustrates the variety of methods and conficting world paradigms apply raising the reader's awareness to the typical social settings for experimental studies, the different degrees of engagement of the researcher with the children and their circle (parents, teachers, friend). Later chapters act metaphorically as a gateway with pointers to specialized literature and resources, overviewing the various artifacts that occupy this subfeld of inquiry, such as dashboards and avatars. It provides overviews of tools supporting data management and sensemaking of learning analytics, and an overview of practices regarding the analysis of learning data in this feld, core concepts like learner-modeling is introduced and different tools to visualize learning, typical measures used in analytics and examples of their application.

Panos Markopoulos

## **Preface**

Many interaction design and learning technology students are introduced to how to design an interface or a service, and how to conduct a study to evaluate it over its indended use. Appropriate methods have been employed extensively over many years to build scientifc knowledge and to understand how to design and evaluate technology that supports human needs (e.g., learning, socializing, and well-being). Depending on someone's background and indended training (e.g., computer science, learning sciences, design) the focus can be on the ideation, design, development, evaluation, continuous use, and/or impact. This short book has been written to support the development of those students, and it is tailored to address how technology can support the learning process and how to take account of the uniqueness of the child as an end-user. Therefore, instead of seeing learning as just one domain among others and the child as just one of many end-users, this book uniquely focuses on child-centered and learner-centered perspectives.

During my PhD work in 2009–2013, child-computer interaction (CCI) research had a steady growth stemming primarily from the human-computer interaction (HCI) feld (but also from the felds of the learning sciences and design), and the learning technology research landscape was revamping due to the introduction of learning analytics. During the last decade we have seen learning technology and CCI leveraging new opportunities stemming from the inception of new methodologies, technological affordances, and computational analysis techniques (usually connected with the use of data science and AI). The work described in this book is based on advancements in learning technology and CCI research from 2010 onwards. The best practices and indicative examples showcased are either directly connected to my own work, or indirectly to works I have participated and/or supervised.

Books focusing in experimental studies with human-factors in IT-related felds do exist, and several of those books can support someone's training in learning technology and CCI. However, those books are not tailored to address how technology can support human's learning or child's needs. Since there is no approach to interaction design and software development that is best for all types of domains

and end-users, a focused approach that takes into account of the uniqueness of the learner and child as end-users is called for.

The book has the following intended audiences:

Many students have learnt to design and develop a system (e.g., CS graduates) or have been introduced to various technologies to support learning or children's lives (e.g., teacher graduates), but have limited training in conducting a study to evaluate the effectiveness or appropriateness of a certain technology. Relevant courses and materials are often a mix of general material (to HCI or the learning sciences) and presentation/papers of the favorite approaches of the instructor. Thus I believe that this short book can provide a comprehensive introduction to CCI and learning technologies as interdisciplinary felds of research and support students' knowledge on how to evaluate the circumstances that favor (or do not favor) the use of experiments; to make the necessary methodological decisions about the type and features of the experiment; to design the necessary "artifacts" (e.g., prototype systems, interfaces, materials, and procedures); to operationalize and conduct experimental procedures to minimize potential bias; and to report the results of their studies for successful dissemination in top-tier venues (such as journals and conferences).

Trondheim, Norway Michail Giannakos

## **Acknowledgments**

A large number of people need to be mentioned for their direct or indirect support. First, I would like to thank the collaborators and co-authors of projects and articles that helped in materializing some of the content of this book. The contribution of most of them is also visible in the references of the book; however, many collaborators have contributed in an indirect way.

During the last almost decade at the Norwegian University of Science and Technology (NTNU), I have had the pleasure to collaborate with a number of master's students, doctoral researchers, postdocs, and faculty members, including Hallvard Trætteberg, Monica Divitini, Letizia Jaccheri, Alf Inge Wang, Patrick Mikalef, Ilias Pappas, Sofa Papavlasopoulou, Kshitij Sharma, Katerina Mangaroska, Boban Vesin, Serena Lee-Cultura, Giulia Cosentino, Guttorm Sindre, Jonas Asheim, John Krogstie, Zacharoula Papamitsiou, Trond Aalberg, Jama Noor, Sobah Abbas Petersen, Lukas Cetusic, Marikken Høiseth, Ioannis Leftheriotis, Jakob Westermoen, Martin Lunde, Evan Niforatos, Camilla Tran, Benjamin Fimreite, Andine Luick, Fredrik Monsen, August Solvang, and Aslak Hollund. Moreover, a number of people have also been infuential through regular lunch or coffee scientifc discourse, including Babak Farschchian, Birgit Krogstie, Yngve Dahl, Patrick Jost, Rune Sætre, Adrian Stoica, Rabail Tahir, Dag Svanæs, Eric Monteiro, Jingyue Li, and Tor Stålhane. I am grateful to my NTNU colleagues, thanks for being such great collaborators and coworkers.

Beyond NTNU, there are also a number of people with whom I have collaborated with during the years among others, Vassilis Kostakos, Peter Brusilovsky, Dragan Gašević, Pierre Dillenbourg, Demetrios Sampson, Niels van Berkel, Panos Markopoulos, Janet Read, Juan Pablo Hourcade, Ole Sejer Iversen, Mutlu Cukurova, Michael Horn, Daniel Spikol, Athanasios Vourvopoulos, Symeon Retalis, Daniele Di Mitri, Xavier Ochoa, and Georgios Yannakakis.

To ensure the quality of this book, I had the fortune of receiving feedback from several people. First, the support of several members from the CCI and the learning technology communities, who provided direct feedback and comments, Panos Markopoulos, Kshitij Sharma, and Sofa Papavlasopoulou. Special thanks to Panos who also accepted to write a foreword for this book. Second, the content behind the

Acknowledgments

book was used as teaching materials for the graduate level "learning technology and analytics" course at NTNU (class 2021–2022). I am very grateful for the feedback and recommendations received from the students of this course.

This book is primarily written to support the students in the Department of Computer Science at NTNU, especially the graduate students participating to our MSc with a specialization on "Interaction Design, Game and Learning Technology." The book would not have been possible without the support from NTNU; the awarded sabbatical allowed me the quality time needed to write up this book. I should also thank the funding bodies of the various projects that contributed to the book; many of the outcomes described in the book have been funded by the Norwegian Research Council, the European Commission, the National Agency for Quality in Education (NOKUT), and the European Research Consortium for Informatics and Mathematics (ERCIM); as well as institutional funding such as the NTNU's Teaching Excellence program (NTNU Toppundervisning) and NTNU's outstanding academic fellowship program (NTNU Stjerneprogrammet). Moreover, I would like to thank NTNU for the fnancial support for publishing this book as open access, I hope that this will not only make it accessible to NTNU students but also allow students from abroad to beneft. Moreover, I would also like to thank the editors of the Springer Briefs of the Association for Educational Communications and Technology (AECT) for accepting to publish this book and especially Michael Spector for his comments and support though his crystallized intelligence.

Finally I would like to thank my parents and my partner for their daily support and love.

Michail Giannakos

## **Abstract**

Experiments and appropriate experimental methods have been employed extensively over many years to build scientifc knowledge and to understand how to design technology to support human needs (e.g., learning, socializing, and wellbeing). The present book discusses ways in which experiments can be employed in the context of research on learning technology and/or child-computer interaction (CCI). Our intention is to support researchers in these felds to employ experimental studies and increase the quality and rigor of their studies. Interim guidelines and best practices exist in neighboring felds with a long tradition in experimental methods (e.g., the learning sciences, behavioral sciences, and psychology), and many of these practices already apply in CCI and learning technology research. However, with this book we provide a complete and comprehensive description on how to design, implement, and report experiments, with a focus on and examples from CCI and learning technology research. The topics we cover include an introduction to CCI and learning technologies as interdisciplinary felds of research, how to design educational interfaces and visualizations that support your experimental studies, types of experiments and their advantages and disadvantages, methodological decisions in designing and conducting experiments (e.g., devising hypotheses and selecting measures), and reporting your results. We give a brief introduction on how contemporary advances in data science, artifcial intelligence, and sensor data have impacted learning technology and CCI research. We also discuss three important issues that a learning technology and CCI researcher needs to be aware of: the importance of the context, ethical considerations, and working with children. The motivation behind and emphasis of this book is helping prospective CCI and learning technology researchers (a) to evaluate the circumstances that favor (or do not favor) the use of experiments, (b) to make the necessary methodological decisions about the type and features of the experiment, (c) to design the necessary "artifacts" (e.g., prototype systems, interfaces, materials, and procedures), (d) to operationalize and conduct experimental procedures to minimize potential bias, and (e) to report the results of their studies for successful dissemination in top-tier venues (such as journals and conferences).

**Keywords** Experimental studies, Experimental methods, CCI, Learning technology, User studies, Methodological decisions, Reporting, Publishing

## **Contents**

## **About the Author**

**Michail Giannakos** is a professor of interaction design and learning technologies in the Department of Computer Science of the Norwegian University of Science and Technology (NTNU). He is the head of the Learner-Computer Interaction lab (https://lci.idi.ntnu. no/), and his research focuses on the design and study of emerging technologies in online and hybrid education settings and on developing new ways for humans to interact with interactive learning systems.

Giannakos has co-authored more than 200 manuscripts published in prestigious peer-reviewed journals and conferences (including *Computers & Education*, *Computers in Human Behavior*, *IEEE Pervasive Computing*, *IEEE TLT*, *BIT*, *BJET*, *ACM TOCE*, *ACM IDC*, *ICLS/CSCL*, *Interact*).

Giannakos, together with Mike Horn, is the Editor-in-Chief of the *International Journal of Child-Computer Interaction* (Elsevier). He is also in the Editorial Board of IEEE Transactions on Learning Technology, *IEEE Transactions on Education*, *Behaviour & Information Technology*, and the *International Journal of Information Management*, and has served as a guest editor on highly recognized journals such as *BJET*, *Computers in Human Behavior*, *ACM TOCE*, and *IEEE Multimedia*. He has served as an evaluator for the European Commission (EC) and the US-NSF, and he recently co-edited *The Multimodal Learning Analytics Handbook* (Springer) and co-authored a textbook on *Educational Data Analytics for Teachers and School Leaders* (Springer).

Giannakos has worked at several research projects funded by diverse sources like the European Commission, Microsoft Research, The Research Council of Norway (RCN), US-NSF, the German Agency for International Cooperation, and Cheng Endowment. Giannakos is one of the experts in the Norwegian task force (formed by the ministry of education and research) for introducing learning analytics to Norwegian K-12 schools and universities. He is also a recipient of a Marie Curie/ ERCIM Fellowship, the Norwegian Young Research Talent Award, and he is one of the outstanding academic fellows of NTNU (2017–2022).

Giannakos usually attends the ACM Interaction Design and Children (IDC) conference (he was a Chair for IDC 2018 and PC-chair for IDC 2022), the ACM conference on Innovation and Technology in Computer Science Education (ITiCSE) (he was a Chair for ITiCSE 2020), and the Learning Analytics and Knowledge (LAK) conference.

## **Chapter 1 Introduction**

**Abstract** As interdisciplinary research felds, child-computer interaction (CCI) and learning technologies have the advantage of enhancing their methods by borrowing from related felds. They represent a research stream that began by applying theories, methods, and tools from a variety of felds, such as the learning sciences, humancomputer interaction, design, and the social sciences. Experiments and appropriate experimental methods have been employed extensively over many years to build scientifc knowledge and to understand how to design technology to support human needs (e.g., learning, socializing, and wellbeing). This chapter discusses how experiments and experimental studies can be employed in the context of research on learning technology and/or CCI.

**Keywords** Learning technology · Child-computer interaction · Experimental research · Experimental studies

Scientifc research follows an iterative process of observation, rationalization, and validation (Bhattacherjee, 2012). As the name suggests, during an observation, we observe (experience/sense) the phenomenon (e.g., event, behavior, or interaction) of interest, and we form an initial research question (RQ). In many cases, the initial question is anecdotal (e.g., you noticed that students who use dashboards complete more assignments, or the ones participating to more classroom quizzes have better mid-term or fnal grades), but it can also be based on some data (e.g., you see that the scores of students who complete tasks in the labs are higher than those of students who complete tasks in the classroom). In the rationalization phase, we try to understand a phenomenon by systematically connecting what we have observed, and this might lead to the formation or concretization of a theory or scientifc inquiry (e.g., research hypotheses). Finally, the validation phase allows us to test potential research hypotheses and/or theories using an appropriate research design (e.g., data collection and analysis).

The research process should be based on the principles of design research with a very intensive collaboration between practitioners and researchers. The ultimate

goal is to build strong connection between research and practice. An emphasis should be placed on the iterative nature of the research process that does not just "test" a technology or a process, but refnes the technology or process while also producing new knowledge (e.g., best practices, design principles) that can support future research and development. Closely situating your work in real-world settings and collaborating with stakeholders, allow you to both clearly identify the problem that you seek to solve, and deploy and evaluate our research in their intended environments. Therefore, the proposed iterative process of observation, rationalization, and validation (Fig. 1.1), should be employed in a way to leverage collaboration among researchers and practitioners in real-world settings and lead to contextuallysensitive knowledge, design principles and theories.

The research that is put into practice varies in type. For instance, the researcher can conduct further observations to rationalize the observations already made (something we used to call inductive research) or test the theory or scientifc inquiry of interest (something we used to call deductive research). The selection of the type of research depends on the researcher's standpoint on the nature of knowledge (epistemology) and reality (ontology), which is shaped by the disciplinary areas the researcher belongs to. Given their interdisciplinary nature, the felds of child–computer interaction (CCI) and learning technology follow both the inductive and the deductive research traditions. Although parts of this book can apply to both types of research, its focus is more on deductive research and how this can be functionalized through experimental studies.

Experimental research has been used extensively as one of the primary methodologies for a wide range of disciplines, from chemistry to physics to psychology to human–computer interaction (HCI) to the learning sciences (LS). The inherent

connections between CCI and learning technology, on the one hand, and HCI and LS, on the other hand, as well as the strong links of all these disciplines to the behavioral sciences, have resulted in the use of experimental studies as one of the predominant modes of research. Experimental studies are often considered to be the "gold standard" (most rigorous) of research designs (Christensen et al., 2011), and from the early 1900s onward, experimental research methods received strong impetus from behavioral research and psychology. The goal of experimental research is to show how the manipulation of a variable of interest (e.g., the resolution of a video lecture) has a direct *causal infuence* on another variable of interest (e.g., students' perception of fractions). For instance, we can consider the following research question: "How does the visualization of students' learning scores via a dashboard affect their future learning performance?"

To conceptualize the RQ, the researcher investigates the effect of the experimental/independent variable on the dependent/outcome variable through an induced "treatment" (a procedure that holds all conditions constant except the independent/ experimental variable). Therefore, any potential signifcant difference identifed when comparing the group with the induced experimental treatment (the experimental group) to the group without the treatment (the control group) is assumed to have been caused by the independent variable (see Fig. 1.2 for a graphical representation.) Such an experiment ensures high internal validity (the degree to which the design of the experiment controls for extraneous factors). Therefore, in contrast to other types of research, such as descriptive, correlational, survey, and ethnographic studies, experiments create conditions where the outcome can be confdently attributed to the independent variable rather than to other factors. Simply put, an experiment is "a study in which an intervention is deliberately introduced to observe its effects" (Shadish et al., 2002, p. 12).

Experiments are not always easy to defne, as they depend on the domain, the RQs, and even the scientist (Cairns et al., 2016). They rely heavily on craft, skill, and experience, and they put tests into practice to trial ideas. In the case of CCI and learning technology, those trials are employed to evaluate existing or new

**Fig. 1.2** Typical representation of an experiment

technologies and interfaces, establish guidelines, and understand how learners/children use technology. The main strength of the experimental paradigm derives from its high internal validity, which allows experimentation to be viewed as an "acceptable" research practice (Hannafn, 1986). Experimental research gives less emphasis to external validity, which concerns the degree to which the results of a study can be generalized to other situations, particularly realistic ones, a focus that is at the center of other research designs and approaches that are commonly employed in CCI and learning technologies (e.g., Barab & Squire, 2004).

As interdisciplinary research felds, CCI and learning technologies have the advantage of enhancing their methods by borrowing from related felds. They represent a research stream that began by applying theories, methods, and tools from a variety of felds, such as LS, HCI, design, and the social sciences. It is not diffcult to see the nature and benefts of interdisciplinarity in CCI and learning technologies that results from the integration of qualities from different felds (e.g., user/learnercenteredness, internal validity, external validity, and accounting for context) and allowing researchers to leverage and combine a wide range of methods, theories, and tools.

The purpose of this book is not to promote or criticize experimental methods, but rather to provide insights for their effective use in CCI and learning technology research. It is important to highlight the importance of "method pluralism" and "letting method be the servant" (Firebaugh, 2018). As in work on experimental methods in human-factor IT-related felds that has criticized "the man of one method or one instrument" (e.g., Hornbæk, 2013; Gergle & Tan, 2014), we want to emphasize the risks of adopting a method-oriented research practice rather than a problem-oriented one. Method-oriented practice is likely to drive researchers to conduct experiments that force-ft the data (Ross & Morrison, 2013) or to dissuade them from conducting experiments when needed, instead relying on methods that center on the experience of the researcher or lead to results that cannot be replicated. As Platt (1964, p. 351) stated, "the method-oriented man is shackled; the problem-oriented man is at least reaching freely toward what is most important."

Experimental studies allow us to isolate which components (e.g., functionalities or affordances) of the technology, the medium, the end-user (e.g., the learner, teacher, or child) or the environment affect the intended goal (e.g., learning or social interaction) and in what ways. In this book, my approach is to present experimental methods as valuable tools for CCI and learning technology research, through the lens of the data-intensive nature of contemporary research. In addition, I emphasize the role of the researcher in using, adapting, altering, and accommodating contextual complexion, relevant theories, and the scientifc inquiry of focus.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 2 Learning Technology and Child–Computer Interaction**

**Abstract** Learning technology research focuses on the design, development, and/or use of technologies that support learning, whereas CCI research focuses on the design, development, and/or use of technologies that support children's lives (with a heavy emphasis on learning). Therefore, learning technology and CCI research can be described as research that focuses on the design, development, and/or use of technologies that support learning and/or children's lives. In this chapter, we provide an introduction to CCI and learning technologies as interdisciplinary felds of research, provide good working defnitions and discuss their commonalities, synergies, and complementarities.

**Keywords** Learning technology · Child–computer interaction · Defnitions · Research felds

## **2.1 Defnitions and Commonalities**

One valid question someone might ask is why this short book focuses on learning technology and CCI. Why not just use some of the previously published books or papers in experimental methods and studies in human-factors IT-related felds (e.g., Hornbæk, 2013; Gergle & Tan, 2014; Ross & Morrison, 2013)? After all, learning technology is closely related to the learning and education sciences, and CCI is closely related to interaction design and HCI. What are the commonalities of learning technology and CCI that require a shared technical note? Do we have enough commonalities to justify a book on the subject?

The truth is that books on experimental methods from neighboring research communities (e.g., the learning sciences and the social sciences), and especially those from human-factors IT-related felds (e.g., HCI and information systems), can indeed be used to support researchers in learning technology and CCI. However, those books are not tailored to address how technology can support the learning process or to take account of the uniqueness of the child as an end-user. Instead, they see learning as just one process among others and the child as just one of many

end-users. Therefore, this book aims to cover CCI and learning technology more comprehensively than books that focus on HCI, LS, or software engineering, with a unique focus on child-centered and learner-centered perspectives.

Another interesting question concerns whether CCI and learning technology (and the respective child-centered and learner-centered perspectives) can go together. First of all, let us consider how the main publication venues, organizations, and societies of these two felds see themselves, their commonalities, and their differences.

In terms of learning technology, *Computers & Education* journal1 (the top-ranked educational technology journal according to Google metrics2 ) welcomes research on "knowledge and understanding of ways in which digital technology can enhance education" and "on the pedagogical uses of digital technology, where the focus is broad enough to be of interest to a wider education community." Another reputable but more technical journal, *IEEE Transactions in Learning Technology*, 3 welcomes research on "advances in learning technologies and their applications, including but not limited to: online learning systems; intelligent tutors; educational games; simulation systems for education and training; collaborative learning tools; learning with mobile devices; wearable devices and interfaces for learning …" The Association for Educational Communications and Technology (AECT) is the oldest professional association for instructional designers and an academic and professional association that promotes educational uses of technology. The AECT defnes educational technology as "the study and ethical practice of facilitating learning and improving performance by creating, using and managing appropriate technological processes and resources" (Richey, 2008, p. 24). Another reputable scholar who has served as AECT president and is editor emeritus of the Educational Technology Research & Development (ETR&D), Spector (2015, p. 10), describes educational technology as the "application of knowledge for the purpose of improving learning, instruction and/or performance." By looking closely at the defnitions of these leading journals, scholars, and the AECT, we see that the focus is on the design, development, and use of learning technology, with some venues focusing on the use and others on the design/development. For example, we regularly see Computer Science departments focusing on design and development, whereas education/learning sciences departments tend to focus on use (without this being an absolute rule). In reality, however, these different aspects need to be studied jointly and iteratively.

The initiation of CCI as a feld of research stems from the 1960s (Giannakos et al., 2020a, b), when pioneering researchers such as Seymour Papert, Edith Ackermann, Marvin Minsky, and Alan Kay explored the design of computer systems for children. In 2002 the research community established the International Conference on Interaction Design and Children (IDC), which is an annual venue for

<sup>1</sup>Computers & Education: https://www.journals.elsevier.com/computers-and-education

<sup>2</sup>Google Scholar metrics, educational technology sub-feld: https://scholar.google.com/ citations?view\_op=top\_venues&hl=en&vq=eng\_educationaltechnology

<sup>3</sup> IEEE Transactions in Learning Technology: https://ieee-edusociety.org/publication/ieee-tlt

research on "the latest research fndings, innovative methodologies and new technologies in the areas of inclusive child-centered design, learning and interaction." The *International Journal of Child–Computer Interaction* (IJCCI), which is the only journal focusing explicitly on CCI, welcomes research on knowledge concerning "the phenomena surrounding the interaction between children and computational and communication technologies" (Giannakos et al., 2020a, b, p. 1).

Therefore, it is safe to say that learning technology research focuses on the design, development, and/or use of technologies that support learning, whereas CCI research focuses on the design, development, and/or use of technologies that support children's lives (with a heavy emphasis on learning). If we try to unfold the scope of those two communities, we realize that there are two pillars: the focus area and the developmental stage/age of the end-user. The focus area of learning technology (e.g., learning, teaching, competence development, assessment, and cognition) is a subgroup of CCI, which covers all the focus areas of learning technology, plus others such as sociability, healthcare, and play. The developmental stage/age of the end-user in CCI (from toddlerhood to adolescence) is a subgroup of learning technology, which covers all the end-user groups of CCI, including university students, lifelong learners, and other adult learners. Based on these considerations, learning technology and CCI research can be described as research that focuses on the design, development, and/or use of technologies that support learning and/or children's lives. To clarify the commonalities and differences between learning technology and CCI, we have made a Venn diagram (Fig. 2.1).

**Fig. 2.1** Venn diagram depicting the relationship between CCI and learning technology. (Note: The size of the diagram and the various spaces and sets does not correspond to the amount of research in each feld)

Besides the obvious differences stemming from the difference in focus (in learning technology, the user as a learner; in CCI, the user as a child), we see a substantial overlap in the research methods employed and the amount of research that is suitable for both venues. For instance, research in learning technology in K-12 schools or kindergarten and preschool settings is welcomed in CCI research communities and venues. Similarly, research in CCI that focuses on computing/programming education, game-based learning, and constructionism/making in childhood is welcomed in learning technology research communities and venues. A recent analysis of the research themes that have been published in the main CCI venues (i.e., the IDC and *IJCCI*), makes evident the inherent connection between CCI and learning technology (Giannakos et al., 2020b). Therefore, the methodological similarities, substantial overlaps between the research communities, and common thematic areas justify a book that considers CCI and learning technology together. Figure 2.2 shows the volumes and relationships of CCI themes (based on the keywords of published papers); the central role of learning technology (e.g., education as a central theme) is clear.

**Fig. 2.2** Network map of the keywords of articles published in the two main CCI venues (IDC and IJCCI), 2013–2018. (from Giannakos et al., 2020b; licensed under CC BY-ND 4.0)

## **2.2 Synergies and Complementarities**

As described above, there are various similarities between CCI and learning technology research, and this is why it makes sense to describe the implementation of their experimental studies together. At the same time, there are differences that prospective researchers should take into consideration. The main difference is that one adopts a child-centered perspective and the other a learner-centered perspective. Although these perspectives are not mutually exclusive, it is not always possible or desirable to maintain both (e.g., when conducting learning-at-scale studies in learning technology). The two perspectives also affect the role and participation of the end-user in research and experimentation. The learner (as a user) demonstrates a higher degree of "autonomy" in the experimentation, while the child (as a user) requires signifcantly more contextual interpretation of the results and investigation to be carried out jointly with their support sphere (i.e., learning facilitators, such as teachers, parents, and therapists).

These differences result in differences in the methods employed and how they are implemented. For example, it is very common in learning technology studies to collect learner-generated data, develop learners' trajectories, and/or explore that data in different ways. This has led learning technology research to focus on the various analytics produced and to explore the role of the various technological processes and resources in supporting learning. A consequence of this paradigm shift (some might say acute need or development in the feld) is the "conception" of learning analytics (Siemens, 2013), defned as "the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs" (LAK, 2011). Although child-produced analytics are also used in CCI research, this development has affected CCI to a lesser extent, since the growth of learning analytics stems from the increase in online learning that has taken place during the last decade, particularly in post K-12 settings (e.g., higher education and learning in the workplace).

In CCI, however, children's roles and participation in experimentation has been studied extensively (Markopoulos et al., 2021). Children's roles in the design and evaluation of technology vary according to the maturity of technology prototypes (e.g., low to high fdelity), the expected end-user of the products (e.g., an auxiliary user, or children as primary or secondary users), and the children's ages and abilities (Giannakos et al., 2020a, b). These roles were initially described using three dimensions: the relationship to adults, the relationship to technology, and the goals of the inquiry. Four initial roles for children in the design process have been defned: users, testers, informants, and design partners (Druin, 2002). This distinction between roles has prevailed in CCI research with further efforts to engage children even more intensively in the design process (Iversen et al., 2017). Therefore, methodological approaches have been designed that embrace the fact that children are central participants in experimentation and not just users of the technology (e.g., Frauenberger et al., 2015). Another particularity of CCI research is that in many cases it is neither practical nor possible to collect data directly from the child (the end-user). Therefore, CCI research is often supported by proxy data collections and experiments that include data collection from the children's support sphere (e.g., surveys or interactions with teachers or parents). This might also need researchers' participation during the intervention to obtain insights that cannot be captured via interactions and that teachers or parents have diffculties in "sensing."

In order to clarify potential differences in the approaches employed, Fig. 2.3 gives an overview of three approaches, each with an example and some potential advantages and disadvantages. The frst approach is typical for studies that investigate how the child or the learner can autonomously use a technology, this approach is also appropriate for learning-at-scale studies (e.g., a large MOOC or LMS-based intervention); the second approach is used with technologies that are designed for the learner/child and someone from their support sphere as joint primary users (e.g., an intervention to provide technology to support classroom teachers or parents); the third is commonly found when technologies require user insights in the early phases of the design (e.g., an intervention to capture the requirements or defne the nature of the services, rather than specifc implementation details).

Before concluding this section, we want to emphasize that there is no "good" or" bad" approach here. The aforementioned approaches have been developed and used to meet different objectives and needs. Therefore, when someone asks which approach to use, the answer is that it depends on the purpose and context. Consider the nature of CCI research and the challenges it involves; for example, primary

**Fig. 2.3** Three different approaches employed in CCI and learning technology research: (1) autonomously use a technology; (2) technologies that are designed for the learner/child and someone from their support sphere as joint primary users; and (3) technologies require user insights and participation in the early phases of the design

school children have not yet fully developed some skills (including language and reading abilities and abstract thinking), and their thinking processes are based on mental representations that relate to concrete events, objects, and experiences. Under those circumstances, it is natural that we should see a number of research endeavors that focus on methods that are participatory and contextually rich. On the other hand, with the increasing interest in "autonomous," "context-independent," and "at-scale" learning technologies (e.g., MOOCs and LMSs), it is also natural to see research endeavors that focus on producing large amounts of educational data to account for different contexts. Of course, synergies between the different approaches do exist, and all the aforementioned approaches are relevant for both CCI and learning technology research.

Advancing our understanding and the appropriate approaches and methodological processes in relation to children's and learners' roles and participation in research, as well as furthering their engagement in the design process, is at the top of both the CCI and the learning technology research agendas (e.g., Giannakos et al., 2020a, b; DiSalvo et al., 2017). Therefore, continuing to develop and advance synergies on the methodological front between CCI and learning technology is a way to cross-fertilize and accelerate progress.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 3 Educational Interface Design and the Role of Artifacts**

**Abstract** User interfaces (UI) are an inherent part of any technology with human end-users. The design of the UI depends heavily on the intended end-user and is therefore extremely important for research in both learning technology (where the learner is the end-user) and CCI (where the child is the end-user). Another important concept of learning technology and CCI research (and also in neighboring felds) is that of "artifact". Artifacts correspond to novel designs (which may be prototype systems, interfaces, materials, or procedures) that have a certain set of qualities or components (such as functionalities and affordances) and that allow us to experiment (e.g., to isolate and test certain components). This chapter describes how researchers can design educational interfaces, visualizations, and other artifacts to support their experiments and enhance learners' and children's experience with technology.

**Keywords** Learning technology · Educational interfaces · Artifacts · Artefacts · Interaction design

## **3.1 Design of Educational Interfaces**

User interfaces are an inherent part of any technology with human end-users. The role of an interface is to facilitate effcient communication and information exchange between the machine (the technology) and the user (the human). User interfaces (UIs) rely on what we call "interface metaphors," sets of visuals, actions, and procedures incorporated into the UI that exploit specifc knowledge that users already have of other domains, such as their homes and working environments. The use of proper interface metaphors allows users to predict the functionalities of each element of the interface (metaphor), resulting in more intuitive use of the interface and more predictable system behavior. Confusion is avoided, as there is no need for explanations of the various elements of the UI, and users are aware of the impact that their actions will have on the system. A time-tested example is the "desktop"

metaphor, which portrays the operating system as similar to objects, tasks, and behaviors found in physical offce environments (Neale & Carroll, 1997).

The appropriate selection and application of UI metaphors make systems easy to use, and so we need to understand how metaphors are perceived by our targeted end-users. Good understanding will allow us to incorporate metaphors effciently into our UIs. Below, we provide some commonly used metaphors that allow UI designers to develop intuitive interfaces. As technology advances and different applications are developed (including new ways of working, living, learning, and communicating), new metaphors need to be established to increase the usability of those applications. The examples show the centrality of metaphor to UI and the importance of drawing on real-world analogies (Table 3.1).

The selection of metaphors and the design of the UI depend heavily on the intended end-user and are therefore extremely important for research in both learning technology (where the learner is the end-user) and CCI (where the child is the end-user). For example, a straightforward note-making metaphor (e.g., for presenting new information) might be good for a technology that targets teachers but less effective for a technology that targets doctors (where a Post-it metaphor might work better). The same applies to all user groups, although learners and children are particularly interesting end-users. Learning is not always an easy process. It is associated with many aspects of interaction and cognition (including diffcult mental operations and cognitive friction), and these differ across the different developmental phases of a child. For instance, for very young children, even time-tested metaphors such as "desktop" can fail to convey the intended information. Therefore, it is important to work closely with the end-user to develop an appropriate set of visuals, actions, and procedures that can be incorporated into the UI to achieve the intended objectives (see for example, Fig. 3.1, Asheim, 2012; Høiseth et al., 2013). Moreover, learning takes place in and across diverse contexts (e.g., online or in classrooms, labs, and maker spaces), and the content area (e.g., math, language, or art) plays an important role in the mental models generated by the user during learning and the ways in which those models need to be taken into consideration to facilitate learning.

The main focus of metaphors is ease-of-use, usability, and utility for representing a system's functionality (Kuhn & Blumenthal, 1996). However, the capacity of UI

**Table 3.1** Some commonly used UI metaphors

**Fig. 3.1** Examples of how to achieve objectives by using familiar and repetitive elements (adopted from: Asheim, 2012; Høiseth et al., 2013, with permission by Asheim and Høiseth)

metaphors to facilitate learning has been historically recognized and valued by both the learning technology and the HCI communities (e.g., see Carroll & Mack, 1985; Neale & Carroll, 1997). Metaphors facilitate learning by leveraging existing mental models or previously learned information and applying them to new contexts (Bruner, 1960). Learning is accelerated when metaphors are used, because they draw on existing knowledge bases to reason about new problems (Streitz, 1988).

Contemporary research and practice recognize the importance of iteration and end-user participation during the UI design (e.g., DiSalvo et al., 2017). Processes from HCI, such as rapid prototyping and low-fdelity paper prototyping (Wilson & Rosenberg, 1988), are commonly used in educational UI. Those practices are advantageous because of their simplicity, low cost (no need for many working hours or materials/tools), and the ease of obtaining early feedback from the end-user. They also adopt the main steps of established instructional system models, such as ADDIE (analysis, design, development, implementation, and evaluation) (Branch, 2009), which allows the necessary steps to unfold iteratively. The powerful progression from a low-fdelity, pen-and-paper prototype to a working system is shown in Fig. 3.2 through two examples, one on the development of a UI for a multi-touch gamifed quiz system that supports learning in museums (Noor, 2016), and one on the development of a UI for a self-assessment technology that supports online learning (Westermoen & Lunde, 2020). As the fgure shows, the initial low-fdelity ideation is created using only pen and paper. The sketches are very basic, but they also useful for determining how the user will interact with the interface; because the sketches in this phase of the design are low-fdelity, it is easy and "cheap" to change them. After the frst iteration, some of the features are developed and tested with a few end-users, but even then it remains easy to test the basic functionalities and accommodate the results from the testing (e.g., in terms of metaphors used, information visualized, and actual functionalities). As the fdelity of the interface increases and more interactive functionalities (and the respective wireframes) are incorporated, it becomes more diffcult and costly to accommodate structural

**Fig. 3.2** The process from low-fdelity pen-and-paper prototype to a working system. Left: The development of a UI for a multitouch gamifed quiz system that supports learning in museums. (From Sharma et al., 2020; licensed under CC BY-ND 4.0). Right: The development of a UI for self-assessment technology that supports online learning. (From Westermoen & Lunde, 2020, with permission by Westermoen and Lunde)

changes. In the fnal stages of the process, we have a working system that can be tested through experimentation.

Within the progression from low fdelity to high fdelity and ultimately the complete UI, the designer needs to make progress in the development of the navigation thread. The storyboarding/navigation thread will cover all the possible use cases and scenarios and the interconnections within the wireframes. Figure 3.3 shows the storyboarding of a self-assessment UI (adapted from Westermoen & Lunde, 2020). During the design of the educational UI, the designer needs to keep in mind who the intended end-users are (e.g., children, other learners); what their characteristics are (age, background knowledge); the expected objectives (learning goals, competence development); the different types of constraints (learning constraints, technological constraints, teachers' competence); the delivery options and expected role of the technology; and its pedagogical underpinning. In addition to answering these very important questions, the UI designer needs to be able to gather information from end-users and test their ideas.

**Fig. 3.3** Storyboarding of a self-assessment UI. (Adapted from Westermoen & Lunde, 2020; with permission by Westermoen and Lunde)

As a result of the iterative design process and storyboarding, a collection of wireframes representing each possible view a user might encounter is created. The fnal UIs need to consider the context of use and provide the necessary guidelines for the implementation of the application. Figure 3.4 shows an example set of UIs in the context of mobile learning in higher education. (More information about this example can be found in Pappas et al., 2017 and Cetusic, 2017)

**Fig. 3.4** Final user interfaces (wireframes) of a mobile learning application. (Adapted from Pappas et al., 2017, with permission by IEEE)

## **3.2 Artifacts and Treatment Design**

One of the frst notions the researcher needs to understand in learning technology and CCI research (and also in neighboring felds) is the unit of analysis (UoA). The UoA is the object that is the target of the experimentation (and whose data we use as a unit in our analysis).  For instance, if we are interested in studying the effect of the use of a dashboard or a representation (an avatar) on students' learning outcomes or attitudes, then the UoA is the student, since we will use the score of each student. If we want to study the introduction of novel technology to support collaboration in dyads or triads, the UoA is the dyad or triad, since we will use the score of each dyad or triad (e.g., scores from a common assignment). Even objects can serve as a UoA; if we want to make an interface more attractive to students, then the UoA is the group of students who use the interface. Identifying

the UoA can be complex, as it is not always static. It is common in a study with a specifc dataset to have different UoAs. For example, an analysis of student scores can be based on the scores of individuals, of classes (if we want to compare the practice of different teachers), or of different groups.

Another important concept that is a cornerstone in learning technology and CCI research (and also in neighboring felds) is that of "artifact" (or "artefact" in British English spelling) (Carroll & Rosson, 1992). Artifacts correspond to novel designs (which may be prototype systems, interfaces, materials, or procedures) that have a certain set of qualities or components (such as functionalities and affordances) and that allow us to experiment (e.g., to isolate and test certain components). Such experimentation serves to advance both empirical and theoretical knowledge, but it also supports the practice of a user (such as a learner or a child) and empowers them to achieve their potential. Artifacts allow us to formulate the necessary conditions by isolating certain functionalities and testing our hypotheses through experimentation. Each experimental study has its own value and should contribute to the main body of knowledge by validly testing theories that are contingent on designed artifacts, or by producing fndings that may be reused to support the design of future artifacts in the form of lessons learned or design implications (Sutcliffe, 2000).

Contemporary learning technology and CCI research focuses on conducting "artifact-centered evaluations" that use artifacts in the experimental process. The most common approaches cascade the experimentation process within a broader research procedure, with the intention of producing new knowledge and models and informing theories and practices. Such approaches inherit the characteristics of design research and are iterative. For instance, design-based research (DBR) is a common approach in learning technology, whereas the task–artefact cycle is commonly employed in HCI (see Fig. 3.5). Such research approaches are important, as they go beyond responding to a particular hypothesis, instead seeking to advance theoretical knowledge in the feld by exploring and confrming various hypotheses and relationships in different contexts (see a representation in Fig. 3.6).

**Fig. 3.5** Representations of common experimental processes. Left: Design-based research (DBR) research, which is commonly used in learning technology research (Barab & Squire, 2004). Right: The task–artefact cycle, which is commonly used in interaction design research. (Adapted from Carroll & Rosson, 1992; Sutcliffe, 2000). Both processes are iterative in nature and focus on producing practical and theoretical knowledge

**Fig. 3.6** Iterative design research focusing on producing empirical and theoretical knowledge

Going back to the important role of artifacts in conducting empirical studies, we now provide some examples of how artifacts allow us to move from observations to designing treatments and testing hypotheses. A very common interface in learning technology research is the dashboard. Dashboards are used in learning management systems (LMSs) such as Canvas, Moodle and Blackboard Learn, and also in the majority of learning technologies (e.g., digital games, educational apps). Although there are differences in the information included, the visualizations employed, and the moments at which the dashboard appears, most dashboards include information related to learners' activity, progress, and learning history. The information provided to the learner (and teacher) is intended to activate their awareness, refection, and judgment (i.e., metacognition), and ultimately to support their potential (by informing them about the amount of time spent on a task, the diffculty of a particular question, and so on). Providing this information in an effcient manner will support learners' self-regulation and motivation, and teachers learning design and decision making, allowing them to make appropriate decisions about allocation of effort, time-management, and skills development (Lonn et al., 2015).

Figure 3.7 (up) shows a learning dashboard, taken from a previously introduced example (Westermoen & Lunde, 2020), this dashboard has been designed and introduced to support students' self-assessment. The dashboard was introduced to one of two groups of students, and a mixed methods study was conducted to investigate the role of the dashboard in digital self-assessment activities (Westermoen & Lunde, 2020; Papamitsiou et al., 2021). Fig. 3.7 (down) shows a teacher dashboard, this dashboard has been designed and introduced to support teachers' decision making (e.g., identifying students' weaknesses and misconceptions, or students who need additional support). The dashboard was evaluated with experienced teachers to identify its usefulness and ability to support decision making and instruction (Luick & Monsen, 2022).

**Fig. 3.7** Up: Student dashboard with task-related analytics for each question. (From Westermoen & Lunde, 2020; with permission by Westermoen and Lunde); Down: teacher dashboard with taskrelated analytics for the whole class and course. (From Luick & Monsen, 2022; with permission by Luick and Monsen)

Another example is provided by artifacts that lie at the intersection of CCI and learning technology. To investigate the effect of avatar self-representation (ASR) (the extent to which the user/child is represented by an avatar) in learning games, we used three games that follow similar game mechanics but have a different approach to user ASR. ASR is classifed as low, moderate, or high, according to the degree of visual similarity (i.e., appearance congruity) between the avatar and the user and the precision and breadth of movement (i.e., movement congruity). Figure 3.8 gives a detailed description of the ASR classifcations and the respective game interfaces.

The group of children experienced all three ASRs (conditions), and during the treatment (i.e., a within-subjects experiment) we carried out various data collections, with the goal of determining the role of ASR in children's affect and behavior in motion-based educational games. The results indicated that moving from low ASR (a cursor) to moderate ASR (a puppet) and then to high ASR (an image of the actual user) decreased users' stress and increased their cognitive load (see Fig. 3.9). You can fnd the complete study, with all the details and results, in Lee-Cultura et al. (2020).

The use of artifacts is powerful, but it also has limitations. For example, the results are associated with the particular artefact under study, and any knowledge obtained is not necessarily reusable or generalizable to other contexts. Nevertheless, artifacts allow us to conduct experiments and test hypotheses effciently so as to enhance relevant practical and theoretical knowledge. In addition, there are certain time-tested approaches in both learning technology and CCI/HCI (e.g., DBR; Barab & Squire, 2004) and the task–artefact cycle (Sutcliffe, 2000) that allow us to leverage iterative experimentation to go beyond context-specifc hypothesis testing and produce reusable/generalizable knowledge.

**Fig. 3.8** Artifacts corresponding to three different degrees of avatar self-representation (ASR). (Adapted from Lee-Cultura et al., 2020, with permission by Lee-Cultura)

**Fig. 3.9** Indicative results of the ASR study. (Adapted from Lee-Cultura et al., 2020, with permission by Lee-Cultura). The blue bars show 95% confdence intervals. Statistically signifcant differences are marked with \* for p < = 0.05, \*\* for p < = 0.001, and \*\*\* for p < = 0.0001

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 4 Educational Data, Learning Analytics and Dashboards**

**Abstract** When learners interact with technologies and the learning context, a large amount of data is created. The collection, analysis, and utilization of those educational data has provided opportunities for learning technology (and CCI) research. In this chapter, we will discuss how learning systems produce and utilize educational data. In particular, we will discuss contemporary developments in the felds of learning analytics, educational data mining, and learner modelling; and how those advancements have impacted the design and functionalities of learning technologies.

**Keywords** Educational data · Learning dashboards · Learning technology · Learning analytics

## **4.1 Educational Data and Learning Analytics**

In the second chapter, we introduced the notion of learning analytics, a central concept of which is the "learning trace," or, more generally speaking, the "user trace." Those traces are left behind when learners interact with technologies and the learning context in general (e.g., other learners, an instructor, or nondigital learning materials), and are represented by different datasets. Learner interaction is often complex (e.g., watching a video or answering a multiple-choice question), and traditional analytics model those interactions as a sequence of logs (e.g., video navigation, response times, and response correctness). These learning traces and the respective representations (visualizations, graphs, or diagrams) are used to improve the system's functionalities and intelligence (through, for example, recommender systems or visualization of an individual's progress) and the respective pedagogy, allowing learners and teachers to be aware of the possible misunderstandings and challenges associated with different content areas. Figure 4.1 depicts how typical use of learning systems (from instructors and learners) produce data that are processed from data analysis methods with an ultimate goal to support learning and instruction.

**Fig. 4.1** Typical use of a learning system from the instructor and the learner, and the production of data that can support the learning and instruction

The datasets employed in learning technology platforms usually follow standards such as the sharable content object reference model or SCORM (SCORM, 2004) or other standardized data structures and formats. SCORM is the most widely used specifcation, with the goal of interoperability and smooth data and content exchange between learning technologies. Most LMSs use (or are at the very least compliant with) SCORM, which has its origins in a cooperation between the Institute of Electrical and Electronic Engineers (IEEE) Learning Technology Standards Committee (https://ltsc.ieee.org/), the IMS Global Learning Consortium (www.IMSproject.org), and the Alliance of Remote Instructional Authoring and Distribution Networks for Europe (ARIADNE). Independently of the standards employed, educational data also employ data structures in various formats, such as JavaScript object notation (JSON), extensible markup language (XML), or commaseparated values (CSV), which enable both the system and the researcher to carry out visualization and analysis. For example, you can fnd here1 example events exemplify how you can utilize learners' tracking logs, those examples are from edX (an American MOOC provider created by Harvard and MIT), edX has an opensource platform (open edX) that powers edX courses. Those events (edX.log fles) have no personally identifable information, but exemplify how they can help us to gain insight into leaners and teachers. Learning interaction data sometimes might be provided in a relatively "primitive" form (the same applies for edX), however, there are several technological tools, such as HarvardX Tools (developed from Jim Waldo, see here: http://github.com/jimwaldo/HarvardX-Tools) that allow us to package, analyze, and manipulate by converting tracking log data to standard and manageable formats (e.g., csv based on ADL's xAPI (https://adlnet.gov/projects/ xapi/)). Another useful resource is the Pittsburgh Science of Learning Center's

<sup>1</sup>EdX guide: https://edx.readthedocs.io/projects/devdata/en/stable/internal\_data\_formats/tracking\_logs.html

DataShop (https://pslcdatashop.web.cmu.edu/) that hosts datasets from learning systems (most of them are from Intelligent Tutoring Systems) and provide them as a service to the learning technology community (e.g., so researchers can store or request and research on learning interaction data).

Learning analytics are then utilizing learning traces via the respective produced datasets to exploit opportunities for improving learning and instruction, to do so learning analytics employ computational analysis techniques coming from data science and AI. The majority of the learning analytics studies utilize descriptive statistics and basic visualization techniques (Fig. 4.2). For example, frequencies (the number of times a particular score or value is found in the data set), percentages (a set of scores or values as a percentage of the whole), means (numerical average of the scores or values for a particular variable) and medians (the numerical midpoint of the scores or values that is at the center of the distribution of the scores) are used often to support learners and instructors. Other statistical techniques such as correlational and regression analysis are also used, however, there are limited studies employing advanced data analysis methods in the context of learning analytics. Figure 4.2 categorizes the results from a literature review conducted from Misiejuk and Wasson (2017), that depict the frequency of the data analysis methods used. Moreover, Fig. 4.3 summarizes the advanced computational analysis techniques that can be used to support learning (Daud et al., 2017). The comparison indicates that learning analytics research and practice makes use of advanced computational analysis techniques, but to a limited extend (this indicates the status until 2017). Although this might connect with researchers' preferences, experience and aspirations; this might also connect with the fact that basic descriptive statistics (e.g., median, mean, frequency) might be easier for the end-users (e.g., students, teachers) to sense-make and act upon.

Types: descripve - exploratory - diagnosc - predicve - prescripve Meaning: what has happened? – why did it happen? - what is likely to happen? - what might happen?

**Fig. 4.2** Frequency of the data analysis methods used in the context of learning analytics research. (Source data came from Misiejuk & Wasson, 2017)

**Fig. 4.3** Summary of the advanced computational analysis techniques that can be used. (Adapted from Daud et al., 2017; licensed under CC BY-ND 4.0)

## **4.2 Learner Modeling**

In the process of standardizing how this information can be used to support the system (e.g., by providing intelligence) or the users (e.g., students and teachers), learning technology research came up with the term "learner model" (or "student model") (Bull, 2020). Learner modeling stems from user modeling, which describes the process of building and updating a conceptual understanding of the user (Fischer, 2001); by analogy, *a learner model describes the process of building and updating a conceptual understanding of the user as a learner*. To do this, it models attributes such as skills, competences, understanding of concepts, and misconceptions, as well as noncognitive attributes such as motivation, engagement, and effort (Bull, 2020). Learner models use a range of data, including response time, response correctness, number of attempts to solve a problem, time spent interacting with learning resources, navigation to various learning resources, activity on the various communication functionalities (e.g., forums), and other learning trace data (Bull, 2020). Learner models are (usually) automatically and dynamically generated and updated

(that is, inferred from the data). However, there are models that use manual input from learners' records or the teacher. The main objective of the learner model is to enable the learning system's intelligence and system functionalities (e.g., personalized learning, recommender systems, dashboards, and adaptivity) to support endusers, be they children, students, teachers, or parents. Bull and Kay (2010) provide several examples of learner models designed to support various users, including children, peers, parents and teacher.

Such functionalities allow technologies to support learners' educational needs, with learner models being a core component in the development of several learning technologies, such as MOOCs (e.g., Cook et al., 2015), LMSs (e.g., Chou et al., 2015), and, in particular, intelligent tutoring systems (ITSs) (Woolf, 2010). During the last years, we have mainly seen open learner models, which externalize their functionalities in a way that allows users to interpret them through, for example, visualizations. In addition, there are several open source tools that allow us to include such functionalities in our systems, such as the open learner model (OLM) application, developed by Susan Bull in the context of an EU project called Next-Tell (http://next-tell.eu/portfolios/olm/). Learner models visualize individual learners' current understanding (knowledge mastery) of a topic (see Fig. 4.4 for some

**Fig. 4.4** Learner modelling tools that visualize individual (or group) learners' current understanding of a topic or their level of competency, and support different processes (e.g., following progress, comparison, identifying misconceptions)

examples). These models are powered automatically from a variety of sources, and they visualize the most up-to-date information about learner competency to allow learners and instructors to identify strengths and areas for further attention.

## **4.3 Educational Dashboards and Visualization**

For the end-users of learning technologies (learners, teachers, and administrators), it is extremely useful to have information presented in an understandable way that supports their objectives of learning, teaching, or making administrative decisions about learning and teaching. Visualizations such as charts, graphs, and maps provide accessible ways to see and understand useful trends and patterns in the data. However, a single chart, graph, or map cannot contain all the information and insights needed to support end-users' informed decision making. The solution is to use combined information visualization techniques with the use of a dashboard, so that the different end-users no longer need to "drive blind" (Duval, 2011).

In the literature, dashboards have been defned as "an easy to read, often singlepage, real-time user interface, showing a graphical presentation of the current status (snapshot) and historical trends of an organizations key performance indicators (KPIs) to enable instantaneous and informed decisions to be made at a glance" (Brouns et al., 2015). In the context of learning technology, a learning dashboard has been defned in the context of a recent literature review as "a single display that aggregates different indicators about learner(s), learning process(es) and/or learning context(s) into one or multiple visualizations" (Schwendimann et al., 2016). The design and use of learning dashboards has increased tremendously in recent years, but there are still several important decisions that need to be made by the designer of the dashboard. For instance, what is the "right" information to visualize for the different end-user groups? How does this information need to be visualized, in which part of the system (which UI), and at which point of use (which part of the storyline)? Although there is unlikely to be a single answer for every learning dashboard, it is important to consider fve main points:

In recent years, various learning dashboards have been designed to support teaching and learning in different contexts. A pioneering project was the EU project ROLE (Responsive Open Learning Environments). Fig. 4.5 provides a range of visualizations employed: (a) in the context of the ROLE project (Santos et al., 2011) and (b) in the context of the work from Luick & Monsen (2022) (10 years later). In line with the frst principle of designing a learning dashboard (that it be purposeful), we can easily identify the goal for these different visualizations. For instance, we can see

**Fig. 4.5** Left: elements of a teacher dashboard designed in the context of the EU ROLE project. (Adapted from Santos et al., 2011; licensed under CC BY-ND 4.0); Right: elements of a teacher dashboard. (Adapted from Luick & Monsen, 2022; with permission by Luick and Monsen)

that most of the students were most active in the period May–July (top-left visualization), that they entered chatrooms much more often than they posted messages (second and third line in middle-left visualization), and that the most "productive" was Monday (bottom-left visualization). In another project conducted at NTNU 10 years later, by Luick & Monsen (2022), we see dashboards that allow teachers to follow students' progress over time (up-right visualization in Fig. 4.5), across different topics of a course and providing insights about the learning content (bottomright visualization in Fig. 4.5). Such information can help students to refect on the activities they engage with and the behaviors they exhibit. It can also help the teacher to reorganize and/or redesign the learning activities in a way that is more engaging for the students, and focus on topics that are diffcult to master.

Recent developments in the design of learning dashboards have incorporated capabilities such as social comparison (where individuals can see their own KPIs together with the cumulative KPIs of the classroom/group), with the goal of supporting self-regulated learning and student engagement. One example is Mastery Grids, which uses learners' data to provide interactive and adaptive visualizations that support their engagement, performance, and motivation (Guerra et al., 2016). Mastery Grids combines open learner model with social comparison, with an ultimate goal to enable the learner to be aware of their own strength and weakness, and empower them. It compares learners' knowledge level (mastery) by colored grids as shown in Fig. 4.6. The four levels portray: student's progress ("Me"), a comparison between the user and other learner in the group ("Me vs group"), group level

**Fig. 4.6** Mastery Grids dashboard visualizing learners' knowledge and progress. (Source: Personalized Adaptive Web Systems Lab)

**Fig. 4.7** A learning dashboard that supports social comparison, designed in the context of ProTuS. (From Vesin et al., 2018; licensed under CC BY-ND 4.0)

progress ("Group"), and overall progress of the learners in the class. Mastery Grids was developed in the Personalized Adaptive Web Systems Lab of the University of Pittsburgh, and is openly available to everyone (see: http://adapt2.sis.pitt.edu/wiki/ Mastery\_Grids\_Interface).

Another example has been implemented in the context of the Programming Tutoring System (ProTuS). It allows students to see how they have performed in the different parts of the course compared to their colleagues in the classroom (see Fig. 4.7). The dashboard uses the results from various quizzes associated with different areas of the content of the course. The ProTuS interface and the analytics component has been employed in the web technologies course at the Norwegian University of Science and Technologies (Vesin et al., 2018).

Research on the design, development and use of learning dashboards is at the forefront of both learning technology and CCI. With contemporary research suggesting that future generation of learning dashboards need to be actionable, tailored to the needs of end-users, responsible, confgurable, interactive, integrated and embedded in both the learning and visual sciences (Verbert et al., 2020).

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 5 Common Types of Experimental Designs in CCI and Learning Technology Research**

**Abstract** Experiments and experimental studies are used to collect data and build scientifc knowledge, and it is one of the primary methodologies for a wide range of disciplines including CCI and learning technology. In this chapter, we provide the basics of experimental research in CCI and learning technology. The goal is to support researchers to gain an understanding of the main methodological decisions and the ways in which experiments can answer RQs. Researchers will also be introduced in evaluating the circumstances that favour (or do not favour) the use of experiments, and how to make the necessary methodological decisions about the type and features of the experiment for CCI and learning technology research.

**Keywords** Experimental designs · Learning technology · Child-computer interaction

One of the main processed for evaluating an artifact (e.g., toward its indented use) and/or testing hypotheses on an artifact (e.g., UI, prototype, interaction technique) is experiment design. Experimental design is usually evaluating a particular system by means of statistical approaches. Detailed descriptions of experimental designs can be found in research textbooks and technical reports (e.g., Campbell & Stanley, 2015; McKenney & Reeves, 2018; Lazar et al., 2017; Cohen et al., 2002; MacKenzie, 2012). For the purposes of this book, we discuss four common experimental designs that CCI and learning technology researchers are likely to employ for their studies. These designs are considered "backbone" designs, in the sense that they leverage the core components that can be used to construct more complex designs. Therefore, understanding these four designs and their components will allow any CCI and learning technology researcher to also understand more complex designs, as well as to adapt and expand them to accommodate their needs.

Before moving on to the four designs, we would like to explain two core notions that will allow you to better understand them. Those two notions (or designs) are "between-subjects" (also known as "between-groups") and "within-subjects" (also "within-groups"). The notion of between-subjects is very common, and because of its roots in clinical trials, it is considered to be the gold standard of experimental research, especially when combined with random assignment of groups. The main idea is that each subject (e.g., a learner or a child) is exposed to only one condition, either the control condition or an experimental condition. Afterwards, statistical analysis investigates the difference in the variable of interest between the control group and the experimental group(s). The notion of within-subjects entails that each subject is assigned to all the treatments, in a single or repeated manner and in a specifed or unspecifed order, depending on the needs and goals of the experiment. The main idea is that the same subject should be exposed to all the treatments, which allows them to serve as "their own control group." Researchers can also combine those two designs in a mixed research design, which is, in effect, a withinsubjects design inside a between-subjects design. This enables multiple comparisons but also increases the logistics and complexity of the study (which becomes, in effect, two studies). Such combinations and extensions of the basic research designs are not necessary for understanding the core designs, and are therefore beyond the scope of this note. Figure 5.1 shows how a simple experimental design with 12 participants and control and experimental conditions would look in the case of a between-subjects design, a within-subjects design, and a mixed design.

Those two notions are very powerful in CCI and learning technology research, and knowing their pros and cons allows researchers to make good choices. It is also important to highlight that there is no right or wrong research design. Instead, researchers should consider their needs (including contextual and disciplinary requirements) and make the most appropriate choice. Table 5.1 summarizes some common decision factors to bear in mind when considering the use of between- and within- subjects designs in CCI and learning technology research.

**Fig. 5.1** A between-subjects design, a within-subjects design, and a mixed research design using the same sample of 12 participants.

**Table 5.1** Decision factors for choosing a between-subjects design or a within-subjects design.

Now that we have explained those two core notions and their inherent qualities, we can distinguish four experimental designs that are commonly used in CCI and learning technology. These are also "backbone" designs in the sense that they can be used to construct more advanced designs. First, we consider randomized experiments (also known as "true experiments") that follow the between-subjects principles and use random assignment to create the control and experimental groups. Next, we consider quasi-experiments, which are mainly between-subjects (although you might see within-subjects experiments known as "repeated measures quasiexperiments"), with nonrandom assignment of subjects. The next design, repeated measures, is a within-subjects design in which all the subjects are exposed to all the conditions. Last, we consider the time series design, which is a quasi-experiment that employs repeated measurements, with the experimental condition(s) induced between the measurement periods. We will consider each of these designs in detail, but keep in mind that they constitute a basic set of research designs that are common in the felds of CCI and learning technology, and that they can be enhanced with "advanced qualities" such as counterbalancing, placebo confederates, and deceits, as required.

## **5.1 Randomized (True) Experiments**

Randomized experiments are the ideal choice for maximizing the internal validity of a study. Their unique characteristic is that the subjects are assigned at random to a condition (the control group or the experimental group), which ensures that there are no signifcant differences between groups (Shadish et al., 2002). The random assignment eliminates any systematic error and ensures that the control and experimental groups are subjected to identical environmental conditions while being assigned to different conditions. This can be achieved by means of any random selection mechanism (e.g., a random numbers table, a random number generator app, or even tossing a fair coin).

A very simple example at the confuence of CCI and learning technology is a randomized experiment on the use of learning dashboards (i.e., a graphical user interface that visualizes students' activity) to support secondary school students. The aim is to identify any potential effect of the use of a dashboard (the independent/manipulated variable) on students' learning performance, such as their scores in weekly tests (the dependent/outcome variable). The students are assigned at random to either the control group (no use of dashboard) or the experimental group (use of dashboard). The experimental group is then exposed to the treatment (using the dashboard) for a period of time (e.g., 2 weeks). At the end of the period, we compare the learning performance scores of the two groups (Fig. 5.2).

**Fig. 5.2** Example of a randomized experiment

## **5.2 Quasi-Experiments**

Quasi means "resembling," and quasi-experiments resemble experimental settings as far as possible without assigning subjects to conditions at random. Quasiexperiments allow the researcher to set the assignment of subjects to a control or experimental condition, depending on contextual factors, the ultimate goal, and any particular needs of the population in focus, according to some criterion (e.g., class, pre-test, or previous grades). In CCI and learning technology research, random assignment may be neither feasible nor practical, and in some cases it may not be ethical. A good example is research that occurs in school settings, where it is almost impossible to formulate random groups within a class environment and expose them to different conditions (and even if it is possible, it will result in very low ecological validity). Although such contextual factors preclude the use of randomized experiment, they lend themselves to the use of quasi-experiment. For instance, the researcher can expose two similar classes to the control and experimental conditions to identify the effect of the treatment (e.g., use of a technology) on the dependent variable (e.g., learning performance).

In a quasi-experiment, biases can easily be introduced. For example, schools may be included that have students with different socioeconomic statuses and different degrees of parental support; within a school, classrooms with different teachers or different curricula can be included. Accordingly, because of the lack of randomization, quasi-experiments face certain internal validity threats, and in many cases researchers use background information (e.g., students' grades or previous performance) or pretests (or even pre- and post-tests) to strengthen internal validity. These additional processes are used to establish group equivalence and to remedy the lack of the equivalence that true experiments obtain through randomization.

In terms of the previous example, a quasi-experiment will assign class A as the control group (no use of dashboard) and class B as the experimental group (use of dashboard). The researcher can also check the average grades between the two classes, or even conduct a pre-test to make sure that there is good group equivalence on the GPA. The experimental group is then exposed to the treatment (using the dashboard) for some time (e.g., 2 weeks). At the end of that period, the researcher can compare the learning performance scores of the two groups (Fig. 5.3).

## **5.3 Repeated Measures Experiments**

A repeated measures design is a within-subjects design where all the participants are exposed to all the conditions. In practice, this means that each participant serves as their own control after being exposed to the treatment. In some cases, using the same sequence (e.g., control frst, then experimental conditions) will work. Usually, however, a stronger design will involve randomizing or counterbalancing the order so as to eliminate any potential ordering effect. An example is when the participants

need to be exposed to the same learning materials, or when they are likely to get tired, as familiarity or fatigue might affect future learning performance. Thus, randomizing the order can help us to remove any potential order bias. Counterbalancing can also be used to deal with potential order effects while reducing potential carryover effects. To achieve complete counterbalancing, we need to make sure that all the participants have been balanced across all the possible condition orders. With two conditions (control and experimental), this is a simple matter. However, when the number of conditions increases, counterbalancing becomes more complex (see Fig. 5.4), with the number of potential orderings growing at a cubic rate C^2 (with C being the number of conditions).

Returning to our previous example, a repeated measures study will expose all the participants to both conditions, probably in a randomized or counterbalanced way. For example, half of the participants will be exposed to the control condition frst (no use of dashboard) and the other half to the experimental condition frst (use of dashboard), and then this order will be reversed. At the end of the set period, the learning performance scores of the two groups will be compared (Fig. 5.5).

**Fig. 5.5** Example of a repeated measures experiment

## **5.4 Time Series Experiments**

The last type of experimental design we will consider in this chapter is the time series design, which involves repeated measurement of a group with the treatment (or treatments) induced. With the democratization of learning analytics and other user analytics (e.g., data collected from user clickstreams, keystrokes, or sensor data), this type of design is becoming more and more popular in contemporary learning technology and CCI research. The absence of randomization and distinct experimental and control groups (as in true experiments) entails all the diffculties associated with a quasi-experiment, not least the impossibility of attributing changes in the dependent variable directly to the treatment (since an improvement or deterioration in the performance of the group participating in the time series may be due to other factors, known as confounds). The uniqueness of this research design lies in the use of continuous measurements. In some cases (such as high-frequency learning analytics), this high frequency diminishes the introduction of confounding variables, many of which are introduced over time in learning technology and CCI contexts (e.g., familiarity with the task/content). Time series designs can take various forms, some of which provide better internal validity than others (e.g., multiple additions and deletions or switching replications; for more details, see Shadish et al., 2002). However, in order to conceptualize and understand this design, we can think of a single-group time series with group (G) and measurements (M) that take place several times prior to and after receiving the treatment (T).

G M1 2 M T1 3 M M4 T M 256 M ⊃

To better grasp this design, let us imagine that we have no dashboard as the control condition, dashboard type 1 as an experimental condition 1 (T1), and dashboard type 2 as another experimental condition 2 (T2). In the frst time segment (e.g., a class hour or day), a group experiences the control condition, and the respective measurements (e.g., surveys or analytics) are taken. In the second time segment, the group experiences dashboard type 1, and again the respective measurements are taken. In the third time segment, the group experiences dashboard type 2, and again the respective measurements are taken. At the end of the period, as with the previous experimental designs, the measurements taken (e.g., learning performance scores) for the control and experimental groups are compared (Fig. 5.6). The time segments can be randomized and/or repeated, and other techniques can be applied that enable us to increase the internal validity.

A time series design is appropriate for longitudinal research designs and highfrequency data collections that involve a group or groups that are measured repeatedly, usually at regular intervals. It is important to remember that although time series design is a special type of quasi-experiment that takes advantage of the qualities of time (e.g., confounds that are introduced over time, altering/repeating conditions over time), it is also vulnerable to the weaknesses of quasi-experiments. Therefore, we should interpret the results with caution. Time series designs may also have specifc weaknesses that must be addressed when analyzing the data. For example, they sometimes produce data points that are autocorrelated (for example, in very high-frequency data collection) and are therefore inadequate for certain statistical analyses (e.g., those that require independent data points; detailed information can be found in Kennedy, 1998). Conceptually, the primary concern is whether there is an exogenous infuence (a confound variable) that takes place at the same time as any of the interventions (e.g., new or signifcantly different content, alterations to the instructions, or a bug in the system), or whether there are signifcant differences in the sample or an environmental condition (e.g., dropouts, fatigue, or changing classrooms).

**Fig. 5.6** Example of a time series experiment

In the past, time series design was mainly employed to detect unstable behavior patterns, and we see relatively few studies using this type of design (Ross & Morrison, 2013). Another reason for the limited use of this research design is the signifcant demands of longitudinal studies and prolonged involvement of human subjects. Moreover, we often fnd time series designs included under the umbrella term of "quasi-experimental design" (Campbell & Stanley, 2015).

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 6 Data Collection and Analysis in Learning Technology and CCI Research**

**Abstract** Conducting experimental studies in learning technology and CCI research entails an iterative process of observation, rationalization, and validation. Although data collection and data analysis procedures may vary widely in complexity, their selection is based on the research objectives, RQs or hypotheses. So the researchers need to carefully select them and make sure that the research design decisions of data collection and analysis, are adequate for the goals of the study. This chapter provides information on the various data collections and analyses that are usually employed in learning technology and CCI research. This chapter is intended to serve as a guide for CCI and learning technology researchers, and help them deciding what data they need to collect and how they should analyze them to address the goals of their study.

**Keywords** Learning technology · Child-computer interaction · Data collection · Data analysis

As mentioned in the introduction, conducting experimental studies in learning technology and CCI research entails an iterative process of observation, rationalization, and validation (see Fig. 6.1). More detailed processes with additional steps, such as conducting a literature review, have been proposed (e.g., Ross & Morrison, 2013). Nevertheless, no matter how detailed description we have, determining, conducting, and reporting the data analysis is fundamental. Although data analysis procedures vary widely in complexity, selection of the appropriate analysis is usually based on two aspects: the RQs/hypotheses and the type of data involved. To clarify the process, Fig. 6.1 shows the steps typically needed to determine the process and conduct the data analysis.

**Fig. 6.1** Typical process for determining and conducting data analysis in learning technology and CCI studies

## **6.1 Data Collection**

There are different ways that researchers collect data. Whether it is qualitative, quantitative or mixed research design, researchers need to collect data that are going to support the rationalization of the study (e.g., respond to the hypotheses or the RQs). In particular, in human-factors IT-related felds, we usually see different quantitative (e.g., log fles/analytics, questionnaire data, sensor data) or/and qualitative (e.g., interviews, feld notes) data collections taking place. Although it is possible to follow some of the principles described in this section with qualitative data as well, through different forms of data quantifcation (e.g., annotations, text mining, expert analysis); most of the practices described here concern quantitative data collections. In several cases those data collections are associated with specifc measurements (which are associated with the RQs), in some data collections the measurements are predefned (e.g., questionnaire data, some log fles) in some other data collections the measurements are post-computed (e.g., from sensor data), and in some other data collections there are no measurements (e.g., this is common in qualitative research studies). In this section, we will see some example data collections that are relevant for a learning technology and CCI researcher.

**Questionnaire Data (Also Known as Survey Data)** The use of questionnaires (also called surveys) has a long history in both HCI and learning technology research. The goal is to understand users/learners attitudes and perceptions toward an artifact, or/and a procedure. Questionnaires are also allowing us to gather information about users' backgrounds (e.g., habits, technology use), demographics and awareness. Questionnaires have been used for several years across different felds, such as social psychology, behavioral research and marketing, and can be put into practice in a pen and paper form or as a part of the system (e.g., integrated questionnaires). Several standardized questionnaires have been developed to gather information about system's perceived usability (e.g., System Usability Scale (SUS) (Brooke, 1996), Computer System Usability Questionnaires (CSUQ) (Lewis, 1995)), users' perceived effort (e.g., NASA Task Load Index (NASA-TLX) (Hart & Staveland, 1988)), and users' attitudes and perceptions (e.g., perceived usefulness,

perceived ease of use (Davis, 1989). Questionnaires are a direct means of measuring users' perceived experience such as satisfaction, enjoyment, ease of use, with many of them having high level of standardization in HCI research (e.g., satisfaction is part of the ISO 9241). In the same vein, questionnaires are systematically used to assess learning experience, several questionnaire instruments have been developed and widely used in the past (e.g., to evaluate a learning system or different aspects of the learning design) (see: Kay & Knaack, 2009; Henrie et al., 2015). Questionnaires is a commonly accepted measure of users' and learners' experience (at least the perceived one), and despite some criticism (e.g., overuse or overreliance on questionnaires), questionnaires will probably continue to be a valid approach for externalizing and quantifying users' perceived experience. Below in Fig. 6.2 you can see two standard questionnaires for measuring system's usability (left) and users' mental effort (right).

In this book we are not going to have a deep discussion on the role of questionnaires in IT related research, such a discussion can be found in Müller et al. (2014) and Groves et al. (2011). However, we will briefy discuss how questionnaires can help us collecting useful data and what are the most common measurements we see in learning technology and HCI research. The most common conceptual constructs (measurements) are multi item (multi questions), so several similar question are used to construct the measurement of the construct. In most of the cases, they are measured using Likert scales (e.g., fve or seven point are the most common) and the wording of the scales can be confgured to match the question.1 Although no strict requirements exist, in large scale studies (usually survey studies) we see expectations for ten respondents per item (question). In experimental designs we see studies with less respondents per item. However, the researchers need to be considerable of the "ecology" of the measurements (e.g., should have a manageable number of questions that allow the user to understand, refect and respond). Beyond the care the researchers need to pay during the research design of a study, there are also procedures for assessing the convergent validity of questionnaire measurements used in a study, for instance Fornell and Larcker (1981) proposed the following three routines, composite reliability of each measurement, usually Cronbach α above 0.7; item/question reliability of the measure, usually factor loading of 0.7 and above for each question (with no cross loadings) is a good indicator; and the Average Variance Extracted (AVE) of the measure, usually it is expected that AVE is equal or exceeds 0.50. In the following table (Table 6.1) we provide some examples of commonly used (in learning technology and HCI) measurements. Those items are properly contextualized and provided as options to a general question such as "Please indicate how much you agree or disagree with the following statements based on your experience with [the artifact]:". Whenever [the artifact] the researchers can use the artifact of interest (e.g., the XYZ mobile application, the avatar, the dashboard).

<sup>1</sup>Examples of Likert Scaled Responses Used in Data-Gathering: https://mwcc.edu/wp-content/ uploads/2020/09/Likert-Scale-Response-Options\_MWCC.pdf

Graded SUS questionnaire demonstrating how the score for each of item of the SUS is determined with odd questions scored 0–4 based on the 1–5 selection, and even questions scored 4–0 based on the 1–5 selection.

The total score for this example is 26. Multiplying 26 \* 2.5gives us a SUS score of 65. From a meta-analysis studies weknow that 68 is the average and your system needs to scoreabove 80 to be on the top 10% of scores. This is also the point where users are more likely to be recommending it to a friend.

high); 50-79

(high); 80-100

(very high).

**Fig. 6.2** Standard questionnaires for measuring: system's usability (SUS; Brooke, 1996) (left). (Image from Klug, 2017; licensed under CC BY-ND 3.0).; mental effort in a task (NASA-TLX; Hart & Staveland, 1988) (right). (Image by https://commons.wikimedia.org/wiki/File:NasaTLX.png)

**Table 6.1** Examples of measurements, their description, their questions and the respective reference

(continued)

**Table 6.1** (continued)

**Analytics (Also Known as User Logs)** In the fourth chapter we discussed about user traces that are left behind when users interact with technologies, and the implications those traces have for learning technology and CCI research. Those traces produce a wide range of insights, including users' response time, response correctness, number of attempts to solve a problem, time spent interacting with learning resources, navigation to various learning resources, activity on the various communication functionalities (e.g., forums), and other learning trace data. Besides the ways systems' can develop intelligence when leveraging on these data, such data can also be used to enrich measurements when conducting experimental studies. As we discussed in the fourth chapter tracking logs are powerful (you can see examples from edX MOOCs here2 ) and can help us to infer useful measurements, see services that host and provide access to learning interaction data such as Pittsburgh Science of Learning Center's DataShop (https://pslcdatashop.web.cmu.edu/). Although a perfect one-to-one relationship between "measurements" and "conceptual constructs" is practically impossible, we see that very close relationships (i.e., analytics that capture the target construct to a great extent) exist and are heavily used to CCI and learning technology research (e.g., learning performance that is defned as the scores of the user in the assessment tasks). This allows us to capture those useful measurements intuitively (e.g., via the log fles). Although such measurements can be post-computed from the tracking logs of the technology and the respective database schema; it is also possible and signifcantly more practical to "architect" analytics when designing and developing the technology. By architecting the analytics, you can develop relational database schemas that organize the data with respect to your needs and meaningful measurements (e.g., see Pardos et al., 2016), architecting analytics is also powerful when you have to work with learning eco-systems, where analytics across systems need to be captured and make sense (Mangaroska et al., 2021). The use of analytics in measurements during experimentation is an

<sup>2</sup>EdX guide: https://edx.readthedocs.io/projects/devdata/en/stable/internal\_data\_formats/tracking\_logs.html

interesting and complex topic. The goal of this book, is not to go deep in this topic, but provide some examples of commonly used analytics based measurements in the context of learning technology and CCI (see Table 6.2). These selection of those measurements needs to take into consideration the context of the study of the technology and be relevant with the intended RQ.

**Table 6.2** Examples of analytics-based measurements, their description, how they are usually computed and an example of their use from the literature

**Sensor-Based Analytics (Sensor Data)** Advances in sensors, social signal processing and computational analyses have demonstrated the potential to help us understand user and learning processes which were either not-possible to be captured or "too complex" for traditional analytics. For example, psychomotor learning with physical objects needs high frequency data and analyses can now happen in a reasonable time-window (Sharma & Giannakos, 2020). Due to the need for combining different expertise (e.g., learning scientists, data scientists, computer scientists), the collection, analysis and interpretation of sensor data in CCI and learning contexts have been a challenging endeavor. Nevertheless, over the last years the Multimodal Learning Analytics (MMLA) research community has managed to gather diverse research expertise's (e.g., educational, computational, psychological), and contributed with rich measurements with respect to HCI and learning. A perfect one-to-one relationship between sensor-based measures and conceptual constructs does not exist (Giannakos et al., 2022), however, MMLA research is achieving acceptable levels of reliability and validity, allowing us to use measurements that provide useful insights (e.g., from eye-activity, facial expression or users' motions and gestures). Table 6.3 depicts some examples of commonly used sensor based measurements in the context of learning technology and CCI. Once again, the selection of those measurements needs to take into consideration the context of the study of the technology and be relevant with the intended RQ, moreover, the researchers also need to consider the level of intrusiveness (the extent to which a measurement is ecologically valid, e.g., does not interfere with the task or impose obtrusive conditions). In different sub-domains of learning technology and HCI, we see researchers coining measurements that align with the objectives of those sub-domains. For example, in the context of Computer Supported Collaborative Learning (CSCL) research, we fnd researchers using a measurement called Joint Visual Attention (JVA) (i.e., the moments more than one users look at the same area) or "with-me-ness" (i.e., the moments the learner is looking on the content delivered by the teacher, e.g., how much the learner follows the teacher), although those measurements are not as general or widely used as the ones we identify in Table 6.3, they are very important for the challenges of this particular subdomain (Sharma et al., 2014, 2017).

**Pictorial Self-Report Data** Traditional verbal questionnaires assume that respondents are able fully grasp a question and think abstractly about their experience. However, several populations (e.g., children younger than 12) have not yet developed these skills or are in conditions that do not allow them to respond those instruments in a valid manner (e.g., a user who has dyslexia or is very tired from the main task); instead, their thinking processes are based on mental representations that relate to concrete events, objects, or experiences. This must be taken into account when adapting the measurement method to meet participants' needs. Following this line of reasoning and related work in child development and psychology (Harter & Pike, 1984), there is an number of instruments that use visual methods (or observations and qualitative, checklist-based measurements), which we know are more

**Table 6.3** Examples of sensor-based measurements, their description, how they are usually computed and an example of their use from the literature

a Stress can also be measured by combining measurements (e.g., temperature, heart rate, galvanic skin response), see: Basjaruddin et al. (2021)

effective than verbal methods (Döring et al., 2010). Such visual analogs represent specifc situations, behaviors, and people to whom a user can easily relate.

Such visual analogs are usually employed to collect data during evaluation of an artefact (as well as during the lifetime of an application). We have seen pictorial questionnaires popping up while we are using an application or at the end of an activity (e.g., after we try a resource that has been recommended to us). Similarly with the verbal questionnaires, pictorial questionnaires are used to qualtify users' perceived experience such as satisfaction, enjoyment, ease of use and alike. Although pictorial questionnaires usually do not follow the multi item (multi question) paradigm of the verbal ones (so the validity is not always being assessed), however, it is easier to employ pictorial questionnaires "on the spot" and capture temporal experience of the users. Moreover due to their usually short reading time, it is also easy to employ them either in selected critical moments (when the user fnished a task) or in a random manner during the activity so we can get repeated measurements (Fig. 6.3).

Pictorial questionnaires are not meant to substitute verbal questionnaires, those two types of self-reporting instruments have been designed to address different research needs. Verbal questionnaires can use the specifcity of verbal communication to extract exact information and the widely used measurements have been extensively validated and standardized. Pictorial questionnaires are used when "verbal communication becomes a challenge" and have the benefts of not increasing users' cognitive load and overall burden, and reducing the time-to-complete. Similarly with verbal questionnaires, the pictorial questionnaires should be properly contextualized and sometimes complemented with minimal text such as "what do you think about [the artifact]:". Whenever [the artifact] the researchers can use the artifact of interest (e.g., the XYZ mobile application, the avatar, the dashboard). Nevertheless, pictorial questionnaires should be self-standing, even if the user cannot read the provided text, depending the end-user, sometimes researchers need to use oral communication to explain what aspects we are asking the end-user to rate with the visual analogs. Similarly with verbal questionnaires, pictorial questionnaires can be used in both pen and paper and in a digital version, however, some of the advantages of digitally administering pictorial questionnaires to assess software (e.g., temporality, overall burden) might be lost or weakened. Table 6.4 depicts

**Fig. 6.3** Three examples of pictorial surveys used to evaluate users' experience

Examples of pictorial survey measurements, their description, how they look like and the respective reference

(continued)

Measurement

Description

Exemplar

References

emotions) bBy using it in a pre-post manner, the child can give a score for the expected (pre) and the experienced (post) fun. Comparing these two scores allows you to see if children have had a good experience (more info in the CCI UX Playbook: https://chici.org/ux-playbook/) some examples of commonly used pictorial questionnaire measurements in the context of learning technology and CCI.

## **6.2 Data Analysis**

To make the process clearer and to provide additional resources, Table 6.5 summarizes the most common data analysis procedures used in learning technology and CCI research. Let us now think of a simple between-subjects design with one control group (e.g., no use of dashboard in the LMS) and one experimental group (e.g., a simple dashboard that provides students' previous test scores), with students' weekly test scores as the dependent variable. In this case, a t-test for independent samples is needed (provided that parametric assumptions are met) to test the hypothesis that introducing a simple dashboard affects students' learning performance. Adding a second experimental group (i.e., a third treatment group) with a dashboard that not only provides but also visualizes students' scores will require a different analysis. In that case, we will need a one-way analysis of variance (ANOVA) (provided that parametric assumptions are met) to compare the three means; if the results of the ANOVA are signifcant, we can conduct a follow-up Tukey or REGWQ post-hoc comparison of means to fnd the pairwise differences. Learning technology and CCI researchers do not have to be data analysts or statisticians, but it is important to provide clear RQs and hypotheses and to follow a few basic rules and guidelines during the data analysis. Clearly formulated RQs will also make it possible to work with data analysts or statisticians if more sophisticated analyses are required that go beyond the scope of this book.

Most of the studies in learning technology and CCI employ the null hypothesis signifcance testing (NHST)3 approaches and analyse data using the variance-based methods we present in Table 6.5. Despite the usefulness of variance-based approaches we have seen an increasing the need for new methods as well as combinations of different methods and approaches that can reduce biases and help us obtain a more holistic understanding of the phenomenon. Examples of such methods that we see being increasingly used in HCI and learning technology/ analytics are Bayesian methods (Robertson & Kaptein, 2016), fuzzy-set qualitative comparative analysis (fsQCA, or simpler versions of it such as QCA) (Pappas et al., 2019; Papamitsiou et al., 2018), process mining (Sharma et al., forthcoming), Hidden Markov Models (Sharma et al., 2020), or different machine learning methods (Kidziński et al., 2016).

As mentioned above, to support learning technology/CCI researchers, we provide a comprehensive how-to guide that allows them to choose between the various analyses by looking at the data types, the function of each of the analyses, working examples, the main conditions and assumptions, and resources for step-by-step

<sup>3</sup>NHST is statistical inference by which an experimental factor is tested against a hypothesis.

**Table 6.5**Comparison of data analysis procedures in learning technology and CCI research (continued)

6.1 Data Collection

Assumptions

Does it

test

Open resourcesb (Based on

Field, 2018, for IBM SPSS

causal

63

any value of X), independence (observations are independent of each other), and normality (for any fxed value of X, Y is normally distributed)bThe resources provided are for IBM SPSS statistics software, but similar resources are available for R (e.g., Field et al., 2012), Python (e.g., Haslwanter, 2016), and other statistical analysis software. The statistical language/tool used does not affect the information provided in this table or the outcome and reporting of the analyses

**Table 6.5**

(continued)

implementation. Novice researchers should be aware that in order to explore causal relationships (cause-and-effect relationships) on the basis of experimental designs that compare outcomes associated with treatments, it is necessary to use tests that test causal effects (e.g., t-tests or ANOVAs) rather than correlational tests (e.g., Pearson correlations).

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 7 Reporting CCI and Learning Technology Research**

**Abstract** Designing and conducting high quality research is extremely important in CCI and learning technology research. However, the same high-quality needs to be followed during the reporting of the work. At the end of the day, this what the reviewers and readers will credit – therefore, reporting is of equal importance. This chapter provides information on how you should structure your article and the information that is usually required. This chapter is intended to serve as a template for CCI and learning technology researchers. Moreover, I have summarized some recommendations based on my experience, as well as on published guidelines and recommendations from neighboring felds.

**Keywords** Learning technology · Child-computer interaction · Guidelines · Reporting research

For CCI and learning technology research to be disseminated, it is important that it be effciently reported (and published) in various proceedings, and journals. Reporting CCI and learning technology research has many similarities with other human-factors IT-related felds, such as game technology, computing education research, HCI/interaction design, and software engineering. This is due to the fact that most of these felds rely heavily on the guidelines of the American Psychological Association (APA) for reporting. The recommendations of this section are based on the authors' experience, as well as on published guidelines and recommendations from the relevant felds (e.g., Ko et al., 2015; Recker, 2012; Wobbrock, 2015; Ross & Morrison, 2013) and the APA journal article reporting standards (JARS) for reporting quantitative research (reporting the fndings of a study using numeric representations),1 qualitative research (reporting the fndings of a study using narrative representations),2 and mixed methods research (reporting the fndings of a

<sup>1</sup>Quantitative Design Reporting Standards: https://apastyle.apa.org/jars/quant-table-1.pdf

<sup>2</sup>Qualitative Design Reporting Standards: https://apastyle.apa.org/jars/qual-table-1.pdf

<sup>©</sup> The Authors 2022

M. Giannakos, *Experimental Studies in Learning Technology and Child– Computer Interaction*, SpringerBriefs in Educational Communications and Technology, https://doi.org/10.1007/978-3-031-14350-2\_7

**Fig. 7.1** Typical structure for a learning technology or CCI paper/report

study using both narrative and numeric representations).3 Therefore, independently of the statistical analysis software or data science programming language you use, your reporting must follow a standard structure and include the information indicated in the conventions. Several resources are available, including the JARS website (https://apastyle.apa.org/jars), which includes details about how to report your results for a wide range of research designs and statistical analyses. There is also a summary of how to report the results of your statistical analyses provided by Dr. Jeffrey Kahn from Illinois State University (https://psychology.illinoisstate.edu/selandau/ ReportingStatisticsinAPAStyle.html). This section provides a brief overview of how to report CCI and learning technology research. Figure 7.1 shows the high-level structure of a typical paper/report.