In this study, context is presented as a narrative prior to the experiment and later enhanced by the simulation that situates participants into the RSD procedure and invites them to take the role of an asylum seeker. To provide stimuli, the study uses references from the list of Affective Norms for English Words (ANEW) [19] for phrasing the emotionally evocative questions. The questionnaire comprises a hundred closed questions, sixty-five percent of which are assumed to be emotionally evocative. The simulation ultimately intends to benefit from the physiological responses generated over the enunciation of the sensitive questions and does not have follow-up inquiries.

Accordingly, as shown in Fig. 2, a question is first enunciated by the virtual officer and presented as text on the screen. Once that is completed, participants are taken to the self-report screen and asked to select from the wheel the label and intensity of the emotion they experienced. After completing the selfassessment, participants are presented with the answer options. Finally, after answering that question, participants are presented with a new one. Randomization of the questions was not included in the design of the simulation.

**Fig. 2.** Screenshots illustrating the loop of tasks through which participants labeled the emotional state experienced for each question presented by the virtual officer

As the experiment design establishes no fixed time for completing the simulation, every time a new question is presented to participants, a mark is created in the database to inform the start of the question. The approach is justified for not inducing pressure on participants during the simulation play.

#### **3.2 Procedure**

It is worth remarking that the experiment was piloted twice to test out a) the simulation setup and questions structure and b) data collection methods. After fine-tuning the latest adjustments, the experiments were carried out in laboratory settings under uniform conditions. Figure 3 illustrates the experimental procedure designed for the study.

Experiments were conducted during the COVID-19 pandemic; thus, the necessary safety measures stood as part of the procedure. At first, participants answered a BIS/BAS questionnaire [20] to analyze the symmetry of personality in terms of the motivational systems underlying behavior and affect responses. When conforming, participants were considered apt for the experiment. Later, information regarding the experiment, the purpose of the study, and the technical apparatus was provided. Participants were exposed to the contextual narrative upon signing the consent form. Following, the sensors were connected, as it is shown in the Data Collection section below. Before engaging in the simulation, participants were invited to perform a simple breathing exercise. It is noteworthy that, while playing the simulation, participants were alone, and monitored from a nearby room via a camera streaming setting.

#### **3.3 Participants**

Seven (7) English-speaking individuals currently residing in Tallinn, Estonia, participated in the study. Participation was voluntary, and the age range of

**Fig. 3.** Stages of the experimental procedure

participants was 18–40 years old. Even though the broader context of the study refers to the vulnerable population (refugees), no actual refugee applicant or any other vulnerable group was included in the study sample. Nevertheless, recruitment embraced participants from foreign countries that would be more likely to have a reference of the border control inquiries.

#### **3.4 Data Collection**

The simulation embeds a data collection procedure that synchronizes participants' physiological data with their emotional self-reports. The goal is to enable the participant to experience and disclose emotional states during the physiological data acquisition without disruption or further interference.

**Physiological Data.** Participants' physiological signals are gathered via BITalino Plugged kit, a low-cost biosignal acquisition device [27]. The sampling frequency of the used ECG is 1,000 Hz, as recommended by [28]. EDA and fEMG measurements are collected at the same sampling rate due to using a single BITalino device to collect the three physiological at the same time. The BITalino device was connected to a high-performance computer via Bluetooth. Participants' physiological information thus is collected from three channels - ECG, EDA, and EMG - and continuously recorded using the compatible software OpenSignals1.

Sensors were placed on the clean surface of participants' skin, following recommendations of reviewed literature [22,27]. Figure 4 illustrates the configuration used in the experiments. Two electrodes were used to measure EDA, placed on the thenar and hypothenar eminence of the right hand as illustrated in Fig. 4a.

1 Available at https://support.pluxbiosignals.com/knowledge-base/introducingopensignals-revolution/

**Fig. 4.** Electrodes placement for biosignal data collection

For collecting ECG, three electrodes were placed on the participant's chest, two on both sides of the clavicles and one at the lower left part of the rib cage, as it is illustrated in Fig. 4b. Additionally, facial muscle activity (fEMG) was acquired via three electrodes placed on participants' face, the reference electrode in the middle of the forehead and the two others on the Corrugator Supercilli and Zygomathicus Major, as in Fig. 4c.

**Emotion Self-report.** To collect the emotional self-reports of experiment participants on the go, the interface simulation uses a digital tool based on the self-report paradigm proposed by the Geneva Emotion Wheel (GEW) [2]. Selfreport paradigms are used to grasp the individual nature of emotional experience, which reflects the integration of mental and bodily changes in the context of particular events. Researchers in emotional labeling point out that even though the results obtained by these paradigms are plausible and interpretable, the statistical analysis is hampered by the abundance of emotion labels collected, making the interpretation complex [2]. This way, embedding the GEW into the simulation allowed the emotional assessment to be done interactively through an adapted version of the tool and provided homogeneity of reports. The choice also benefits the processing and synchronization of physiological data collected through the sensors.

#### **3.5 Preprocessing Data**

The multisource dataset required a composite of processing and analysis methods. Constraints faced at this stage caused the removal of the fEMG data from analysis. At first, the ECG and EDA data was downsampled from 1000hz to 100hz to reduce computational overhead. Thereafter, filtering to remove noise from the ECG and EDA data was done automatically as part of the Neurokit2data preprocessing pipeline [18]. Furthermore, we divided the ECG and EDA data into epochs with a duration of 10 s, starting one second before the question presentation and finishing nine seconds after that. Compiling the information into epochs returned a list of numeric identifiers for the physiological reactions regarding each question. The 10 s epoch duration was based on the calculation of the participant's question average time (Max: 12 min 32 s, Min: 08 min 15 s). Hence, each participant produced 100 data samples, thus giving us a total of 700 data samples. Two features were extracted from ECG data, ECG Rate, and ECG R Peaks. From EDA data, we obtained EDA Phasic and EDA Tonic features. All these were achieved using the Neurokit2 python library [18].

#### **3.6 Analysis**

Following the dataset containing the labeled psychophysiological data of the sample (N = 7) was statistically analyzed, focusing on features of the physiological measurements acquired alongside emotional self-reports. Independent t-tests (performed two-sided) were used to draw comparisons between the emotional reports and their specific physiological reactions. A P value of 0.05 was assumed to indicate the statistical significance of the tests. Furthermore, correlation tests were performed to observe the relationship between the physiological data features and the emotion labels indicated by the participants. The statistical analysis was performed using SPSS software (ver. <IP_ADDRESS>).

## **4 Results**

Among the negative valence emotions, the most reported emotion was Disappointment, representing 12.9% (n = 90) of the total answers. Fear was reported in 9.4% (n = 66), followed by Sadness in 9.1% (n = 64) of the answers gathered. Among the positive valence emotions, Relief was the most reported, chosen at 8.0% (n = 56) of the time, followed by Interest, reported at 6.4% (n = 45), and Pride, 4.0% (n = 28). Issues with the self-assessment led the analysis not to include the variations of intensity reported by participants. Manifested differences between the data across participants hampered a separate analysis of the emotional reactions, leading the study to consider the emotions between negative and positive valence.

The analysis has shown that the negative stimuli increased cardiac activity in participants compared to positive stimuli. The mean of the ECG Rate for Positive emotions was M = .6452 (SD = .6760), and for Negative emotions, it was M = .6645 (SD = .7092). However, no significant difference was found in the ECG Rate between the emotions of positive and negative valence (T623 = *−.*330, p = .741). This way, tracing general conclusions about participants' emotional arousal through examination of cardiac activity was unattainable.

A similar issue was found in the features of EDA data. Measurements of the EDA Tonic were scattered and not pronounced. The little difference between the EDA Tonic values could indicate that the quick flow of questions had piled up different stimuli and affected the measurements in the tonic component, which is distinguished for happening slowly over time. On the other hand, measurements of the EDA Phasic suggested that negative emotion might have increased electrodermal activity response if compared to the positive reports. The mean of EDA Phasic for Positive emotions was M = *−*1.5396E*−*4 (SD = 1.243E*−*3), while Negative emotions had a mean of M = *−*4.9404E*−*4 (SD = 1.859E*−*3). Additionally, the parametric test revealed a significant difference in the means of EDA Phasic between groups of positive and negative emotions (T601.852 = 2.728, p = .007).

Furthermore, Pearson's correlation tests were performed to observe the relationship between the EDA and ECG measurements. Results have indicated that ECG Rate and EDA Phasic have a significant, weak negative correlation (r = *−*.316, p*<*.001). In contrast, ECG Rate and EDA Tonic have a significant moderate, positive correlation (r = .603, p*<* .001). Such findings can indicate measures to consider in future studies.

## **5 Discussion**

In an experimental investigation, we showed that our prototype enabled the simultaneous collection of participants' quantitative and qualitative affective information during a simulated interview. This pilot study allowed us to identify what needs to be iterated for the upcoming trials of this study. Moreover, the process of the study illustrated the complexities entangling psycho-physiological studies in HCI. The emotion elicitation method and designed data collection interface had a positive outcome. However, generalized conclusions regarding associations of self-reported emotional experience and the two features of physiological measurements, EDA and ECG, were not attainable.

The analysis of the ECG and EDA measurements across different emotion reports did not allow the distinction of specific emotions experienced by the participants. Still, the results aligned with the previous studies showing that negative emotions are characterized by strong cardiovascular and electrodermal activity [12]. Besides, the results suggested that combinations of measures of ECG Rate could be used alongside EDA Tonic or EDA Phasic to detect positive and negative emotions.

The interview context was appraised negatively among participants, as Disappointment and Fear were the most reported emotions among the responses. For that, the fictitious scenario was considered enough to situate the participants in the research context and allow them to identify with the emotional situation. Besides, the different sorts of appraisal triggered by the simulation suggested that using words from the ANEW list [19] had contributed to the necessary stimuli and evoked different assessments of the given stimuli.

### **6 Conclusion**

In general, emotion recognition is based on automated classifiers trained to identify general emotional states from patterns found in the users' biosignals, whether explicit (i.e., from facial expressions and gestures) or implicit (i.e., from biosensors acquisition). Converged tasks are required for achieving accuracy and reliability in such systems [11,14,22]. HCI practitioners and designers should take advantage of physiological and affective computing methods to develop accurate emotion recognition models. This study adds to the evidence that simulated settings could be used for eliciting emotional reactions, especially if the content of the simulation is framed through validated emotion elicitation approaches, such as the affective words of the ANEW list [19]. The findings of this pilot pave the way for novel methods that enable the understanding of emotional experiences in simulated interviews. We envision applications of the methods for assessing the user experience's emotional aspects with other interactive systems and different contexts.

**Limitations and Future Studies.** We acknowledge that emotion is a subjective context-dependent experience bound by different factors, whereas the individual appraisal of the situation, culture, age, and gender, among other aspects. Thus, the specificity of each individual appraisal and reaction to the emotional stimuli makes the generalizations about the relationships between the perceived affective information and the physiological reaction measurements very complex. Future research should account for the different physiology of people. Moreover, calibrated individual measurements could help to investigate bodily reaction patterns. Adjustments to experiment settings should cover the definition of a fixed length for the stimuli, the review of the simulation content, and the possibility of a less extensive experiment setting. Also, a larger sample size would be required. Refinements could also approach developing the necessary pipelines and testing different data processing methods, feature extraction, and analysis to achieve generalization. Trends among the data could be analyzed in future research by classification and clustering. Future studies would also benefit from using an alternative experimental design to test the influence of the specific aspects of the simulation.

**Ethical Implications.** Recognizing the emergent risks connected to the automation of refugee status determination assessment is necessary. The novelty of the IVAs may serve as a tool for facilitating the initial screening at the border control stations. Nonetheless, the emotion monitoring for the adaptability of such systems should be made explicit to the users before the interaction. The access to a person's mental and emotional states may be seen as invasive and place users in a vulnerable position, besides compromising the levels of trust in such interactions. Foremost, the assessment of implicit biosignals should not serve surveillance purposes nor deception detection.

**Acknowledgments.** The work by Debora C. Firmino De Souza and Pia Tikka has been supported by the EU Mobilitas Pluss Top Researcher grant awarded to Pia Tikka by the Estonian Research Council (MOBTT90).

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **"NAO Says": Designing and Evaluating Multimodal Playful Interactions with the Humanoid Robot NAO**

Ilona Buchem(B) , Lukas Brömmling , and Niklas Bäcker

Berlin University of Applied Sciences, Berlin, Germany<EMAIL_ADDRESS>
**Abstract.** NAO is a small humanoid robot which affords multimodal interaction through speech, non-verbal sounds, visual pattern recognition, gestures and touch. NAO can be animated to move its head, arms, legs in space and to manifest emotional reactions through dialogue, sounds, body movements and light effects. This paper reports on the design, implementation, play-testing and evaluation of a multimodal, playful interaction with NAO in two pre-studies and one pilot study with altogether 209 participants of all age groups. The application "NAO says" was designed based on the popular imitation game "Simon says", in which three or more players follow the command "Simon says". In "NAO says" the robot plays the role of Simon and asks the players to play a series of mini-games by imitating body movements and solving simple mathematical riddles. The design of "NAO says" focuses on creating an experience of a less constrained, playful interaction rather than following strict rules of the game. The paper describes the design of the game, the implementation in pilot studies and the results from three evaluations which investigated the perceptions of NAO as a game leader, and perceived psychological stress before and after the playful interaction with the robot. The results indicate that the robot was perceived as a friendly, joyful and pleasant interaction partner and that perceived stress was lower after playing the game.

**Keywords:** Human Robot Interaction (HRI) · Playful interaction · Interactive games · Humanoid robots · Social robotic game · Stress reduction

## **1 Introduction**

Humanoid robots offer new opportunities for playful interactions with humans. Playful interaction design in Human Computer Interaction (HCI) is rooted in the perspective of humans playful creatures or as "Homo Ludens" engaging in playful, ludic activities, which take place within fixed limits of time and place, and according to freely accepted but binding rules [1]. Playful activities absorb the players and evoke intense feelings of joy and tension or excitement, bringing the players beyond the experience of the "ordinary life" [1]. The distinction between playful interactions, serious games and gamification has been made in relation to the non-utilitarian character of play [2]. Playful interactions have been associated with such aspects as curiosity, exploration and the experience of wonder [3], as well as with an inherently pleasurable experience [4]. Research studies explored the possibilities and effects of playful interaction in the field on Human Robot Interaction (HRI) including educational robotics. For example, [5] developed an educational robotic system with a driving robot and a programming-board with command-bricks to support tangible, social and playful interactions in context of school education. Other studies with children, focused on the potential of playful humanrobot interaction for learning and cognitive development. For example [6] designed playful child-robot interactions for language learning and the results showed promising effects. Studies have shown that playful interactions with technologies can be beneficial not only for children but for human players of all ages. Playful activities develop their potential in physical and social interactions especially through the engagement of the whole-body of all players [7]. These aspects, especially physical and social aspects of playful interactions including whole-body engagement, have played an important role in the design of the application "NAO says" presented in this paper.

The application "NAO says" is based on the popular imitation game "Simon says", in which three or more players follow the command "Simon says" and act on the followup task. If, however, the game leader Simon does not say "Simon says" but a player acts according to the task, this player must quit the game. In "NAO says" the humanoid robot NAO plays the role of Simon and asks the players to engage in playing a series of mini-activities by following the command "NAO says". NAO is a small humanoid robot which affords multimodal interaction through speech, non-verbal sounds, visual pattern recognition, gestures and touch. NAO can be animated to move its head, arms, legs in space and to manifest emotional reactions through dialogue, sounds, body movements and light effects. The mini-activities include imitating body movements of the robot and solving simple mathematical riddles while playfully interacting though speech and touch. These playful interactions are embedded in a social context and include a group of players following the commands of the robot together at the same time. "NAO says" as a design-led intervention aims to create a pleasurable experience through playful human-robot interaction embedded in less constrained settings, without the pressure to follow strict rules of a game.

This paper reports on the design, programming, implementation, playtesting and evaluation of this multimodal, playful interaction with NAO in two pre-studies and the main pilot study with altogether 209 participants of all age groups. The reminder of this paper is structured as follows. Section 2 outlines the design and the programming of the application "NAO says" with references to the different versions of the game "Simon says". Section 3 describes the design of the studies themselves, i.e. two pre-studies with university students and the main pilot study, and the research methods applied in these studies. Section 4 presents the results from the two pre-studies and the main pilot study focusing on players' perceptions of the NAO robot as game leader and the game "NAO says", and perceived levels of stress before and after playing the game. The paper ends with conclusions and recommendations for further research.

### **2 Design and Programming**

The application "NAO says" was designed to afford playful interactions with the humanoid robot NAO and was inspired by the popular imitation game "Simon says". The game "Simon says" is structured around playful interactions in a group of players, who follow the command "Simon says", engaging in a series of playful mini-activities. These mini-activities in the "Simon says" game vary from game to game version, but usually include physical, cognitive and social components which allow to engage gaze, speech, and motion. "Simon says" game has been developed and applied in many different versions. The classic version of the game involves human players playing in the physical room and one of the players taking the role of Simon. Studies on "Simon says" with human players explored its effects for learning outcomes, especially in language learning. The study by [8] showed that playing "Simon says" had a significant effect on listening comprehension of senior high school students. The study by [9] showed that the game improved vocabulary mastery in learning English.

More recently, different versions of the "Simon says" game have been developed using technologies to support the gameplay. For example, [10] developed the "Simon says" game as a mobile application for mobile phones with commands focusing on color identification. A study in the context of long term care (LTC), applied a "Simon says" activity with a robot, in which older adults took turns as leaders [11]. The researchers concluded that robots are promising for social engagement of older adults who suffer from apathy [11]. Another version of the "Simon says" game was developed with a humanoid social robot and included a computational model of turn-taking to support a more natural interaction during the gameplay [12]. Finally, the study by [13] focused on bodily movements and implemented the human pose detection library OpenPose to capture players' poses [13].

#### **2.1 Design**

The design of the "NAO says" game presented in this paper focused on the multimodal and multi-sensory playful interaction of the NAO robot and human players of all ages. The use case scenario for the design of the "NAO says" game was a popular public event "Long Night of Sciences" which takes place every year at research institutions all over Germany on a specific day in June. On this day, scientific and science-related institutions open their doors and invite general public to visit and actively participate in experiments, demonstration, lectures, science shows, and guided tours. Playing "NAO Says" was offered as an interactive game event during "Long Night of Sciences 2022" at Berlin University of Applied Sciences, Germany, on 2nd July 2022. The "NAO says" game was embedded in a social context of this public event with multiple, voluntary participants engaging in playful interactions with the robot. The setting of the scenario was defined to be a university laboratory room, which was open during the event for general public to walk in and participate in the game at defined times. Based on previous experiences from the event "Long Night of Sciences" and the character of the "NAO says" game, the scenario defined families with children and young people as the primary audience and target group.

The "NAO says" game was designed from the human-centered design (HCD) perspective, following the scenario-based design (SBD) approach [14]. Scenarios are task based and descriptive, i.e. events and activities are strung together in purposeful sequences and provide a real-world description of the contents, flow, and dynamics [14]. The design was developed and tested iteratively. The design process included a joint co-creation of a scenario-based script in a project team, joint programming by two authors of this paper, playtesting in two pre-studies with university students, and finally the implementation and evaluation in the main study with 190 participants of all ages during the event "Long Night of Sciences 2022".

The gameplay in "NAO says" includes a series of playful mini-activities which encompass physical and cognitive tasks. During the gameplay, human players are asked by the NAO robot to follow only when they hear the command "NAO says". The rule from the classic version of the "Simon says" game, in which a player drops out of the game if he/she follows although there was no command, was not included in the interaction design. The reason not to include this rule was the focus on the playful, more stress-free and less competitive interaction rather than following the strict rules of the game and players having to quit the game.

The "NAO says" gameplay includes a total of ten playful mini-activities. Some of these mini-activities were based on existing animations for NAO, which are available in standard libraries of the Choregraphe software used to program the NAO robot. These included the "Saxophone", "Elephant", "Gorilla" and "Take a picture". These readymade building blocks for the game were selected as suitable for playful interactions, since they contain both clear body movements and corresponding sounds, which enhance playful engagement. Further existing animations, such as the "Air guitar", were combined with new sound effects, which were imported as free sound files from the Internet and integrated in Choregraphe. For the purpose of the "NAO says" game, some own animations were programmed in Choregraphe and added to the gameplay. The selfdeveloped animations included: "Stand on one leg", "Rub tummy, pad head", "Wave arms above the head", "Smile" and "Maths". In total, the following ten mini-activities were used in the "NAO says" gameplay: (1) Gorilla, (2) Elephant, (3) Air guitar, (4) Saxophone, (5) Take a picture, (6) Stand on one leg, (7) Rub tummy, pad head, (8) Wave arms above the head, (9) Smile, and (10) Maths. Figure 1 visualises all mini-activities arranged into categories: animals, music, body and other.

**Fig. 1.** Playful mini-activities included in the "NAO says" game.

## **2.2 Programming**

The programming of the "NAO says" game was done using the Choregraphe software (Version 2.8.6). The game was designed in the English and German language versions which were tested in two pre-studies with students. Figure 2 visualises the programming of the "NAO says" game in Choregraphe with different elements such as animations, animated say, speech recognition and tactile sensors (bumpers).

**Fig. 2.** Programming of "NAO says" in Choregraphe (English version).

When programming the game, four main challenges emerged: (1) How to program new mini-activities and which existing animations can be adapted for the purpose of "NAO says"?; (2) How to make the interaction with NAO possibly seamless with a larger number of humans players and observers present in the room and standing at a relatively large distance from the robot?; (3) How to make the gameplay exciting without repeating the same sequence of mini-activities? These challenges were addressed in the programming phase as described below.

**How to program new mini-activities and which existing animations can be adapted to the purpose of the game?** In order to create new mini-activities, such as "Stand on one leg" or "Rub tummy, pat head", the Timeline editor, an integrated tool in Choregraphe, was to store different positions of the robot in a timeline and play one after the other as a fluid sequence of animations. The Choregraphe Timeline editor enables the programmer to adjust every single angle of NAO's motors using simple sliders. Additionally, this method was combined with the use of the Animation mode. When NAO is in Animation mode, the angles of the joints on the real robot can be adjusted and the joint position saved as a point on the timeline. From there, NAO can be moved to the next position. This procedure results in a fluid animation from the individual positions at the end. When creating animations, special attention must be paid to the length between the individual positions of the animation. If the length is too short, jerky movements can occur. These not only look unnatural, but also cause the motors to heat up more quickly which can even damage the robot. In addition, special care must be taken to ensure that the NAO robot is always in a stable position. The ready-made animation "Gorilla" available in the Choreographe library demonstrates this problem: In "Gorilla", NAO drops forward onto its hands and sometimes falls over, either because the animation is executed too quickly or because the ground is not ideally flat. Therefore, creating new animations like "Stand on one leg" was particularly difficult to implement, as NAO had to be kept in a stable position during the entire animation.

**How to make the interaction with NAO possibly seamless with a larger number of humans players and observers present in the room and standing at a relatively large distance from the robot?** The scenario was designed for participation of multiple players and observers, all present in one room with NAO during the public event at university premises. The presence of many participants enhances the risk of a high volume of background noises which may impede speech recognition of the NAO robot. Therefore, the decision was made to limit the number of mini-activities with humanrobot interaction via speech. In fact, the only mini-activity, in which speech input from the participant is necessary, is the maths activity. In the maths activity NAO asks "How much is 3 multiplied by 3?" and expects the answer "nine" from the participants. Also during the game "NAO says" NAO also asks a number of times "Did you understand everything?" and waits until the answer "Yes" is said by a participant. However, these interactions via speech are only possible when there are no background noises in the room and possibly only one participant at a time speaks loudly and clearly. In the scenario at the public event with approx. 20 participants in the room at the same time, a loud and clear response was foreseen to not be feasible. Therefore, it was decided to keep the threshold very low so that everything that sounds similar to "nine" could be recognized by the NAO robot as nine. However, this method had the disadvantage as, for example, "nineteen" or other numbers are also recognised as "nine". Therefore, the final threshold was set to 30%, i.e. any utterances that sounds 30% like "nine" are recognized as a "nine".

**How to make the gameplay exciting without repeating the same sequence of miniactivities?** As described above, a pool of mini-activities was created to provide a variety of non-recurring playful mini-activities, and in this way to enhance the user experience. In order to make the gameplay exciting, the randomization principle was applied in the programming of the game. Randomness of game elements is linked to uncertainty, which is a frequently overlooked in game design, but an important element for the overall game experience as it holds players' interest and enhances engagement [15]. To incorporate the randomization for the path-finding command "NAO says" before movement, a random variable was included which was then queried with an If-condition. The challenge here was that the positive feedback (praise) given by the NAO robot at the end of the miniactivity was not in the same programming level as the dice rolling of the random number. To address this challenge, a separate random number was chosen for each mini-activity and duplicated this mini-activity. In the final version, different responses of the robot were programmed depending on the use or non-use of the command "NAO says".

The different versions of the "NAO says" game were tested in the two pre-studies and the final version in the main pilot study at the public event as described below.

### **3 Methods and Studies**

Following the design and the programming phase, the game was play-tested with university students in two pre-studies and the final version was implemented and evaluated during the public event with participants of different ages. Playtesting is a popular method in game research used to test perceptions and preferences of players, allowing designers to modify the game before delivering the final version [16]. The key facts about the pre-studies and the pilot study are summarized below and in Table 1.

**Pre-study 1.** The first pre-study involved a sample of ten university students, who volunteered to test the English version of the initial version of the "NAO says" game. Participants were asked to fill in an online survey before and after the game. One of the key results from the first pre-study was the wish of students to play the game in the German language version. Therefore the German language version was created in Choregraphe and tested in the second pre-study.

**Pre-study 2.** The second pre-study involved a sample of nine students, who volunteered to test the German version of the "NAO says" game and did not participate in the first pre-study. This version of the game also included a slower pace of NAO's speaking as the result from the first pilot study clearly indicated the need for slower speed to understand better what to do in each mini-activity. Like in the first pre-study, the participants were asked to fill in an online survey before and after the game.

**Pilot Study.** The main pilot study took place during the public event "Long Night of Sciences" with participants of different ages. Out of approx. 260–280 participants on that day, 190 persons filled in the evaluation survey which was administered before and after playing the game with NAO. The survey was paper-based to ensure high participation of persons without digital devices and of younger children.

**Table 1.** Summary of the pre-studies and the main pilot study of the game "NAO says".

## **4 Results**

The key results from the studies related to: (1) perceptions of the robot and the game "NAO says", and (2) perceived stress level before and after playing the game are described in the sections below.

### **4.1 Perceptions of the NAO Robot and the Game "NAO Says"**

The data about the perceptions of the participants of the NAO robot as game leader and of the game "NAO says" was collected via online surveys in the pre-studies and via a paper-and-pencil survey in the main study. Both online surveys included additional questions which were not asked during the main study due to the specific conditions of the public event. The online surveys ask the question How did you perceive NAO as a game leader? This question was answered by rating five pairs of semantic items from the Likeability Scale of the Godspeed questionnaires rated on a scale from 1 to 6 [17]. Table 2 summarises the results from the Likeability Scale.

**Table 2.** Perceptions pre-studies and the main pilot study of the game "NAO says".

The results show that in both pre-studies NAO was perceived as a likeable, friendly, kind, pleasant and nice game leader. The data also shows higher values in the pre-study 2 in which the German version of the game was used which may indicate that the use of the local language may have enhanced positive perceptions of the robot.

Next, perceptions of playful interactions with the robot were captured in all three studies using the simple question "How did you like the game?" and asking participants to assess their perception on a scale from 1 = not at all, to 6 = very much. The mean values were as follows: (1) pre-study 1, n = 10: M = 5.10 (SD .738), (2) pre-study 2, n = 9: M = 5.33 (SD .707), (3) pilot-study, n = 190: M = 5.25 (SD = .885). These results indicate that participants in all three studies enjoyed playful interactions with NAO. The foreign language version of the game in English in the first pre-study received the lowest value, which again indicates that the language choice affects user experience. The high average rating of M = 5.25 in the main study with 190 participants show that participants in different age groups liked the game.

#### **4.2 Perceived Level of Stress Before and After the Game**

Perceived psychological stress was measured to explore whether there were any changes in how stressed or relaxed participants felt before and after playful interaction with NAO. The data about perceived stress was collected via online surveys in the two pre-studies and via a paper-and-pencil survey in the main study. Psychological stress was reported by the participants before and after playful interactions with NAO using the Perkhofer Stress Scale, which is a validated single item scale [17]. The participants assessed their stress level on the scale from 1 = no stress ("fully relaxed") to 6 = fully stressed ("anxious") before and after the game. To explore the differences in perceived stress before and after playing "NAO says", the dependent samples (paired) t-test was computed at the 95% confidence level and two-tailed p-value using IBM SPSS software. The comparison of means showed that in all three studies the mean values for perceived stress before the game were slightly higher compared to the values after playing the game. In the first pre-study (n = 10) the mean values were M (before) = 2.60 (SD .843) and M (after) = 2.40 (SD 1.174). In the second pre-study (n=9) the mean values were M (before) = 2.56 (SD 1.014) and M (after) = 1.33 (SD 1.000). In the third pre-study the mean values were M (before) = 2.11 (SD .910) and M (after) = 1.74 (SD .917). Table 3 summarises the results for all three t-tests.

**Table 3.** Paired samples t-test: Perceived psychological stress before and after the "NAO says" game.

The results show that the means of perceived stress before and after the game differed significantly only in the main pilot study. It can be concluded that the perceived level of stressed was statistically lower after the game "NAO says" and changed from 2.11 ± 0.91 to 1.74 ± 0.92 (p *<* 0.001). The results also show that the initial stress level of the 190 participants in the main study during the public event was slightly lower compared to the two pre-studies with students, which may be explained by the leisure character of the event compared to participation in classroom settings. Nevertheless, the participation of volunteers visiting the laboratory during the public event limits the possibilities of generalizing the results of the study. It scan be assumed that the participants in the main study differed from the general population and from populations with special needs in regards to their level of initial motivation to participate as well as their interest in and attitudes towards robots. Therefore it is recommended to conduct a broader follow-up study involving a more diverse sample and including variables such as interest, motivation and attitudes towards robots.

## **5 Conclusions**

This paper reported on the design, programming, implementation and evaluation of playful interactions during the game "NAO says" with the humanoid robot NAO in two pre-studies with students and one pilot study with 190 participants of different ages. The exploratory results in all three studies showed that the players perceived NAO as a likeable, friendly, kind, pleasant and nice game leader, and enjoyed playful interactions with NAO. Additionally, there was a significant difference in the perception of own's psychological stress before and after the game with NAO in the pilot study with 190 participants. The results also indicate possible effects of different language versions of the game on user experience. The results presented in this paper are to be understood as preliminary, exploratory results and as a starting point for further research. Further studies should be conduced with diverse samples and look closer into possible effects of different versions of the game. The paper also pointed out several challenges in the design and programming of the game "NAO says" and how these were addressed. Further studies could explore in more detail which design strategies of playful interactions in games like "NAO says" and which types of feedback from the robot are most effective for specific target audiences. Furthermore, future studies could explore how different types of playful interactions with robots may affect the perception of mood and stress as well as physical stress measures.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Representation of Air Pollution in Augmented Reality: Tools for Population–Wide Behavioral Change

Grzegorz Pochwatko1(B) , Zbigniew Jędrzejewski1 , Wiesław Kopeć2 , Kinga Skorupska2 , Rafał Masłyk2 , Anna Jaskulska3 , and Justyna Świdrak4

1 Institute of Psychology, Polish Academy of Sciences, Warszawa, Poland

<EMAIL_ADDRESS>2 Polish-Japanese Academy of Information Technology, Warsaw, Poland 3 Kobo Association, Oświ¸ecim, Poland

4 August Pi & Sunyer Biomedical Research Institute, Barcelona, Spain

Abstract. Air quality affects health and, unfortunately, has been deteriorating rapidly recently. The problem is significant in smaller towns where the important source of pollution is the heating of households and water from individual sources. Therefore, the inhabitants have an influence on a significant reduction of pollution in their area, but at the same time, they are very often not aware of it. Raising awareness about household-related air pollution plays a vital role as systemic solutions proposed by state and local authorities require the support of local communities. The situation has recently become even more serious as we are facing a crisis caused by the Russian war in Ukraine, which has led to an increase in energy and fuel prices and has postponed restrictions on the use of solid fuels or even incentives to use inferior fuels.  One of the ways of raising citizens' awareness was to be public, easily accessible information about air quality. Many portals, services, and applications currently provide local air quality data, but few people use them. One reason may be that the figures and graphs can be confusing or unattractive to audiences who are not used to reading scientific reports. Visualizing air quality with augmented reality overcomes these obstacles. A mobile application that can use local elements as triggers and a symbolic representation of air quality based on data read in real-time from sensors is simple, attractive for non-experts, and has an additional educational value. We present the experience of creating such an application and prototype tests with the participation of potential users. Unfortunately, the collected results confirm the low awareness of excessive pollution in a given area and its negative impact on health. However, the interest of potential users and positive opinions about the tested prototype fill with optimism.

Keywords: augmented reality *·* air pollution *·* user experience *·* art and science

### 1 Introduction

Visualizing air pollution in the real world is not a new phenomenon, but it has largely been contained in the domain of arts, with projects aiming to make the local population aware of the pollution's negative effects on their health. A traveling art project, giant lungs made of air filters whose color changed with time when exposed to polluted air, in multiple editions toured the most polluted cities in Poland1 Such projects have great value, yet they have limitations as they are confined to a specific location and do not show dynamic changes in pollution rates. Still, such campaigns are a crucial step in limiting pollution in the most polluted areas, especially if pollution comes from residential sites. A decrease in emissions can be seen following informative action [4], and such initiatives often pave the way for necessary policy. Still, it is as they garner public support for such initiatives. For these reasons, we have decided to design and verify an Augmented Reality (AR) application representing current pollution levels based on location data. Seeing real-time pollution data on a personal mobile device and having it visualized in a manner that is meaningful to the device-holder at a place where they are currently has the potential to increase awareness of air pollution presence and its negative health effects.

Fig. 1. PM2.5 values measured every hour during one day in winter and summer 2021; a town with up to 25,000 inhabitants, data from the Environmental Protection Inspection [5]

#### 1.1 Household Related Air Pollution

Air quality affects the health of citizens and, unfortunately, recently has been deteriorating rapidly. However, when effects are delayed, people tend to disregard their severity, even though multiple papers closely link negative health outcomes, especially respiratory diseases, and reduced lung capacity, to air pollution

1 The images from the project, by campaign creators, are available here: https://www. purpose.com/poland-air-pollution-campaign/.

[11]. The problem is particularly important in smaller towns where the important source of pollution is the heating of households and water from individual sources. One of the most dangerous pollutants is particulate matter, especially its fine fractions - PM2.5 [1,6,7]. Figure 1 shows how much household heating can contribute to a dangerous increase in PM2.5 levels. The chart shows the pollution levels recorded in hourly intervals in summer and winter in one of the Polish small towns (up to 25,000 inhabitants), which is dominated by singlefamily housing with its own heating sources. The PM2.5 level in the summer is low. In winter, however, it exceeds the norm many times in the afternoon and in the evening, when citizens usually return to their homes from work and turn on the heating.

### 1.2 Air Pollution Representation

The choice of the pollution representation is crucial considering the fact that the current methods of presenting information on the air quality turned out to be ineffective or gave inconclusive results (see e.g., [10]). For example, Campaña and Domínguez [3] proposed to present the different types of pollutants as enlarged models of chemical molecules. In another case, Prophet et al. [13] encouraged participants to take care of air quality in a symbolic way - by taking care of a virtual tree in an application using the data of the local pollution measurement station.

In our opinion, the choice of the correct representation of air pollution should not be arbitrary. We do the choice in a multi-stage process, which we started with a participatory workshop with potential users [8] and continued with laboratory research (see e.g., [12]). Initially, four types of visual stimuli were selected and tested: 2 positive vs. negative and 2 concrete vs. abstract. At the time when we were preparing and testing the prototype presented in this paper, the laboratory studies were still in progress, so for the purposes of the AR app tests, we chose a concrete and negative representation - enlarged PM particles.

#### 1.3 Augmented Reality

Augmented reality (AR) is a technology that is gaining in popularity and has a chance to stay ahead of immersive virtual reality due to the availability of interesting solutions for mobile devices[9]. 3D and animated objects can be easily placed in real space thanks to toolkits such as Vuforia2 which make developing AR applications easier for various context, varying for workplace instructions, entertainment to product design.

Vuforia is an AR add-on to one of the commercially available development engines for creating games, called Unity3d. The principle is that with the help of an application designed for a mobile device, an additional layer of abstraction

2 Vuforia is a comprehensive software development kit (SDK) for Virtual Reality applications available here: https://developer.vuforia.com/.

can be added to the area seen by the camera. In this way, the visible area can be expanded with graphic elements, usually in the form of useful information.

For this to happen, it is necessary for the application designer to design and prepare (in appropriate formats) two types of graphic elements. The first is the so-called marker, otherwise known as a trigger - that is, an object to whose presence in the field of view of the camera the application will be "sensitive", and the second is a target object (static image, animation, 3d model and many others) which appears in the field of view at or near the predefined marker (Fig. 2). Today, AR technology is increasingly used in industry, social media, and everyday products, among others. This is due to several basic needs that the market places on this technology, i.e.: product recognition and training, product visualisation, customer self-service, guided user manuals, and part recognition [14]. The increasing use of AR apps has been noted by several online industry journals. Insider Intelligence [15] notes the increasing share of users using augmented reality functionality, and Forbes [2] notes that this technology could be the future of social media and beyond.

Fig. 2. First draft of the VAPE AR app: scanning QR code to obtain the app (left), scanning the trigger to launch Vuforia AR contents (center), exploring AR contents pollution representation overlay (right).

## 2 VAPE Augmented Reality App Design

#### 2.1 Purpose of the AR Application

The aim of the design of the application was to create an engaging experience presenting air pollution with visual cues and to communicate scientific data in a comprehensible way, by doing so, potentially raising awareness of environmental and health-related issues caused by air pollution. The application was designated to be pretested during a scientific event held in Myszków (Poland)3, with the intention of further development in the future. The constraints affected by the out-doors-held event called for an easily accessible, compact and portable setup in addition to minimum device requirements.

3 Myszków is a town with one of the highest air pollution levels in Poland and Europe. It was on the list of 50 cities with the most carcinogenic air in Poland.

## 2.2 Accompanying Poster

The intention of creating an informative poster complementing the experience with scientific data has affected the decision to employ a trigger-activated AR system as a base design of the application. A 100 cm per 200 cm sized roll-up banner commonly used in advertising at events was chosen as the print medium for the informative poster. The medium allowed easy and self-contained construction, a great level of portability, and re-usability, additionally to the vast canvas surface. The poster was designed and prepared for print in Adobe Photoshop (CS6), composing hand-drawn digital illustrations and text created beforehand in Procreate. The overall design of the poster was intended to be simple with key scientific information, not to overwhelm the participants of the event and to encourage engagement with the application (Fig. 3).

Fig. 3. Testing the prototype - the trigger (left) and a user scanning the trigger (right)

#### 2.3 Implementation of Air Pollution Visual Representation

The VFX representing air pollution was created in Unity's Visual Effect Graph. The node-based Visual Effect Graph allows the creation of procedural animations of a vast number of real-time rendered particles. The particles were set to spawn evenly within a cube shaped-region of base measuring 6 m by 7 m and reaching 5 m high above the ground. The particles were spawned constantly with their basic life-span set to 10 s (not counting the added variation), and the overall capacity of particles of the VFX was set to 100000. The animation of the particles was driven by a turbulence node effecting in an appearance of the particles floating freely in the air. Each of the particles was assigned with a random texture from a set of 16 images using a flipbook particle output method. The images were edited microscopic photographs of actual air pollution particles. The assigned textures were given random sizes ranging from 0.5 cm to 2 cm. A gradient of colours from dark grey to black was assigned to the particles.

In order to raise awareness of health-related issues caused by air pollution, a visualisation of lungs has been chosen additionally to the visualisation of air pollution particles. The 3D model of lungs implemented in the application has been generated by segmentation of trachea and bronchi based on a computed tomography medical scan using 3D Slicer (4.11.20210226) software. The generated 3D model was edited and retopologiesed using Pixologic Z-Brush (4R7). The model was exported to Unity using an fbx file. The model was enhanced with a VFX animation of the same air pollution particles used in the air floating VFX. The animation shows the particles streaming through the trachea and obscuring the lungs (Fig. 4). The 3D model was visualised and augmented in front of the poster.

## 2.4 Application Development and Implementation

The application was compiled in Unity (2021.2.0b12) Game Engine with a Vuforia (10.0.12) plug-in installed. The Vuforia plug-in allows easy setup of AR trigger-activated system. The visualisation augmented upon the surroundings of the user was set to be triggered by targeting the smartphone's camera on the illustration of lungs presented in the centre of the informative poster. The scene build in Unity contained a Vuforia image trigger setting, animated Visual Effects (VFX) representing air pollution particles, and a 3D model of lungs with its own VFX animation. The application was built for android devices and preinstalled on a smartphone that was later used as a visualising device during the event. The presented application was a prototype and was not available for download from the internet by the participants, yet such an option of dissemination is possible in the future.

Fig. 4. The animation of the particles entering via the trachea and obscuring the lungs.

## 3 Pretest Results

The prototype of the AR application was presented to volunteers, participants of an open event promoting pro-ecological behavior in the city of Myszków. The event was co-organized by local authorities and non-governmental organizations. Due to the restrictions caused by the COVID-19 pandemic, all events were held outdoors with appropriate precautions. The presentation of the prototype was one of the activities that the participants of the event (mostly parents with children) could get involved in. In order to better simulate the conditions in which the application will be used in the future, it was not announced in the information inviting to the event (posters hung around the city and spots on local TV). Participants could take part in a pretest, which was organized as one of many stands (e.g., art classes with local artists, Storm Hunters information stand, non-contact sports activities, 3D printing show). Participants who decided to test the AR application were encouraged to provide feedback to the researchers operating the stand. Many participants of the event spontaneously reacted to the information posted on the poster/trigger. As intended, they tried to scan the trigger on the poster with their own phones. In such a situation, the experimenters approached potential users, explained that it was a pre-test of the prototype and, for security reasons, it was not possible to use their own devices yet, but they could see how it works on the device provided for this purpose. They then showed participants how to scan a trigger on a poster, handed over a phone with the app installed, and began observing their interactions. If test subjects watched only the poster and lungs in the AR for a long time, the experimenters encouraged them to turn around and observe the area with the layer of pollution overlaid. Most of the participants spontaneously made such an exploration. Careful observation of the lungs with pollutants entering them was rather avoided. Most users focused on watching pollution particles in the air. Below are representative testimonials from participants in the pretest:

«*All this dirt makes a terrible impression. I would definitely like to have such an application to check whether, for example, I can go on a trip with my children.*»

«*I didn't know that so much dust was flying in the air, especially since it's a nice day today, you can't smell the coal smoke*»

«*A man wants to do something for his health, he runs, exercises, and here you have something. If you check «on the portal», you can't see it.*»

«*Maybe it would make people realize that you can't burn just anything*» «*Someone would have to want to use it. Some people don't care.*»

Thanks to positive and negative comments from users, we have the opportunity to develop the application. There is a need to conduct broader research, which would also indicate ways to popularize the application and ensure its usability, information value and impact on changing user behavior.

## 4 Conclusions

Visualising key data with mobile AR is a reliable way of grounding the understanding of information within the context of the environment it is displayed over. In this way, people living in a certain area may better realize the invisible presence of air pollution and how it may affect their health - but, as our subjects mentioned, only if they care to stop and check. Still, we hope that our application prototype will serve as a proof-of-concept for larger information campaigns on air pollution so that they are created in a more impactful and understandable way, relevant to their target users.

Acknowledgments. We would like to thank our participants, volunteers and citizens of Myszków, Poland, who participated in our actions; The research leading to these results has received funding from the EEA Financial Mechanism 2014-2021 grant no. 2019/35/HS6/03166.

## References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Ukrainian Version of the Copresence Scale**

Lyubov Naydonova1 , Grzegorz Pochwatko2(B) , Mykhaylo Naydonov3 , and Justyna Swidrak ´ 4

1 Institute for Social and Political Psychology, National Academy of Educational Sciences, Kyiv, Ukraine 2 Institute of Psychology, Polish Academy of Sciences, Warsaw, Poland

<EMAIL_ADDRESS>3 Institute of Reflexive Investigation and Specialization, Kyiv, Ukraine

<EMAIL_ADDRESS>4 August Pi & Sunyer Biomedical Research Institute, Barcelona, Spain

**Abstract.** Millions of Ukrainians are fleeing the war to the EU countries; in particular, Poland has accepted more than three million refugees, among whom are mostly women and children. Some of them continue to work remotely in Ukraine and communicate with friends and relatives at a distance through technologymediated ways. The aim of our study was to observe the role of perceived copresence during social contacts, both of professional and private character, and validate the Ukrainian version of the copresence scale among Ukrainian migrants. We collected 221 responses in an online and face-to-face study from Ukraine migrants. Analyses revealed that the Ukrainian Perceived Copresence scale has one factor and appropriate internal consistency. Perspectives of future research are proposed.

**Keywords:** Copresence *·* Ukrainian *·* Migrant *·* Mediated communication

## **1 Introduction**

## **2 Theoretical Context**

## **2.1 War Migration from Ukraine in Poland**

Migration makes it necessary to adapt to a new social environment and at the same time to maintain distance communication with those who remained in the previous place of residence, for this migrants use various technologies to mediate communication. The wave of mass migration from Ukraine as a result of the war and the military aggression of the Russian Federation became a necessary means of preserving life. As of mid-September, according to the UNHCR [10], more than 7 million temporarily displaced persons from Ukraine are in European countries, 1.3 million of them are in Poland. Since the beginning of the full-scale offensive, more than 6 million people have crossed the border from Ukraine to Poland. Distant communication for work and private conversation became an important part of the Ukrainian migrant community in Poland. As communication is needed for mental health, research on the copresence effect in communication is actual, but for these purposes, Ukrainian language methods must be developed and adapted.

**Fig. 1.** Mediated communication (photo by Julia M. Cameron; pixels.com)

### **2.2 Copresence**

Copresence has become a popular research topic in recent years, especially after the COVID-19 outbreak which forced millions of people to stay in isolation for many days or weeks. The term "copresence" was suggested to mark the quality of a communication medium in human-human or human-machine interaction [7] and is defined as the degree of person-to-person awareness which occurs in the computer environment [9]. Our previous study [8] has proved a relationship between the amount of interpersonal communication, housing conditions (a shared or private room, number of children and adults in the household), and copresence with mental well-being in confinement during the COVID-19 lockdowns. Here, we present the validation of the Copresence scale [8] to Ukrainian language, which was used in a study that aimed to replicate these results in another special context of forced disrupted social relations, this time due to migration forced by war. This "war migration situation" has put millions of migrants in new housing conditions, at a distance from their work and relatives, and in emotional positive and negative contacts. Here we describe a study carried out from May to September 2022 in Poland, in which we aimed to validate the Ukrainian version of the copresence measurement instrument.

#### **2.3 Mediated Communication and War Migration**

Chen's study of the role of internet communication in migrant adaptation shows that immigrants who remotely communicate online more frequently with relatives and friends in their original country are less adaptive in terms of sociocultural adaptation. However, communicating with relatives and friends in the original country is no longer a significant predictor of intercultural adaptation when introducing demographic characteristics into the regression analyses: no significant impact is identified for online communication with fellow immigrants on intercultural adaptation [3]. Copresence may be important to overcome the isolation of people using VR technology as it was approved for anxiety disorder [6]. Migrants experience "co-presence" with their loved ones through social media. Among second-generation Turkish-Dutch migrants who grew up in the Netherlands and migrated to Istanbul in adulthood [1] are in two different models of communication: ambient, fast-paced, background communications and also more direct, immersive, conversational modes. The careful shifting between these modes of social media communication produces migrants' experiences of transnational emotional intimacy when emotion is mediated through digital media and gives psychological support to migrants. During the war and migration of Ukrainian family members, distant communication is the factor that influences the moral and psychological state of the combatant when family problems impact the psychological state of the service members during deployment in active military zones of anti-terrorist operations [4,5].

## **3 Method**

#### **3.1 Sample and Data Collection**

The study was carried out online in the form of a self-report questionnaire. Data were collected from May to September 2022 among Ukrainian adult migrants. The study obtained approval from the ethics committee of the Institute of Psychology at the Polish Academy of Sciences. Participation in the study was completely anonymous and voluntary. 221 Ukrainian migrants participated in the study (195 participants had valid data in both private and work-related conditions; 196 and 219 respectively), and the majority of them were women (72%). 78% of migrant participants entered Poland after the 24th of February 2022, after the Russian military invasion of Ukraine. One person did not want to reveal their gender, and three persons were nonbinary. 56 people were interviewed face-to-face and others via the internet through various platforms and social media groups to reach as heterogeneous groups of potential participants as possible. The average age was 38 (SD = 13, min = 20, max = 72) for Ukrainian women and 31 (SD = 13, min = 18, max = 71) for UA men. Ukrainian participants lived before migration in a city (48%), 34% in a town, and 16% in a village; after migration 85% in a city, 12% in a town, and 3% in a village. Since the study was carried out online mainly among volunteers, it was impossible to avoid the selection bias completely.

#### **3.2 Measures**

**Copresence.** We used the Polish adaptation of the Social Presence Survey from Bailenson, Blascovich, Beall, and Loomis [2] in a variant by Swidrak, Pochwatko, Matejuk [8] for co-presence measurement. Participants filled the scale twice, separately for workrelated (variable copresence-work) and private (copresence-priv) communication. Participants responded on a Likert scale from 1 (I strongly disagree) to 5 (I strongly agree). The score of each variable was calculated by extracting the mean of all items. The Ukrainian version of the Perceived Copresence Scale (PCS-U) was developed by 3 steps translation of instruments procedures. First, the original Polish PCS was translated into Ukrainian by two professionals (a psychologist, and a software developer) from outside the research team, both of whom were bilingual and fluent in Ukrainian and Polish. Next, the two translations were compared, and discrepancies were reconciled until one working draft in Ukrainian was achieved. Second, a bilingual expert panel consisting of two original translators and two science researchers reviewed the draft Ukrainian translation to make cultural adaptations as necessary. Third, the corrected Ukrainian scale was back-translated into Polish by one bilingual translator. The back-translated Polish version was then compared to the original Polish version and reviewed by the original author to ensure that the questions were translated correctly and a cultural and language equivalency was reached.

**Housing Conditions.** We controlled whether the participant lived alone or with others by asking about the number of adults, children, and small children living in the same household. In analyses we used two variables: the number of adults living with the participant (variable cohab-adult) and the total number of children (variable cohabkids). We also inquired whether participants had their own room or had to share it (variable ownroom, 0 - no, 1 - yes). Communication conditions were: the number of hours spent at work on mediated communication (h-work), the quantity of video-calls (video-work), and the speed of the internet connection (internet-speed).

**Hours of Private Calls and Hours of Work-Related Calls.** To measure the quantity of mediated communication, we asked participants to estimate the number of hours they spend daily on private and work-related communication using: a virtual reality headmounted display, a computer, and a tablet/smartphone. In the next step, we summed the hours separately for work-related and private calls (variables h-work and h-priv).

**Internet and Electronic Devices.** The Internet connection speed was measured with a single-choice question: My internet connection is (1) slow, (2) average, (3) fast, (4) very fast (variable: internet speed). We also asked about the percentage of calls in both contexts being video calls (variables: video-priv and video-work).

## **3.3 Data Analysis**

Data analysis was carried out in SPSS version 28. We have used descriptive statistics to analyse the demographic data and housing conditions, a principal component analysis (PCA) to test the factorial structure of the Ukrainian Perceived Copresence Scale; and calculated the Cronbach's alpha to measure its reliability. We took an explorative approach, following Gellman and Hill [2]. To test the external validity of the scale, we calculated the correlation between the number of hours spent on work-related and private video calls.

**Table 1.** Means and standard deviations of co-presence.

## **4 Results**

#### **4.1 Descriptive Statistics**

**Housing Conditions and Internet Usage.** Housing conditions of Ukrainian migrants differed: 30% of them were working in an office and going to work each day, only 8% worked remotely and 10% worked in a hybrid form (partly remote and partly personally); 22% were students of various levels and forms of learning, 34% were temporarily unemployed, home caregivers or pensioners. Only one in ten respondents lived alone and 28% had own room; 24% lived with a partner, 38% lived with two or three people. Regarding the cohabitants, two thirds of participants lived with children, 21% with small children, 41% with one child. Majority of migrant participants (74%) had mobile/phone internet connection, only 4.5% used a cable internet connection, and 2.3% had the optical fibre connection. Only 1.5% of respondents declared no internet at home. Overall, the vast majority have a fast or rather fast connection (8% very fast, 44% fast, 38% average/medium), with only 9,5% declaring slow connection. Reliability of internet connection was rated as relatively low with half of the sample responding that it was sometimes interrupted, and was slowing down; 38% that it was often interrupted.

#### **4.2 Validation of the Ukrainian Perceived Copresence Scale (PCS-U)**

The PCA was used to verify the factor structure of the Ukrainian scale. Bartlett's test of sphericity was significant (χ2 = 1219 p <.001) and the Kaiser-Meyer-Olkin test score was .84. The PCA revealed one factor for the Ukrainian version, explaining 67% of the variance. Reliability analysis revealed very high level of internal consistency of PCS-U (Cronbach's alpha =.928, valid cases N=195). The correlation between copresence and the number of hours spent on work-related and private video calls was .174 (p<.05) and .289 (p<.001) respectively (Table 2).

**Fig. 2.** Perceived Copresence Factor Analysis: scree plot.

Extraction Method: Principal Component Analysis.

## **5 Discussion**

The first result of the study is the validation of the Ukrainian Perceived Copresence Scale (PCS-U) as a one-factor measurement instrument with appropriate internal consistency. This gives reason to think about the life situation of migration as one that changes the conditions of work and personal interactions and reduces the feeling of copresence in remote communication, at least during the adaptation period. It means that copresence plays different functions during work and private communications in conditions of war migration; its pattern analysis is the task of the next study. We need to research copresence in private and work communication during the collective trauma healing process. Types of trauma events and the character of conversations about events and emotional stress may be independent variances for future research. Mass migration during the war changed the importance of having a proper physical workspace isolated from other cohabitants to participate in work-related calls efficiently. Copresence in the context of post-war society transformation, peacebuilding, and construction of historical remembering by war migrants, is an important subject for future research for a deeper understanding of copresence. There are several limitations to the study, such as not being representative of the Ukrainian migrant community sample, including gender representation. Future studies of migrants' technology-mediated communication and well-being should focus on research in a more controlled study design.

**Acknowledgments.** The research described in this publication was funded through an internship awarded to Professor Lyubov Naydonova by the Director of the Institute of Psychology at the Polish Academy of Sciences.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Modular Platform for Teaching Robotics**

Damian Nagajek1(B) , Michal R - apala1 , Kamil Woloszyn2 , Krzysztof Turchan2 , and Krzysztof Piotrowski2

1 Space Research Centre of the Polish Academy of Sciences, Zielona G´ora, Poland

<EMAIL_ADDRESS>2 IHP - Leibniz-Institut f¨ur Innovative Mikroelektronik, Frankfurt (Oder), Germany<EMAIL_ADDRESS>https://cbkpan.pl, https://www.ihp-microelectronics.com

**Abstract.** The Fourth Industrial Revolution causes changes in the economy and shifts the job market demand towards workforce with high technical skills. To keep an undisturbed economic growth we have to encourage more young people to develop competences in STEAM (science, technology, engineering, arts and mathematics). This has met with a response from some education systems which have prepared special programmes focusing on developing technical skills among students. One of the desired field is robotics, which involves constructing and programming. We have already conducted some workshops for high school students in this subjects and we would like to find the correct teaching tools to attract primary school students. Our idea is to create a modular platform, which elements could be used as black boxes, to teach robotics to young children. We have noticed that Arduino based kits are a bit too complicated and we decided to test a LEGO Technic set equipped with an external microcontroller. We have verified the interest level of children and the difficulty and time needed for a teacher to master the whole teaching platform. According to our study, LEGO attracts students much more than Arduino and is easier in operation and less time consuming during classes for teachers.

**Keywords:** Autonomous robots *·* Recognition algorithms *·* LEGO *·* Robotics *·* Neural networks *·* OpenVINO *·* Microcontrollers *·* Python

## **1 Motivation**

The 21st century brought us the Fourth Industrial Revolution (4IR). This industrial development is fueled by rapid changes in the digital technologies and industry, especially in the areas of artificial intelligence and advanced robotics. 4IR causes incredible modification of the global production and supply network by introducing smart technologies, wider communication between machines, internet of things (IoT), autonomous process monitoring, self-diagnostic etc.

The transformation from traditional manufacturing to full automatisation of industrial processes totally shifts the role of human in the whole system; stronger dependence on robotics moves us from the main workforce in a factory to auxiliary function of maintenance and emergency service. Undoubtedly, this situation has got many advantages like: less strains put on workers body, faster and cheaper production, better repeatability of a production process etc. Inevitably, the fast pace of drastic changes during the 4IR brings many challenges and dangers; especially socio-economic risks are raised in [10,11] by some authors. One of the problems that arises is lack of qualified workforce that can operate at the newly created intelligent workplaces. According to Spoetti and Windelband [14] vocational education and training of the workforce will be highly relevant to successful implementation of the 4IR. This burdens the sector of education with an expectation to follow the needs of the industrial development and employment, what has a strategic importance for a stable and undisturbed economic growth.

The mentioned challenges affecting schools have been already spotted in some countries. In Poland, the Ministry of Education and Science has commissioned a study which defines an actual demand for particular professions and which of them the economy lacks the most. The published documents from 2021 [12] and 2022 [13] confirm the statements from the study by Spoetti and Windelband [14]. Three professions occurred on the list each year: mechatronics engineer, automation technician and robotics technician. In response, the Ministry of Education and Science in cooperation with the GovTech Center of the Poland's Prime Minister's office have created an education programme destined for primary schools called "The Laboratory of the Future". The idea is to financially support entities that develop students' competences in STEAM (science, technology, engineering, arts and mathematics). New workshops will be created, which should encourage pupils to learn these subjects. A huge focus has been put on robotics, more precisely on a controller programming.

### **2 Observations**

Our laboratory together with a student research group called "KNIK" from the University of Zielona G´ora and our partner in the EU's SpaceRegion project - IHP institute from Frankfurt Oder have organized in 2022 some controller programming workshops for high and vocational schools' students.

We have used Arduino starter kits composed of an Arduino Uno microcontroller board, a breadboard, jumper wires, a power unit, transistors, sensors, LEDs, motors, resistors, buttons, an LCD display etc. All in all, it is a quite versatile but simple to use set to introduce pupils to electronics and programming. During some classes also custom made microcontroller boards designed by IHP have been used.

In the previously mentioned "Laboratory of the Future" programme, schools are free to select appliances' models, since only a mandatory equipment type is specified. A microcontroller board is one of them. We have assumed that many will choose Arduino based sets due to relatively low price, good availability, comprehensive documentation and huge community support.

Not a lot of students have attended the organised workshops. From the destined group, only about 10% in the high school and about 20% in the vocational school. The Arduino sets have seemed to be a bit too laborious and complicated to operate, since for many of them it has been a first contact with a microcontroller programming. Also handling of the accessories with universal connectors is a bit different than what students are used to - connecting equipment with dedicated plugs. Additionally, they have been a bit older and more skilled than primary school students to whom the new ministerial programme is devoted.

Another problem that can occur in many schools is lack of specialized teachers. A computer science tutor is skilled in programming but can have inadequate knowledge in electronics. Additionally, the time needed to put together a working unit based on the Arduino is relatively long, and taking into account very complex primary school syllabus, effective utilization of lesson's time is crucial.

### **3 Idea**

In cooperation with our partner IHP we have looked for a solution which would appeal more to younger people. One of the most widespread toy types are bricks especially LEGO sets. Almost all children are familiar with them and they are generally appreciated.

Another advantage of a LEGO set with a microcontroller is simplicity of construction process. All parts can be easily and tightly connected, and the amount of possible configurations is vast. It has to be underlined that for primary school students, the playing aspect is very important, and good entertainment can help to stay focused and encourage them to learn. From our observations, quite raw Arduino sets, very utilitarian in their nature, are not really rewarding for students. On the contrary, LEGO gives a possibility to make your own construction move by adding motors and sensors in the right spots. It gives more freedom in model creation and saves lesson time due to relatively simple way of connecting parts. Students can be very creative and engaged in building new constructions and simultaneously, without a lot of effort, learn how to design control systems and program them.

For our tests, we have chosen a LEGO set featuring electric motors and a wheeled platform, see Sect. 4. Additionally, some external electronic parts and a control software written in Python have been implemented. We believe that these elements are relatively easy to handle, and the learnt skills would be useful at the job market. We had decided to give the construction and programming tasks to a PhD student inexperienced in such subjects to mimic a teacher's situation and to verify how challenging and time consuming is gaining a new knowledge. Young students reaction and interest level have been checked during a public science exhibition at IHP institute in Frankfurt.

### **4 Construction**

A robot that has been created by the PhD student has been a wheeled platform with a camera. He has used some out of the box parts to simplify the development process and to make it more reproducible. The construction components are shown in Fig. 1. The entire based structure has been assembled with elements from the LEGO Technic 42114 Volvo 6 *×* 6 set (Fig. 1a [4]). Motion system has consisted of four Lego Large Motors 88013 (Fig. 1f [3]) installed on the platform. They have been connected to a Raspberry Pi Build HAT (Fig. 1h [7]) which is an extension (shield) for the main computer - Raspberry Pi B4 (Fig. 1d [6]). An Intel Neural Compute Stick 2 (Fig. 1c [1]) and a Raspberry Pi Camera HD v2 (Fig. 1g [8]) have been linked to the main computer. The power has been supplied by a LiFePO4 battery (Fig. 1e [2]) connected to a step-down Voltage Inverter LM2596 (Fig. 1b [9]).

**Fig. 1.** Used components.

Only the compatible components have been selected, thus assembling of the entire platform has been relatively straightforward. A first three-wheeled prototype had been constructed within one week. It had been rear-wheel drive, and its drivetrain had included a front-mounted self-aligning wheel. Unfortunately, such a solution had encountered some mobility problems, so it has been decided to transform the entire structure to a four-wheeled vehicle. Still, a simple attachment of the wheels via LEGO cross axle has caused them to fall out of their mounts during sudden turns. Moreover, a rather low torque and fast movement of the motors have made a precise control of the whole structure relatively cumbersome. It has oscillated or on the grippy surface has not moved at all. To solve these problems, we have applied a gear multiplier equipped with a dedicated hub for each wheel. In addition, we have had to upgrade the entire structure to make it more stable and to withstand the forces exerted by relatively heavy battery and electronic parts. A final construction is shown in Fig. 2.

**Fig. 2.** Final construction.

The Raspberry Pi 4B has been the main computer board for this setup. It has been extended with the Raspberry Pi Build Hat which is a dedicated add-on board (shield) for the Raspberry Pi computer. Generally, this shield is compatible with various LEGO motors and can control up to four different units or sensors of distance, color, pressure etc. Moreover, it allows users to precisely control motors and read data from encoders i.e. an absolute motor position, its current speed or rotation direction. Additionally, a motors control via PWM signal is possible. A minimal possible rotation angle of used LEGO Large Motors 88013 has been 6◦.

The Intel Neural Compute Stick 2 (Intel NCS2) is a plug-and-play USB hardware deep neural-network inference accelerator for computer vision and deep learning. The acceleration is working by assisting the main computer's processing unit (CPU) by taking over mathematical calculations required for running deep learning models. Moreover, the Intel NCS2 is supported with an OpenVINO (Open Visual Inference and Neural Network) library, which includes multiple pre-trained models e.g. face or text detection, formula recognition etc. A wide variety of them allows users to choose the most suitable for desired tasks. The biggest advantage of OpenVINO is its application simplicity. This library can be used as a ready-made module. A person without specialized knowledge is able to apply it without modifications and understanding of its working principles. Inexperienced students can treat it as a black box and quickly create a fully functioning device.

For people and facial recognition task we have used the compatible Raspberry Pi v2 camera with an 8 MPx resolution. This device works with drivers included in Raspbian - Raspberry Pi's operating system, moreover it has hardware support what totally limits a consumption of CPU's computing power.

The whole system has been powered by LiFePO4 battery connected to the step-down Voltage Inverter LM2596. A connection diagram is shown in Fig. 3.

**Fig. 3.** Connection diagram.

## **5 Algorithm's Working Principles**

The main task of the used algorithm has been recognizing a human posture and robot's movement control. It has had to follow and hold a desired target (a human) in the middle of the observed area.

We used an OpenVINO pretrained model from the Object Detection Models library called "person-detecion-retail-0013" [5]. The control algorithm has been designed to center the detection frame in the middle of the observed area and then try to follow the target and hold it in that position (Fig. 4). The size of the detected target has had to be kept within defined limits what has forced the robot to zoom in or out by moving forward or backward.

Software code changes implemented by users can be kept to a minimum. It is only necessary to declare the camera resolution, the desired data detection frame size and the motors' speed. The whole algorithm works in real time. Users have to bare in mind that a wide accepted model size limit gives better error tolerance but can cause some control problems like lack of a trigger to move. On the other side, a too small limit can cause permanent movement due to defined, finite minimal possible motor's rotation angle. The software gets caught in an infinite loop while trying to center the detection frame because the minimal

**Fig. 4.** Visualization of the algorithm.

movement value is bigger than the accepted margin. A similar situation will occur if the engines' speed is too high; the control unit is not able to proceed collected data fast enough and to send control commands on time. Also a too high video resolution can cause delays between target recognition and motors movement due to limited CPU's calculation power.

Generally, the camera constantly sends data to the computation unit. When a target occurs in the detection area and is recognized, the main computer transmits a control signal to motors via the Raspberry Pi Build HAT. The motors are running until the target's frame is centered in the middle of the detection area. The platform rotation is caused by movement of the wheels placed on different sides in the opposite direction. Afterwards motors shut down and wait until target leaves the center of the detection area. Moreover, the robot tries to keep the correct target size by moving forward and backward by rotating all wheels in the same direction. The robot follows a detected person to keep him within a certain distance. This has been very encouraging for children, who have thought that the robot has been scared by them and has run away to keep a save distance every time they have come closer.

All pretrained models are implemented in .xml and .bin files and can be used like black-boxes in the modular fashion. Their content do not need to be changed by users. E.g. changing our software to recognize particular objects instead of people comes down to replacement of the complete .xml and .bin files to ones containing the correct detection model. This totally simplifies the software development and can be quickly executed even by inexperienced primary school students guided by a teacher who has completed only a short training.

### **6 Tests**

Real life tests have been conducted during the Doors Open Day 2022 at IHP institute. The robot has been placed at the floor next to a stand. Other experiments have been also presented in the same space. The forward-backward operation feature has been disabled to prevent the robot from moving away from the stand and colliding with someone.

The reaction of the targeted age group - children aged 7–15 years has been very enthusiastic. The bright yellow colour of the robot attracted their attention. Smaller children have been mostly interested in playing with the robot moving from side to side and observing how it has followed them. Older primary school students generally have wanted to know how the whole thing works and what more it could do. We have received questions about used components and software. All in all, the presented robot has drawn much more attention and vigorous reaction than any of the Arduino sets used during the organized by us workshops mentioned before.

## **7 Conclusions**

Our observations show that teaching primary school students using LEGO sets with additional electronic extensions seems to be a much better idea than introducing them to the Arduino. LEGO is well recognised among children, they know how to use it and what can be constructed with it. Younger children associate it with good fun and creativity freedom, older ones additionally with complex mechanics and possibility to control the components movements of LEGO Technic sets. Complementing LEGO with external microcontrollers and simple artificial intelligence creates a perfect educational set for primary school students. They can learn programming through playing and improve their creativity and technical skills by building diverse interactive constructions fulfilling various tasks. These can encourage them to develop competences in STEAM what is the aim of the "Laboratory of the Future" educational programme and will be desired by the economy changed under the influence of the Fourth Industrial Revolution.

**Acknowledgements.** This work was supported by the European Regional Development Fund within the INTERREG V A BB PL2014-2020 Programme, "Reducing barriers using the common strengths", project "SpaceRegion: Cross-border integration of the space sector", grant number 85038043.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## A Method for Co-designing Immersive VR Environments with Users Excluded from the Main Technological Discourse

Wiesław Kopeć1,2,4 , Anna Jaskulska4 , Barbara Karpowicz1, Grzegorz Pochwatko2,4 , Monika Kornacka3,4 , and Kinga Skorupska1,4(B)

1 Polish-Japanese Academy of Information Technology, Warsaw, Poland<EMAIL_ADDRESS>
2 SWPS University of Social Sciences and Humanities, Warsaw, Poland

3 Institute of Psychology, Polish Academy of Sciences, Warsaw, Poland

4 Kobo Association, Warsaw, Poland

Abstract. With the advent of new interfaces and modes of interaction related to virtual, augmented and mixed reality (VR, AR, MR) or voice user interfaces (VUI) a need to explore new approaches to foster rapid software prototyping and development emerges. Drawing from our experiences in human, cooperative, and collaborative aspects of software engineering, based on IT-empowerment and participatory design with older and younger adults, we propose and examine some methods, practices and tools for co-designing software for the immersive extended reality continuum (XR), including VR, AR or VUI. In a series of empirical and field studies we examined various experimental setups for stakeholders' participation within and across different steps and phases of the software development process for immersive extended reality environments (IERE). In this article we provide an overview of selected methods, practices and tools, that we found best support communication, collaboration, and cooperation among stakeholders, especially the members of vulnerable groups who are often excluded from the main technological discourse and need more empowerment.

Keywords: Virtual Reality *·* Participatory Design *·* Software Development

## 1 Introduction

Intensive growth of virtual, augmented and mixed reality solutions opens up new possibilities for better immersive experience of software solutions for end users. However, the rapid growth of those areas that comprise the immersive extended reality continuum (XR), brings new challenges to the software development process and teams. Despite the wide range of existing methods, tools and approaches for developing diverse mobile and desktop environments, including user-centered design coupled with end-users participation, i.e. co-design and participatory design approaches, there is still a need to select and tailor the most promising solutions to new interfaces and modes of interaction of the XR continuum. Recently increasing pace of both hardware capabilities and its proliferation puts pressure on software developers to deliver more content as more diverse groups of users enter the market for Immersive Extended Reality Environments (IERE)1. Therefore, the objective of this paper is to present our findings regarding the methods, practices and tools aimed to facilitate direct involvement of users of varying ICT-proficiency levels, particularly from vulnerable groups, in the participatory design process for the development of new interfaces and modes of interaction related to virtual, augmented and mixed reality (VR, AR, MR). In a series of empirical and field studies [1,2,6,9–11,18,23] we examined various experimental setups for stakeholders' participation within and across different steps and phases of the software development processes. Therefore, we provide an overview of selected methods, practices and tools, that we found best support communication, collaboration, and cooperation among stakeholders, especially members of vulnerable groups who are often excluded from the main technological discourse.

## 2 Related Work

Participatory design, or co-design, engages users in the development processes [20,21], allowing them to directly shape the designed solutions at their core. Users can provide insights at different stages of the process [5,15] - or, according to Ladner [14], become designers themselves, in Design for User Empowerment. Without this, even solutions meant to work towards Social Good can be tinted by stereotypes [24], which can become evident in unexpected places due to unconscious biases held by the project team, putting into question the ethical aspects of such solutions, as some some stereotypes may trickle down into the immersive environments built. [4] To prevent this, user engagement during the design and development is crucial. One way of approximating it is by forming Living Labs. The term *Living Lab* itself was coined by William Mitchell from MIT [16] and indicated a space where routines and everyday life interactions of users and new technology could be observed and recorded [3,22], to examine the users' needs and avoid business risks [19]. However, besides problems with maintaining long-term and sustainable user communities [17] a realistic Living Lab, requires significant investments. Some alternatives, like lightweight living labs [9], were proposed where users were animated with interesting activities which keeps them engaged. This approach is also useful for new and emerging interfaces such as

1 In this paper we define Immersive Extended Reality Environments (IERE) as environments which use new technological solutions running software to enhance or replace the experience of reality by appealing to the users' various senses. What these environments have in common is the use of the real life metaphor, thus making participatory design rely more on ethnographic studies which evaluate and elaborate on the nature of users' interactions with technologies which have a chance to become truly embedded in their daily lives.

extended or mixed reality (XR), like VR, AR or voice interaction. There is also a lot of room for further research to establish a comprehensive and ubiquitous interface that combines all of the extended reality technologies. In this vein, it is possible to engage users in a distributed Living Lab environment [11] aimed at testing and developing these solutions, and eventually even engage them in start-up development teams, as in the SPIRAL method described in one of the previous CHASE workshops [9]. Our RAPID approach draws from all of these methods, while opening the discussion for further exploration of new and emerging interfaces on the extended reality continuum in the context rapid software development.

## 3 RAPID Approach Outline

Based on our previous experience with empowering older adults and other vulnerable groups for participatory design, we developed an instant environment called RAPID (Rapid and Agile Participatory Interactive Development) devoted to lightweight Living Lab conditions. The RAPID approach consists of three phases divided into steps and for each of these we list related methods and expected outcomes.

## 3.1 I. Preliminary Phase: Team Formation

This preliminary stage consists of two steps, that can be realized in a few days or partially omitted, in particular step 1. in experienced teams or even step 2. based on communities such as Living Labs or local activity groups.

Step 1. Core XR Team Formation and Empowerment. Internal workshops with methods presentation, discussion and roles assignment. This is an important step when we prepare the team for direct cooperation with users as they may hold some stereotypes about them as well. Internalization of methods is important, especially to ensure unbiased and open cooperation with vulnerable users and among each other in the in-team role assignment, i.e. product owner, team leader, technology advisor, developer, designer etc.

Expected Result: Team members understand the importance and goal of the process and are open to direct cooperation with end-users, while remaining aware of their unconscious biases.

Step 2. XR Team Extension: User Engagement. Preliminary trials with representatives of the group of end users, user engagement, recruitment and selection in order to extend the core team (from local communities, groups, Living Labs). User engagement session. Preliminary trials with end-users as potential team members. Controlled audio-visual immersion into the IERE world. Direct interaction (i.e. controllers) optional, not required.

### Methods and Tools that Can be Used:

Expected Result: Recruitment of two to six end-user representatives with high motivation and a creative flair for the main co-design phase session and team participation.

### 3.2 II. Main Phase: RAPID IERE Development

Main phase consists of two stages: user empowerment stage (steps 3 and 4) and co-design development stage (steps 5, 6 and 7). Each stage can be realized in one day, or extended as needed.

Step 3. Introduction: Discussion of the Goal of the Workshops and Current Pre-workshop Expectations of Participants. An informative and motivating introduction is key, as both the potential users and the development team need to realize the purpose and importance of their work. Additionally, the participants ought to introduce themselves, to provide their background on technology use, and their experience with the specific topic and their own expectations of these workshops. When engaging vulnerable groups it is crucial to not strain their resources and to ensure that the benefit is mutual for the parties involved. Next, in order to set baseline expectations, we introduce our participants to the idea of the project without putting it in the context of specific technologies nor equipment.

## Methods and Tools that Can be Used:

Expected Result: The participants understand the importance and goal of the workshops and share their experiences with the subject matter of the workshop (e.g. in our cases: banking technology and ATMs, potential uses of Smart Home technology with VUIs and BCI, workplace stress, intrusive thoughts and relaxation techniques).

Step 4. Immersion with Interaction: Hands-On Presentation of the IERE Technology. This step is crucial for the target group to get a feel of what is possible in the technology of their choice, so that they are not limited by their presuppositions or lack of experience in the development stages that follow. It may also be beneficial for the target group to witness the development team also discovering something new and engaging within the demos. Moreover, experiencing immersion may get the users excited about this technology, awakening their creativity. In this step the target users try IERE gear and engage in interactive activities, which to some extent may be similar to the ones being developed later, either in scale, means of interaction or goal (Fig. 1).

Fig. 1. In one of our workshops older adults played the NVIDIA-R VR Funhouse game, to become aware of the wide range of what is possible in VR

Expected Result: The participants understand the range of possibilities that exist in IERE and get excited about the technology.

Step 5. Development - Environment in IERE. While working on any type of project that involves software engineering many environmental factors come into consideration, such as type of devices, software choices or general theme. While in our cases most of those factors were predetermined by the concepts of the project, hardware solutions available to us at the time and team's skill, we were able to involve all of our participants in the process of designing XR playing space and brainstorming on the potential future uses of other IERE solutions, while also gaining valuable insights in their real life preferences.

Fig. 2. We printed out VR Sketch Sheets, to be used for web-based XR environment design from https://blog.prototypr.io/, however, using them directly with members of our target group proved to be too reliant on an unknown metaphor and too detached from the familiar experience.

Expected Result: The participants are involved in project defining decisions, can visualize and evaluate the suitability of different environments, weigh the dangers and benefits associated with each choice, and come up with their own ideas regarding environmental variables.

Step 6. Development of the Concept. Working in IERE environments opens a whole new range of possibilities regarding UI and UX, as such interfaces rely on approximations of real interactions to a greater extent, therefore it is of utmost importance to have as much feedback as possible. By using quick photoshop mockups, 3D modeling software and interaction and flow mock-ups in web-based tools like Proto.io, we were able to gather important insights on what our participants expect from our projects, while allowing them to quickly see mock-ups of their ideas (Fig. 3).

Fig. 3. Older adults were shown web-based VR mock-ups from Proto.io on smartphones

2 The Sketch Sheets are available from: https://blog.prototypr.io/vr-paperprototyping.

Expected Result: The participants conceptualize and create UI and UX elements while having the ability to give immediate feedback to our rough interpretations.

Step 7. Development - Functionalities. This step, as one too technical for easy and quick implementation, focuses solely on the participants' ideas and insight without prototyping the functionalities there and then. Participants explore ideas and conceptualize their implementations and restrictions, to go along with the project concept and future and existing assets. It is important to give freedom to participants' ideas and leave out any implementation restrictions. While applicable pre-made sets of interactions can be presented during discussion as a starting point, it is not necessary.

## Methods and Tools that Can be Used:

Expected Result: The participants conceptualize project functionalities, without software/engine restrictions.

#### 3.3 III. Closing Phase: XR Product Delivery and Testing

This phase is from the classic user-centered design approach, so we do not elaborate on it.

Step 8. Testing with End Users. We would like to point out three important groups of methods, i.e. qualitative interviews and observations and quantitative sensor-based methods and tools.

#### Methods and Tools that Can be Used:

Expected Result: Verifying the results with extended team members and new participants, as well as the extensive network of contacts they all have to gather fresh perspectives. Testing is important and can be done outside of the work setting, by presenting the demo at a conference or attending fairs, events where members of the target group may be present.

## 4 Discussion

Our RAPID approach was developed and verified in design cases, for example a series of design workshops of a virtual simulator of ATM and well-being XR environments [12], while its user-empowerment elements were tested in research related to Voice User Interfaces (VUI) [13] and Brain Computer Interfaces (BCI) [8] for IoT, as well as in a case study of an all round XR-hackathon with several dozens of stakeholders from various groups, including software engineers and content developers [6,7]. Overall, the RAPID approach is well-suited for creating XR solutions with users who need to be empowered, in our case older adults3, and we expect this to be true for other vulnerable groups at risk of being viewed through the lens of stereotypes or just feeling vulnerable in new circumstances.

## 4.1 Phase I: Team Formation

In the case of participatory design high motivation and engagement are crucial, which is why RAPID Phase I is essential for the success of the approach. Here are some takeaways from our practice:

## 4.2 Phase II Development

After the users excluded from the main technological discourse join the development team, empowerment is even more important. They are now among people who are tech-saavy, driven and may dominate the discussion. Some takeaways from our studies indicate that:

3 The groups needing empowerment may be not as obvious, as in different contexts we enter different relationships where knowledge or power are not equally distributed. This is the case with students vs teachers, employers vs employees but also clients and contractors or even scientists in interdisciplinary teams.

## 4.3 Phase III Closing

The UX tests were conducted both by the users and the development team in order to maintain the fresh approach and view on the different aspects of the ATM and its usability, controls, environment and the final purpose as a training simulation and the well-being environment (Fig. 4).

Fig. 4. Example of immersion verification with eye-tracking and psychophysiology

## 4.4 Other Considerations

Scheduling. RAPID is designed to be concise but scalable. This means, that it can extended or compressed to just five days, as follows:

Moreover taking into account the typical iterative nature of agile approach and rapid prototyping, some steps can be omitted i.e. core team empowerment or user recruitment, based on previous cycles of software development process or previous projects. Below we present a possible schedule of implementation of the RAPID approach as a sprint.

It is vital to schedule the meetings with enough time for questions and digressions since the whole team and the users alike need to feel their presence and insights are of importance - to an extent even if they relate to the issues outside of the main goal of the workshop. While they may not be crucial for the end product, these questions are essential for the process that relies on open sharing. Often such workshops run long, so ordering catering is another good practice.

Venue. The RAPID approach will work best if the venue changes, depending on the focus of each phase.

#### 4.5 General Discussion

Despite the fact that many methods and tools were developed in recent years, there is still a room for discussion to propose a better, more comprehensive approach for the software development teams to address the challenges of constantly changing hardware landscape of IERE. For instance, many of use cases described at premier software engineering and HCI conferences were based on the rapid prototyping using cardboard platforms, thus became more or less obsolete or hard to implement, as support for these weakened or ceased. On the other hand, there are also some shortcomings of paper prototyping that are not easy to overcome, because new interfaces and modalities are not just a simple extension of the WIMP paradigm (windows, icons, menus, pointers). In contrast, methods for prototyping 2D interfaces are well-established in both software engineering and human-computer interaction with plethora of effective tools, including low fidelity mock-ups that can be prepared even by the end-users themselves.

Thus, drawing from our research experience, we concluded that if even users who are generally excluded from the main technological discourse can creatively and competently engage in such 2D prototyping activities with a proper empowerment. On the other hand, during our research on software engineering for advanced and multi modal IERE interfaces we observed that unlike prototyping mobile or web-based flat environments and applications there is a gap between proficient software designers, developers and end-users. This gap is evident in the field of paper prototyping of immersive interactive environments, which constitutes an important barrier to direct cooperation with end-users, content designers and software engineers. Even empowered users have significant difficulties in bridging the gap between low-fidelity paper prototypes and hi-fidelity immersive multidimensional interaction. This situation resembles the case of architectural design, where drawings and schemes are not enough for many end users to fully understand and imagine the final solutions, hence the industry practice of 3D visualisation of models and spaces, more recently with the use of VR-based solutions. In our case, the same metaphor applies, where prototyping for IERE has to "hit closer to home" to enable the users to fully understand the functionalities of end products and to be able to contribute feedback and relevant insights. These low-fidelity functioning prototypes are generated based on insights from the earlier steps in the development cycle, which rely heavily on the empowerment of users, consisting of demonstrations, free use of technology and discussions with the team, unhindered by fears and false preconceptions, and open to constructive criticism.

## 5 Conclusions

In this paper we outlined the RAPID approach with example methods and tools based on a series of case studies. It is an attempt to discuss and explore new approaches to rapid software prototyping and development relying on userempowerment of users typically excluded from the main technological discourse and facing unconscious biases in the context of immersive extended reality environments (IERE). The proposed approach allowed us to gain valuable insights on the development process of Immersive Extended Reality Environments, including such aspects as UX, UI, 3D models, web-based mock-ups and current and future functionalities, but it remains to be seen whether the software produced with this approach perform reliably better and are more likely to be used by these groups. At the same time we are excited about the prospects of the discussions and follow-up studies, further testing the limits of such sprints, and refining such IERE development methods to facilitate the creation of great experiences directly with users excluded from the main technological discourse, empowered to share their opinions, needs and aspirations.

Acknowledgments. This study constitutes an example of a bottom-up participatory research initiative done in the spirit of transdisciplinary collaboration between scientists, practitioners and volunteers. It was conducted without a dedicated grant to further the understanding of key concepts in HCI in the context of immersive virtual environments (IVR). Therefore, we would like to thank the many people and institutions gathered together by the Living Lab Kobo and HASE Research Group. First, we would like to thank all the members of HASE research group (Human Aspects in Science and Engineering) and Living Lab Kobo for their support of this research. In particular, the members of XR Lab and XR Center of Polish-Japanese Academy of Information Technology (PJAIT), Emotion-Cognition Lab SWPS University (EC Lab), Psychophysiology Lab of the Institute of Psychology Polish Academy of Sciences, Kobo Association, as well as Living Lab Kobo community, especially older adults, for supporting recruitment and their participation in the lab studies.

## References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Improving the Usability of Requests for Consent to Use Cookies**

Kristina Lapin(B) and Laima Volungeviˇci ¯ut˙e

Vilnius University, Vilnius, Lithuania<EMAIL_ADDRESS>
**Abstract.** An HTTP cookie (hereinafter cookie) is a piece of information that maintains a state in a stateless HTTP protocol. Recently established privacy and security regulation imposes an obligation on the service provider to obtain the user's consent to use cookies. This paper is aimed at studying usability guidelines of requests for consent to use cookies (hereinafter consent requests). The consent requests have usability issues that make it difficult for the users to choose the right privacy and security options. A study of privacy and security regulation is aimed at extracting design requirements. Manipulative designs also known as dark patterns are explored and applied to assess consent requests of two of the most popular Lithuanian news portals. An evaluation revealed the presence of dark patterns in consent requests as well as violation of privacy and security requirements. As a result, usability guidelines on the design of cookie consent requests are developed.

 This is a collection of information the server creates when a user visits a website [2]. The information stored in the cookies helps the website to identify the user or restore the options set by the user – it can be login, the pages visited by the user on the website, or other usability options, such as language and font size.

Cookies are used to check whether two received requests were sent from the same web browser and to remember the state of the website, for example, stored information about what goods are placed in an e-shop cart. Although the purpose of cookies is to track the state of the web page which would be lost when the user leaves the domain, more detailed purposes are often distinguished [3]: (a) *strictly necessary cookies* for the smooth functioning of the website; (b) *preferences cookies,* such as language, font size, login name, and password; (c) *statistics cookies* collect information about the user's behavior on the website, such as links the user visited; (d) *marketing cookies* track user activity on the Internet to help deliver personalized advertising.

Depending on the collected content, cookies may endanger users' privacy [4]. Users' devices and their information are recognized as personal space in the European Union (hereinafter EU) law. According to EU law, to process users' personal data, service providers must inform them about the methods of data processing and obtain their consent [5]. Implementing the law, cookie consent requests are introduced on the websites. However, the consent requests require additional mental effort that occurs unexpectedly when opening the page, thus usability issues arise.

The design of consent requests usually does not help the users understand what they are agreeing to. Even more, designers use knowledge of user behavior (e.g., psychology, A/B testing) to generate as many consents as possible when designing consent requests. Such misleading designs are called dark patterns [6]. They are also widely used in social blogs, online stores, mobile apps, and even computer games [7]. Consent requests include various dark patterns, such as information hiding, manipulation of element positions, formatting, blocking of the website content, the abundance of choices, hard-to-see text or links to additional settings, and pre-marked choices [8]. This manipulation degrades the user experience of using the website.

This paper aims to formulate usability guidelines facilitating the design of consent requests that meet privacy and security requirements, are easier to understand, and are less annoying to users. For this purpose, the next chapter explores essential privacy and security requirements. The third section examines dark patterns found in cookie consent requests. Further, the evaluation of consent requests of the most popular Lithuanian news portals based on revealed requirements and dark patterns is conducted. Finally, the usability guidelines facilitating the design of more usable cookie consent requests are formulated.

### **2 Privacy and Security Regulation**

The laws regulating the processing of personal data in the European Union are defined in the General Data Protection Regulation (hereinafter referred to as GDPR). In this regulation, personal data is defined as any information relating to a person whose identity can be or has already been identified [9]. According to the EU directive on privacy and electronic communications supplementing it (hereinafter referred to as E-Privacy Directive) [3], service providers must obtain the user's consent to use cookies that are not necessary for the provision of the service [10]. To obtain consent, service providers use requests on their websites that are accepted when the user first visits the website. To process personal data, service providers have to receive the consent of the person concerned. The GDPR sets out the rules for consent-based data processing, some of which relate to the usability of the consent requests [11]:

According to the E-Privacy Directive, cookies may be used provided that users are clearly and accurately informed about the purposes for which they are used [11]. If cookies are not necessary for the functionality provided, such as tracking cookies for market research, it may be necessary to obtain user consent [12]. Users must be informed about what information is provided to the device they use. The user must also be able to opt-out of cookies when other users have access to personal information stored on the device. Information about the purposes and subsequent uses of cookies must be provided together with information about the user's right to refuse cookies before they are used. If the user decides to refuse cookies, the service provider must still provide the minimum services – for example, the website with restricted content.

To process user data, at least one of the several conditions for the processing of personal data provided for by EU laws must be met (e.g. obtaining the consent of the individual). If service providers seek to obtain a person's consent, it should meet certain requirements and recommendations. Table 1 summarizes the requirements and recommendations that consent should be sought.

**Table 1.** Privacy and security requirements for cookie consent requests

### **3 Dark Patterns in Cookies Consent Requests**