**Ilana Nisky · Jess Hartcher-O'Brien · Michaël Wiertlewski · Jeroen Smeets (Eds.)**

# **Haptics: Science, Technology, Applications**

**12th International Conference, EuroHaptics 2020 Leiden, The Netherlands, September 6–9, 2020 Proceedings**

# Lecture Notes in Computer Science 12272

# Founding Editors

Gerhard Goos Karlsruhe Institute of Technology, Karlsruhe, Germany Juris Hartmanis Cornell University, Ithaca, NY, USA

### Editorial Board Members

Elisa Bertino Purdue University, West Lafayette, IN, USA Wen Gao Peking University, Beijing, China Bernhard Steffen TU Dortmund University, Dortmund, Germany Gerhard Woeginger RWTH Aachen, Aachen, Germany Moti Yung Columbia University, New York, NY, USA

More information about this series at http://www.springer.com/series/7409

Ilana Nisky • Jess Hartcher-O'Brien • Michaël Wiertlewski • Jeroen Smeets (Eds.)

# Haptics: Science, Technology, Applications

12th International Conference, EuroHaptics 2020 Leiden, The Netherlands, September 6–9, 2020 Proceedings

Editors Ilana Nisky Ben-Gurion University of the Negev Beer Sheva, Israel

Michaël Wiertlewski Delft University of Technology Delft, The Netherlands

Jess Hartcher-O'Brien Delft University of Technology Delft, The Netherlands

Jeroen Smeets Vrije Universiteit Amsterdam Amsterdam, The Netherlands

ISSN 0302-9743 ISSN 1611-3349 (electronic) Lecture Notes in Computer Science ISBN 978-3-030-58146-6 ISBN 978-3-030-58147-3 (eBook) https://doi.org/10.1007/978-3-030-58147-3

LNCS Sublibrary: SL3 – Information Systems and Applications, incl. Internet/Web, and HCI

© The Editor(s) (if applicable) and The Author(s) 2020. This book is an open access publication.

Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

# Preface

In this volume, you will find the proceedings of the EuroHaptics 2020 conference, held during September 6–9, 2020, in Leiden, The Netherlands. EuroHaptics is a major international conference, organized biannually, that features the latest advances in haptics science, technology, and applications. The 2020 edition of the meeting had a special focus on accessibility, and in the spirit of the focus, we have made the proceedings open access.

The organization of the conference faced a unique challenge, as the start of the review process coincided with the beginning of the COVID-19 pandemic. Like in all aspects of our lives, we also had to reinvent our scientific life, and find a new way to work. Conferences were canceled, moved to later dates, or re-organized as online virtual conferences. Eurohaptics 2020 was rescheduled; instead of late spring (June) it was held in September. Many of the authors, reviewers, and editors faced challenges: lab shutdowns, the transition to online teaching, and caring for our families to name just a few. Considering the event, we are thankful for the effort and dedication to scientific rigor of the 220 external reviewers and 35 associate editors as they juggled additional tasks and limited resources to finish the editorial process. We also want to acknowledge the diligence of authors to improve their papers in the midst of uncertainty and, for many of them, without access to their usual resources. Their commitment though the hardship of the situation made this conference possible.

All aspects of haptics, including neuroscience, psychophysics, perception, engineering, computing, interactions, virtual reality, and the arts were covered in the initial submissions for the full-paper category. The 111 submissions came from 24 countries: 50% from Europe, 37% from Asia, and 10% from North America. The Application, Science and Technology track chairs invited experts to the Program Committee and provided initial advice on three or four papers. The Program Committee members invited three independent reviewers and provided an assessment on the basis of a minimum of two reviews. To be accepted, we expected an innovative idea as well as a diligent experimental evidence, and 60 of the submissions (55%) were accepted for publication. These papers were presented either as oral (36) or as poster presentations (24). In the proceedings, the two types of presentations are published with no distinction between them.

The conference also featured work in progress abstracts which allowed for presentation of late-breaking results as a poster, receiving feedback from the community. The abstracts of these posters are not featured in these proceedings.

Finally, we are delighted to be part of such a dynamic and innovative community, and we believe that the proceedings reflect its creativity and depth of thought. We hope

#### vi Preface

that the works presented in these proceedings will serve as inspiration and spark new ideas for a better future where everyone is haptically connected.

September 2020 Ilana Nisky Jess Hartcher-O'Brien Michaël Wiertlewski Jeroen Smeets

# Organization

# General Chairs

### Webmaster

# Program Committee

Cagatay Basdogan Koç University, Turkey Matteo Bianchi University of Pisa, Italy Luca Brayda Acoesis, Italy Manuel Cruz Immersion, Canada Hsin-Ni Ho NTT, Japan Scinob Kuroki NTT, Japan Alejandro Melendez-Calderon Sabrina Paneels CEA Tech, France

Massimiliano Di Luca University of Birmingham, UK Ildar Farkhatdinov Queen Mary University London, UK Francesco Ferrise Politecnico di Milano, Italy David Gueorguiev Sorbonne Université, France Matthias Harders University of Innsbruck, Austria Jess Hartcher-O'Brien TU Delft, The Netherlands Hiroyuki Kajimoto The University of Electro-Communications, Japan Ayse Kucukyilmaz University of Nottingham, UK Yoshihiro Kuroda University of Tsukuba, Japan Vincent Levesque École de Technologie Supérieure, Canada Monica Malvezzi University of Siena, Italy Sarah McIntyre Linköping University, Sweden The University of Queensland, Australia Anna Metzger Justus Liebig University Giessen, Germany Alessandro Moscatelli University of Rome, Italy Ilana Nisky Ben-Gurion University of the Negev, Israel Shogo Okamoto Nagoya University, Japan Claudio Pacchierotti IRISA, Inria Rennes, France Gunhyuk Park Gwangju Institute of Science and Technology, South Korea Hannes Saal The University of Sheffield, UK Evren Samur Boğaziçi University, Turkey Jeroen Smeets Vrije Universiteit Amsterdam, The Netherlands Massimiliano Solazzi Scuola Superiore Sant'Anna, Italy Luigi Tame University College London, UK Femke van Beek Facebook Reality Labs, USA Michaël Wiertlewski TU Delft, The Netherlands Tae-Heon Yang Korea National University of Transportation, South Korea Mounia Ziat Bentley University, USA

#### External Reviewers

Jake Abbott Diar Abdlkarim Arsen Abdulali Rochelle Ackerley Yasuhiro Akiyama Noor Alakhawand Felipe Almeida Visar Arapi Fernando Argelaguet Chris Awai Yusuf Aydin Mehmet Ayyildiz Stephanie Badde Alfonso Balandra Giulia Ballardini Sandra Bardot Angelo Basteris Edoardo Battaglia Philipp Beckerle Niek Beckers Ayoub Ben Dhiab Corentin Bernard Gemma Bettelani Tapomayukh Bhattacharjee Jeffrey Blum Christian Bolzmacher Luca Brayda Jonathan Browder Joshua Brown Gavin Buckingham Ozan Caldiran Antonio Cataldo Luigi Cattaneo Müge Cavdan Sonny Chan Jacob Cheeseman Katie Chin Francesco Chinello Manolis Chiou Youngjun Cho Inrak Choi

Roger Cholewiak Jean-Baptiste Chossat Simone Ciotti Ed Colgate Patricia Cornelio Martinez Celine Coutrix Manuel Cruz Heather Culbertson Thomas Daunizeau Brayden De Boon Anouk de Brouwer Victor Adriel de Jesus Oliveira Xavier de Tinguy Davide Deflorio John de Grosbois Benoit Delhaye Arthur Dewolf Charles Dhong Massimiliano Di Luca Mariama Dione Kouki Doi Lionel Dominjon Knut Drewing Lucile Dupin Laura R. Edmondson Mohamad Eid Sallnas Eva-Lotta Grigori Evreinov Simone Fani Carlo Fantoni Ildar Farkhatdinov Ahmed Farooq Rebecca Fenton Friesen Manuel Ferre Francesco Ferrise Joshua Fleck Euan Freeman Ilja Frissen Daniel Gabana Chiara Gaudeni Frédéric Giraud

James Goodman Roman Grigorii David Gueorguiev Taku Hachisu Reza Haghighi Osgouei Matthias Harders Jess Hartcher-O'Brien Yuki Hashimoto Zhenyi He Seongkook Heo Souta Hidaka Hsin-Ni Ho Thomas Howard Hsien-Yung Huang Barry Hughes Thomas Hulin Ali Israr Daisuke Iwai Bernard Javot Seokhee Jeon Carey Jewitt Wafa Johal Lynette Jones Fabian Just Hiroyuki Kajimoto Christoph Kanzler Zhanat Kappassov Astrid Kappers Kazuo Kiguchi Heba Khamis Vahid Khoshkava Sunjun Kim Yeongmi Kim Ryo Kitada Christian Koch Ayse Kucukyilmaz Shivesh Kumar Yoshihiro Kuroda Scinob Kuroki Julien Lambert Hojin Lee Daisy Lei Betty Lemaire-Semail Fabrizio Leo

Daniele Leonardis Vincent Levesque Weihua Li Juhani Linna Tommaso Lisini Baldi Daniel Lobo Céphise Louison Jani Lylykangas Yasutoshi Makino Monica Malvezzi Ahmad Manasrah Maud Marchal Juan Martinez Sarah McIntyre Alejandro Melendez-Calderon Mariacarla Memeo Luke Miller Roberto Montano Taha Moriyama Cecily Morrison Selma Music Abdeldjallil Naceri Hikaru Nagano Saad Nagi Takuto Nakamura Frank Nieuwenhuizen Ilana Nisky Andreas Noll Cara Nunez Seungjae Oh Shogo Okamoto Victor Oliveira Bukeikhan Omarali Ata Otaran Lucie Pantera Stefano Papetti Gunhyuk Park Roshan Peiris Nuria Perez Luka Peternel Evan Pezent Thomas Pietrzak Myrthe Plaisier Mithun Poozhiyil

Maria Pozzi Roope Raisamo Roberta Roberts Carine Rognon Daniele Romano Robert Rosenkranz Jee-Hwan Ryu Hannes Saal Jamal Saboune Satoshi Saga Justine Saint-Aubert Mike Salvato Harpreet Sareen Katsunari Sato Immo Schuetz Hasti Seifi Lucia Seminara Irene Senna Yitian Shao Majid Sheikholeslami Anatolii Sianov Stephen Sinclair Nish Singh Jeroen Smeets

Giovanni Spagnoletti Adam Spiers Angelica Torres Matteo Toscani Ugur Tumerdem Femke van Beek Vonne van Polanen Yasemin Vardar Mickeal Verschoor Yem Vibol Fernando Vidal Verdú Julie Walker Ben Ward-Cherrier Antoine Weill-Duflos Michaël Wiertlewski Michele Xiloyannis Heng Xu Hiroaki Yano Yongjae Yoo Juan Zarate Jacopo Zenzeri Claudio Zito Talee Ziv Aaron Zoeller

# Sponsors

# Contents

#### Haptic Science

#### Haptic Technology

#### Haptic Applications

# **Haptic Science**

# The EmojiGrid as a Rating Tool for the Affective Appraisal of Touch

Alexander Toet1(&) and Jan B. F. van Erp1,2

1 TNO Human Factors, Soesterberg, The Netherlands<EMAIL_ADDRESS>2 University of Twente, Enschede, The Netherlands

Abstract. We evaluated the convergent validity of the new languageindependent EmojiGrid rating tool for the affective appraisal of perceived touch events. The EmojiGrid is a rectangular response grid, labeled with facial icons (emoji) that express different degrees of valence and arousal. We previously showed that participants can intuitively and reliably report their affective appraisal of different sensory stimuli (e.g., images, sounds, smells) by clicking on the EmojiGrid, without additional verbal instructions. However, because touch events can be bidirectional and are a dynamic expression of action, we cannot generalize previous results to the touch domain. In this study, participants (N = 65) used the EmojiGrid to report their own emotions when looking at video clips showing different touch events. The video clips were part of a validated database that provided corresponding normative ratings (obtained with a 9-point SAM scale) for each clip. The affective ratings for inter-human touch obtained with the EmojiGrid show excellent agreement with the data provided in the literature (intraclass correlations of .99 for valence and .79 for arousal). For object touch events, these values are .81 and .18, respectively. This may indicate that the EmojiGrid is more sensitive to perspective (sender versus receiver) than classic tools. Also, the relation between valence and arousal shows the classic U-shape at the group level. Thus, the EmojiGrid appears to be a valid graphical self-report instrument for the affective appraisal of perceived touch events, especially for inter-human touch.

Keywords: Affective touch Valence Arousal Emotions EmojiGrid SAM

# 1 Introduction

Next to serving us to discriminate material and object properties, our sense of touch also has hedonic and arousing qualities [1]. For instance, soft and smooth materials (e.g., fabrics) are typically perceived as pleasant and soothing, while stiff, rough, or coarse materials are experienced as unpleasant and arousing [1]. This affective component of touch plays a significant role in social communication. Interpersonal or social touch has a strong emotional valence that can either be positive (when expressing support, reassurance, affection or attraction: [2]) or negative (conveying anger, frustration, disappointment: [3]). Affective touch can profoundly influence social interactions [4]. For example, touch can lead to more favorable evaluations of the toucher [5], can persuade [6], and can regulate our physical and emotional well-being [7]. Since it is always reciprocal, social touch not only emotionally affects the receiver [7] but also the touch giver [8]. Touch is the primary modality for conveying intimate emotions [7, 9]. Current technological advances like the embodiment of artificial entities, the development of advanced haptic and tactile display technologies also afford mediated social touch [10]. To study the emotional impact of touch and to design effective haptic social communication systems, validated and efficient affective selfreport tools are needed.

In accordance with the circumplex model of affect [11], the affective responses elicited by tactile stimuli vary mainly over the two principal affective dimensions of valence and arousal [12]. Most studies on the emotional response to touch apply two individual one-dimensional Likert scales [13] or SAM (Self-assessment Mannikin: [14]) scales [15] to measure both affective dimensions separately. Although the SAM is a validated and widely used tool, it also has some practical drawbacks. People often fail to understand the emotions it depicts [16]. While the SAM's valence dimension is quite intuitive (a facial expression going from a frown to a smile), its arousal dimension (which looks like an 'explosion' in the figure's stomach) is often misunderstood [16], also in the context of affective touch [15]. Hence there is a need for new rating scales to measure the subjective quality of affective touch [17].

We developed a new intuitive and language-independent self-report instrument called the EmojiGrid (see Fig. 1): a rectangular response grid labeled with facial icons (emoji) expressing different levels of valence (e.g., angry face vs. smiling face) and arousal (e.g., sleepy face vs. excited face) [16]. We previously found that participants can intuitively and reliably report their affective response with a single click on the EmojiGrid, even without verbal instructions [16]. This suggested that the EmojiGrid might also be a general instrument to assess human affective responses.

In this study, we evaluated the convergent validity of the EmojiGrid as a self-report tool for the affective assessment of perceived touch events. We thereto used the EmojiGrid to measure perceived valence and arousal for various touch events in video clips from a validated affective image database, and we compared the results with the normative ratings that were obtained with a conventional validated affective rating tool (the SAM) and that are provided with this database. It appears that the brain activity patterns elicited by imagined, perceived and experienced (affective) touch are highly similar. To some extent, people experience the same touches as the ones they see: they have the ability to imagine how an observed touch would feel [18]. This affords the use video clips showing touch actions to study affective touch perception.

Fig. 1. The EmojiGrid. The facial expressions of the emoji along the horizontal (valence) axis gradually change from unpleasant via neutral to pleasant, while the intensity of the facial expressions gradually increases in the vertical (arousal) direction.

#### 2 Methods and Procedure

#### 2.1 Stimuli

The stimuli used in this study are all 75 video clips from the validated Socio-Affective Touch Expression Database (SATED: [15]). These clips represent 25 different dynamic touch events varying widely in valence and arousal. The interpersonal socio-affective touch events (N = 13) show people hugging, patting, punching, pushing, shaking hands or nudging each other's arm (e.g., Fig. 2, left). The object-based (non-social) touch events (N = 12) represent human-object interactions with motions that match those involved in the corresponding social touch events, and show people touching, grabbing, carrying, shaking or pushing objects like bottles, boxes, baskets, or doors (e.g., Fig. 2, right). Each touch movement is performed three times (i.e., by three different actors or actor pairs) and for about three seconds, resulting in a total of 75 video clips. All video clips had a resolution of 640 360 pixels.

Fig. 2. Screenshots showing an interpersonal (socio-affective) touch event (left) and a corresponding object-based touch event (right).

#### 2.2 Participants

English speaking participants were recruited via the Prolific database (https://prolific.ac ). A total of 65 participants (43 females, 22 males) aged between 18 and 35 (M = 25.7; SD = 5.0) participated in this study. The experimental protocol was reviewed and approved by the TNO Ethics Committee (Ethical Approval Ref: 2017–011) and was in accordance with the Helsinki Declaration of 1975, as revised in 2013 [19]. Participation was voluntary. After completing the study, all participants received a small financial compensation for their participation.

#### 2.3 Measures

Demographics. Participants reported their age, gender, and nationality.

Valence and Arousal. Valence and arousal were measured with the EmojiGrid (see Fig. 1; this tool was first introduced in [16]. The EmojiGrid is a square grid that is labeled with emoji showing different facial expressions. Each side of the grid is labeled with five emoji, and there is one (neutral) emoji located in its center. The facial expressions of the emoji along a horizontal (valence) axis vary from disliking (unpleasant) via neutral to liking (pleasant), and their expression gradually increases in intensity along the vertical (arousal) axis. Users can report their affective state by placing a checkmark at the appropriate location on the grid.

### 2.4 Procedure

The experiment was performed as an (anonymous) online survey created with the Gorilla experiment builder [20]. The participants viewed 75 brief video clips showing a different touch event and rated for each video how the touch would feel. First, the participants signed an informed consent and reported their demographic variables. Next, they were introduced to the EmojiGrid response tool and were told how they could use this tool to report their affective rating for each perceived touch event. The instructions stated: "Click on a point inside the grid that best matches how you think the touch event feels". No further explanation was given. Then they performed two practice trials to familiarize themselves with the EmojiGrid and its use. The actual experiment started directly after these practice trials. The video clips were presented in random order. The rating task was self-paced without imposing a time-limit. After seeing each video clip, the participants responded by clicking on the EmojiGrid (see Fig. 1). Immediately after responding the next video clip appeared. On average the experiment lasted about 10 min.

#### Data Analysis

IBM SPSS Statistics 25 (www.ibm.com) for Windows was used to perform all statistical analyses. Intraclass correlation coefficient (ICC) estimates and their 95% confident intervals were based on a mean-rating (k = 3), absolute agreement, 2-way mixed-effects model [21]. ICC values less than .5 are indicative of poor reliability, values between .5 and .75 indicate moderate reliability, values between .75 and .9 indicate good reliability, while values greater than .9 indicate excellent reliability [21]. For all other analyses, a probability level of p < .05 was considered to be statistically significant.

For each of the 25 touch scenarios, we computed the mean valence and arousal responses over all three of its representations and over all participants. We used Matlab 2019a (www.mathworks.com) to investigate the relation between the (mean) valence and arousal ratings and to plot the data. The Curve Fitting Toolbox (version 3.5.7) in Matlab was used to compute a least-squares fit of a quadratic function to the data points.

#### 3 Results

For each touch-scenario, the mean and standard deviation response for valence and arousal was computed over each of its three representations and over all participants.

To quantify the agreement between the ratings obtained with the EmojiGrid (present study) and with the 9-point SAM scales [15] we computed Intraclass Correlation Coefficient (ICC) estimates with their 95% confidence intervals for the mean valence and arousal ratings between both studies, both for all touch events, and for social and non-social events separately (see Table 1). For all touch events, and for social touch events in particular, the valence ratings show excellent reliability and the arousal ratings show good reliability. For object-based touch events, the valence ratings also show good reliability, but the arousal ratings show poor reliability.

Figure 3 shows the correlation plots between the mean valence and arousal ratings obtained with the EmojiGrid in this study and those obtained with the SAM in [15]. This figure shows that the mean valence ratings for all touch events closely agree between both studies: the original classification [15] into positive, negative and neutral scenarios also holds in this result. A Mann-Whitney U test revealed that mean valence was indeed significantly higher for the positive scenarios (Mdn = 7.27, MAD = .31, n = 6) than for the negative ones (Mdn = 2.77, MAD = .35, n = 6), U = 0, z = −2.88, p = .004, r = .58 (large effect size). Additionally, mean valence differed significantly between positive social touch scenarios and object-based touch scenarios (Mdn = 5.13 MAD = .35, n = 12), U = 0, z = −3.37, p = 0.001) and between negative social touch scenarios and object-based touch scenarios, U = 0, z = −3.42, p = 0.001, r = .68 (large effect size).

Whereas the mean valence ratings closely agree between both studies for all touch events, the arousal ratings only agree for the social touch events, but not for the objectbased touch events. The mean arousal ratings for object-based touch events are consistently higher with the EmojiGrid than with the SAM. We also compared mean arousal between social touch and object touch. The results revealed that social touch was rated as more arousing (Mdn = 5.35, MAD = 0.89) than object-based (non-social) touch (Mdn = 4.10, MAD = 0.37, U = 27.0, z = −2.77, p 0.009, r = .55 (large effect size). Social touch is more arousing than object-based (non-social) touch. This agrees with Masson & Op de Beeck's conclusion that completely un-arousing social touch does not seem to exist [15].

Figure 4 shows the relation between the mean valence and arousal ratings for all 25 different SATED scenarios, as measured with the EmojiGrid in this study and with a 9 point SAM scale in [15]. The curves represent least-squares quadratic fits to the data points. The adjusted R-squared values are respectively .75 and .90, indicating good fits. This figure shows that the relation between the mean valence and arousal ratings provided by both self-assessment methods is closely described by a quadratic (Ushaped) relation at the nomothetic (group) level.

All results are available at the OSF repository with https://doi.org/10.17605/osf.io/ d8sc3.

Table 1. Intraclass correlation coefficients with their 95% confidence intervals for mean valence and arousal ratings obtained with the SAM and with the EmojiGrid (this study), for video clips from the SATED database [15].

Fig. 3. Correlation plots illustrating the relationship between the valence (left) and arousal (right) ratings provided with a 9-point SAM scale in [15] and those obtained with the EmojiGrid in the present study. The numbers correspond to the original scenario identifiers in the SATED database. Scenarios 1–6 were originally classified as positive, scenarios 8–13 as negative, and scenarios 7 and 14–25 as neutral.

Fig. 4. Relation between the mean valence and arousal ratings for affective touch video clips from the from the SATED database, obtained with the a 9-point SAM rating scale (blue dots: [15]) and with the EmojiGrid (red dots: this study). The curves represent fitted polynomial curves of degree 2 using a least-squares regression between valence and arousal. (Color figure online)

#### 4 Conclusion and Discussion

The affective (valence and arousal) ratings obtained with the EmojiGrid show good to excellent agreement with the data provided with the database for inter-human (social) touch events. Also, our results replicate the U-shaped (quadratic) relation between the mean valence and arousal ratings, as reported in the literature [15]. Thus, we conclude that the EmojiGrid appears to be a valid graphical self-report instrument for the affective appraisal of perceived social touch events. The agreement for object touch events is good for valence but poor for arousal. This may be related to the instruction to the participants to rate "how the touch would feel". Although not explicitly stated to do so, participants may have preferred the receiver perspective above the sender perspective of the interaction. This perspective has less meaning in the object touch events probably resulting in a switch to the sender perspective. Classic rating tools may be less sensitive to the different perspectives which is only relevant in touch events, hence the extremely low arousal scores with the SAM tool and the low intraclass correlations.

#### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **A 2-DoF Skin Stretch Display on Palm: Effect of Stimulation Shape, Speed and Intensity**

Ahmad Manasrah(B) and Shahnaz Alkhalil

Al Zaytoonah University of Jordan, Amman 11947, Jordan<EMAIL_ADDRESS>https://sites.google.com/view/ahmadmanasrah/home

**Abstract.** Skin stretch has been widely utilized as a tactile display in different haptic applications. However, there has been little research focusing on skin stretch as a modality on the palm of the hand. In this study, a two dimensional tactor apparatus was designed and built to investigate the effects of stimulation speeds, shapes and intensities of skin stretch display on the palm. The tactor moved across the palm at different speeds to create stimulation shapes on the skin. Subjects reported the intensity of perceived stimuli and predicted speed rate of the tactor and stimulation shape and size. The results showed that there were statistically significant differences in the intensity of perceived tactile displays between different stimulation shapes and sizes. The results also showed the sizes and intensity of the stimulus grow larger with slower tactor speeds.

**Keywords:** Skin stretch *·* Palm *·* Stimulation intensity *·* Tactile display

### **1 Introduction**

There have been significant advances in the field of tactile displays. The literature shows that tactile displays have the potential to improve the user's experience in gaming, teleoperation, and virtual environment simulation. For instance, tactile displays were utilized in delivering shape and material information to users [10] and, also, to deliver instructions and navigational commands [16]. Most of these haptic feedback technologies transfer information via vibrotactile actuators due to their effective perception on the skin and simple implementation [3,7]. However, the vibrotactile feedback does not necessarily provide directional information unless multiple actuators are utilized [17]. This type of tactile display may also create desensitization and discomfort to users at relatively high intensities [21,26].

Recently, skin stretch has been introduced and implemented as an alternative tactile feedback modality in multiple studies. The skin stretch approach has the

**Fig. 1.** Rubber tactor moves in two dimensions on the palm of the hand with different speeds to create a shape. In this case, a circle.

potential to deliver directional information especially with continuous stimuli using only one actuator [2]. The moving actuator applies a directional shear force on the surface of the skin, thus, activating the mechanoreceptors that are responsible of detecting and monitoring moving objects on the skin [15]. Shear forces, applied tangentially to the surface of the skin, result from the friction between the tip of the tactor and the skin [34]. Larger friction forces, for instance, trigger the mechanoreceptors with higher rates, thus increase the intensity of stimulation. In this study, we hypothesize that the speed of the actuator as well as the shape that is created on the palm of the hand can affect the user's perception of the intensity and area of stimulation. A two dimensional plotter mechanism was designed to move a rubber tactor across the skin as illustrated in Fig. 1. The haptic cue is conveyed when the tactor creates deformations of the skin in the direction at which it is traveling, hence, users can identify the drawn shapes and estimate their areas comparatively.

The purpose of this research is to investigate this hypothesis related to how users perceive tactile displays of different shapes and speeds. The results of this investigation may provide a new perspective on skin stretch as a method of delivering information and help the development of tactile display systems in many applications.

#### **2 Background**

Different tactile displays are perceived by different mechanoreceptors in the skin. Each type of mechanoreceptor is excited with the presence of certain stimuli. For instance, fast adapting receptors FA-II perceive the vibration stimulus with different sensitivities based on the frequency and the duration of the vibration [30]. Skin stretch stimuli, however, are perceived by the slowly adapting receptors SA-II [6]. These receptors stay active as long as there is a stimulus on the skin. Moreover, it has been shown that tangential forces caused by skin stretch can be perceived accurately and quickly by humans [4].

Skin stretch stimuli have been widely implemented in tactile display systems on different locations of the skin. Earlier studies have focused on the fingertip to investigate skin stretch as a modality to deliver information. For instance, a previous research showed that humans can differentiate between different tangential and normal forces when the stimulus is applied on the fingertip [20]. Gleeson et al, studied the effects of speed, displacement, and repetition of tangential skin stretch stimuli on the fingertip [12]. Their results showed that higher stimulus speeds improved the accuracy and perception of direction. Other studies investigated creating complex displays of virtual objects on the fingertip using skin deformation [14,25]. Yem et al. even produced a rubbing sensation on the fingertip using one degree of freedom movement [32].

There are also other studies that investigated skin stretch displays on the forearm [8] and on the lower extremities [7]. However, in comparison to other locations on the skin, few studies are found regarding skin deformation on the palm of the hand. Even though the palm is less sensitive than the fingertips, it has similar densities of SA-II receptors [29] and provides a larger display area for skin stretch stimulation [13]. Studies have shown that tactile displays which are applied on the fingers and the palm simultaneously improve users' perception of relatively large virtual objects [27]. Skin stretch stimuli on the palm have been used to deliver driving information and direction to drivers through the steering wheel [23].

Other studies have investigated several factors that may affect skin stretch and skin deformation. For example, Edin et al. applied tangential forces at different locations on the body including the palm of the hand. Their results proved that the speed and direction of the stimulus affect the perception of skin stretch displays [9]. A more recent study also showed that the intensity of skin stretch is affected by the speed and displacement of the stimulus [13]. The method presented here investigated how the intensity and size created by the stimulus were perceived based on different speeds and stimulation shapes.

#### **3 Method**

This investigation consisted of one set of experiments studying several tactor speeds and stimulation shapes. In each experiment, subjects perceived a random stimulation shape with a certain size and speed. Three stimulation shapes were used: a circle, a square, and an equilateral triangle at two different sizes and two tactor speeds.

#### **3.1 Apparatus**

A two dimensional plotter was designed and built specifically for the purpose of this study. Unlike conventional 2D plotters, the stylus was replaced with a tactor and directed upwards so that the palm can rest on it. The tactor's tip was made out of rubber and had a spherical shape with a diameter of 4*mm* which is about the size of a board pin. The plotter moved the tactor using two stepper motors attached in an H-bot connection capable of producing different speeds as shown in Fig. 2. The tactor was mounted on a 3D printer base that was connected to the plotter contact surface. All the internal components and connections of the two dimensional plotter were covered with a white box which was cut out in the center to allow the tactor to move freely in a 7 *×* 7 cm space which is slightly less than the size of the average palm for an adult [24].

**Fig. 2.** Two dimensional plotter was built using two stepper motors. The rubber tactor is installed on the mount.

### **3.2 Experimental Setup and Procedure**

At the beginning of the experiments, the nature of the study and procedures were generally explained to the subjects. After that, they were asked to sit in front of a screen and rest their right hand on the apparatus without applying extra force. Participants were also asked to wear headphones playing steady music to block noises generated from the apparatus and surroundings. Before the experiments began, a test-experiment was given to subjects where a stimulation shape is randomly applied on the participants' palms at two speeds 10 mm/s and 20 mm/s. Participants were then asked to rate the intensity of stimuli on a scale from zero to four with zero being the weakest and four the strongest. The reason of conducting the test-experiment is to ensure that subjects had a reference intensity point to compare it to the stimuli that they were about to experience at the experiments.

During the experiments, three stimulation shapes were tested, a circle, square, and an equilateral triangle. Each shape was tested with two speeds (10 mm/s and 20 mm/s) and two areas. The stimulation areas were divided into "small" and "large" areas based on the shape. However since three stimulation shapes were used, the areas were not exactly equal for all the shapes. For instance, the "small" area of the circular stimulus was about 4*.*5 cm2 (*r* = 1*.*2 cm), while as for the square, the area was 4*.*4 cm2 (*w* = 2*.*1 cm) and for the equilateral triangle the area was 1*.*7 cm2 (*l* = 2 cm). Table 1 shows the areas and perimeters for all three shapes.

**Table 1.** Stimulation shapes, speeds, and sizes.

With each experiment, a random stimulation shape was applied counterclockwise on the palm with a certain speed and size. The screen in front of the participants displayed all six stimulation shapes (two sizes for each shape). The sizes were exaggerated so that subjects can identify large and small shapes on the screen easily. The display also showed the intensity scale that was described previously. After each stimulation shape was applied, participants recorded their response using the information on the screen. A total of 12 shape, speed, and size combinations were randomly applied on each participant. The experiments took 15 min in total, including a few minutes break. Figure 3 shows the full experimental setup with the screen.

#### **3.3 Participants**

A total of ten subjects participated in the experiments, six males and four females. All of them were healthy, right handed, and between the age of 18 and 50. Each subject read and signed a consent form agreeing to participate in the experiments.

**Fig. 3.** Experiment setup where subject's right hand rests on the apparatus. The screen shows the stretch intensity scale and the shapes and sizes used in the experiments.

#### **4 Results**

In this study the collected data was analyzed using ANOVA with a dependent variable of stretch intensity and four independent variables of tactor speed, stimulation shape and size, and subject. Another ANOVA with a dependent variable of subject answers (shape, size) and three independent variables of stretch intensity, tactor speed, and subject was also conducted. When the results showed statistical differences, a Tukey's honest significant difference (HSD) test was performed as a post-hoc test. All statistical tests were based on alpha value of 0*.*05.

The results of the first analysis showed that the "slow" tactor speed (10 mm/s) had a statistically significantly smaller stretch intensity (*F*(1*,* 106) = 58*.*06, *p <* 0*.*001) than the "fast" stimuli (20 mm/s) in all stimulation shapes.

**Fig. 4.** The means and standard errors for the perceived stretch intensity at two tactor speeds. The average stretch intensity of slow tactor speeds was statistically significantly less than fast speeds.

**Fig. 5.** The means and standard errors for the perceived stretch intensity at two stimulation sizes. The average stretch intensity of small stimulation sizes was statistically significantly less than large sizes.

Figure 4 shows the mean intensity of stimuli for the "slow" and "fast" tactor speeds. The slow tactor speed recorded a mean stretch intensity of 1*.*75 which was between "weak" and "normal". The mean stretch intensity of the fast tactor speed, however, was around 2*.*7, closer to "strong" on the stretch intensity scale. Moreover, the results showed that "small" stimuli sizes also had statistically significantly less stretch intensity (*F*(1*,* 106) = 58*.*67, *p <* 0*.*001) than "large" stimuli sizes within all the applied shapes as illustrated in Fig. 5. The results of stimuli shapes versus stretch intensity showed that the circular shape had statistically significantly less stretch intensity (*F*(2*,* 106) = 26*.*27, *p <* 0*.*001) than the other rectangular and triangular shapes regardless of the size. Figure 6 shows the stretch intensities of the three stimulation shapes. The results did not, however, show any statistical significant differences between subjects.

The second analysis focused on the perception of shapes and sizes of the stimuli. The results showed that subjects perceived the "slow" tactor speed statistically significantly "smaller" (*F*(1*,* 106) = 15*.*79, *p <* 0*.*0001) than the "fast" tactor speed as shown in Fig. 7. Furthermore, seven out of ten subjects missidentified the "small" square at "fast" tactor speed as a circle of the same size.

**Fig. 6.** The means and standard errors for the perceived stretch intensity at three stimulation shapes. The average stretch intensity of the circular shape was statistically significantly less than other stimulation shapes.

#### **5 Discussion**

In this study, the shape, speed, and intensity of skin stretch stimulation were investigated on the palm. The experimental results showed that the "slow" tactor speed was perceived with statistically significantly less intensity than the "fast" speed. These results agree with previous findings where the intensity of skin stretch perception increases as tactor speed increases [13]. They concluded that the relation between speed and intensity, however, was nonlinear. Other studies showed that an increase in the tactor speed enhances the intensity and accuracy of the perceived stimulus on the finger [12].

Further, the results of our experiments showed that "small" stimuli shapes were perceived with statistically significantly less intensity than "large" stimuli shapes. A small stimulation shape is created via short stimuli distance, thus, producing less stimulation intensity. Caswell et al. showed that shorter tactor displacement yielded a higher accuracy in predicting the direction of stimuli since their intensity increased [5]. The same results were also found in [13]. However, these findings can be related to the spatial distribution of the receptors perceiving the stimuli. Studies have shown that mechanoreceptors are not evenly distributed on the palm as their densities increase towards the fingertips [1,29]. Moreover, the ridges on the palm may have played a role on increasing the intensity of perception for the "large" stimulation shapes. Previous studies indicated that the microstructure of skin strongly affect the perception of skin stretch displays [22].

**Fig. 7.** The means and standard errors of average stimulus sizes at two tactor speeds. The areas created by slow tactor speeds were perceived statistically significantly larger than fast speeds.

Despite the statistical differences between stimuli sizes, the experiments showed that the "circle" stimulation shape had statistically significantly less intensity than the "square" and "triangle" as illustrated in Fig. 6. It seems that the presence of corners in the latter shapes amplified the intensity of stimulation even though the tactor did not stop at those corners during the experiments. These corners might have increased the friction coefficient between the tip of the tactor and skin, thus, increasing the intensity. It has been shown that tangential displacements with friction is perceived with higher sensitivity than frictionless motion [19,28]. Moreover, the triangular shape was perceived with the highest intensity out of the three stimulation shapes. This suggests that acute angles of a stimulus may have an impact on its intensity. Although this effect is distinctly different than the priming effect, where multiple stimuli in the same direction increase the sensitivity of perception [11], it is however worth investigating in the future.

In addition to the statistical differences in the intensity between tactor speeds, the stimulation size, and shape, the experimental results showed that "slow" tactor speeds were perceived statistically significantly larger than "fast" tactor speeds even though the 12 experiments were randomly conducted. Whitsel et al. studied the velocity of a stimulus against its perceived motion and found that distances can be perceived shorter as the stimulus velocity increased [31]. Others have concluded that a stimulus length is perceived as result of speed and duration [33]. Such a phenomenon can occur due to the spatial properties of the receptors in the skin [18]. The time duration in our experiments varied from approximately 7 seconds to 15 seconds for each experiment depending on the tactor speed and stimulation shape. It is possible that subjects may have perceived the stimulation shapes of "fast" tactor speeds as smaller. Moreover, seven out of ten subjects perceived the "small" square at "fast" tactor speed as a small "circle". This might be related to the similarities between the small square and circle in the perimeters and areas as shown in Table 1.

#### **6 Conclusion**

In this paper, the effects of stimulation speed, shape and intensity on skin stretch were investigated by conducting a series of experiments on the palm of the hand. Ten subjects participated in these experiments where two tactor speeds and three stimulation shapes were tested. The results showed that stimuli intensities of relatively slow tactor speeds were statistically significantly less than the fast ones. The experimental results also showed that relatively small stimulation sizes and circular shapes were perceived with less intensity than large stimulation sizes and other shapes respectively. Further, there were also differences in the perceived sizes of stimuli among stimulation shapes. Future work will focus on studying the direction of the stimulus and stimulation angles of different shapes on the skin.

**Acknowledgment.** The authors would like to acknowledge Al Zaytoonah University of Jordan for funding this research under the grant number 2017-2016/64/17.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **User-Defined Mid-Air Haptic Sensations for Interacting with an AR Menu Environment**

Lawrence Van den Bogaert(B) and David Geerts

Meaningful Interactions Lab (Mintlab), KU Leuven, Leuven, Belgium<EMAIL_ADDRESS>
**Abstract.** Interfaces that allow users to interact with a computing system by using free-hand mid-air gestures are becoming increasingly prevalent. A typical shortcoming of such gesture-based interfaces, however, is their lack of a haptic component. One technology with the potential to address this issue, is ultrasound mid-air haptic feedback. At the moment, haptic sensations are typically designed by system engineers and experts. In the case of gestural interfaces, researchers started involving non-expert users to define suitable *gestures* for specific interactions. To our knowledge, no studies have involved – from a similar participatory design perspective – laymen to generate mid-air haptic sensations. We present the results of an end-user elicitation study yielding a user-defined set of mid-air haptic sensations to match gestures used for interacting with an Augmented Reality menu environment. In addition, we discuss the suitability of the end-user elicitation method to that end.

**Keywords:** Mid-air haptics *·* End-user elicitation study *·* Participatory design

# **1 Introduction**

While advancements in gesture recognition technologies (e.g., Microsoft Kinect, Leap Motion) open up exciting new ways of human-computer interaction, one important constraint they face is their lack of a haptic component. One technology that carries the potential to address this shortcoming, without impeding the user with wearables or hand-held devices, is ultrasound mid-air haptic feedback [2,3]. By generating pressure fields in mid-air via an array of ultrasound transducers, this technology allows the user to experience a sense of touch on the palm and fingers of the unencumbered hand. The ultrasound waves that create these pressure fields can be meticulously altered, opening up a quasi-infinite set of sensations, shapes and patterns. Typically, it is experts and specialists who are tasked with their design. For gestures, however, it has already been argued that even though designer-designed gestures might be expertly crafted, end-user elicited gestures are more intuitive, guessable and preferred [4]. Accordingly, we believe this might be true for mid-air haptic sensations too. To our knowledge, so far only one other study has involved end-users to create mid-air haptic sensations [5]. However, whereas their goal was for participants to create mid-air haptic sensations to convey and mediate emotions to others, we wanted to test whether novice users can ideate haptic sensations to match gestures used for operating a menu in Augmented Reality (AR). To this end, we turned to Wobbrock et al.'s *end-user elicitation methodology* [12]. Typically, such elicitation studies present participants with one or more *referents* for which they are asked to ideate a suitable *symbol*. A referent is the effect of an interaction with an interface (e.g., an increase in volume on a music player). The symbol is the corresponding action (input) that is required to invoke the referent (e.g., turning the volume knob clockwise). In principle, a symbol can be anything that computing systems are able to register as input. This method has been used to design symbols (input) to actuate, among others, touchscreens [13], virtual & augmented reality [6], smart glasses [9], home entertainment systems [10] and public displays [7]. Whereas up until now participants of end-user elicitation studies were tasked with the ideation of the symbol, we assigned ours with the elicitation of what we suggest to call *intermediary referents*. 'Intermediary' means it concerns not the main system output (referent), but rather a form of feedback that accompanies it. Common examples might be the vibrotactile feedback we feel when typing on our smartphone, or the beeping tones we hear when adjusting the volume.

With this work, we present a set of user-defined intermediary referents in the form of mid-air haptic sensations to match a set of gestures used for interacting with an AR interface. We discuss the eligibility of the end-user elicitation method for their design.

#### **2 Study Setup**

Twenty-four non-specialist participants were invited individually to our lab in Leuven, Belgium to elicit a set of five mid-air haptic sensations to match five gestures actuating an AR menu. Given the novelty of mid-air haptics, participants were recruited from a list of people who had previously taken part in a study involving ultrasound mid-air haptics. They were still non-experts, but had at least a basic understanding of what the technology does and how it feels. Ages ranged from 19 to 56 (*µ* = 26*,* 2). Ten participants were male, fourteen were female.

#### **2.1 Apparatus and Gesture Selection**

For our participants to interact with an AR menu, we chose Microsoft's HoloLens as it is by default gesture-controlled and free of hand-held controllers or wearables. Three proprietary gestures allow for its actuation: 1) 'bloom' (used to evoke the home menu at any time to easily navigate in and out of applications); 2) 'air tap' (used to select things, equivalent to a mouse click on a desktop); 3) 'tap and hold' (equivalent of keeping a mouse-button pressed, e.g., for scrolling on a page). 'Air tap' was left out because it is similar to 'tap and hold' in every aspect except for its duration ('air tap' being more brief and thus less suitable for mid-air haptic actuation). In addition to the default HoloLens commands, we turned to Piumsomboon et al.'s user-defined AR gesture set [6] to supplement the HoloLens gestures. From it, we selected three additional gestures for our study: a) 'swipe' (used to rotate a carousel); b) 'stop' (used to pause, e.g., a song or video); and c) 'spin [index finger]' (used to increase or decrease the volume). We chose these three gestures because they were directly applicable to interactions that are possible in the existing HoloLens menu environment (e.g., by using the multimedia player) and because they leave at least part of the palm and fingers unadorned and thus free to be actuated by ultrasound waves (whereas, e.g., a clenched fist would occlude each operable area of the hand).

#### **2.2 Procedure**

The goal of the study was explained to each participant as follows: while wearing a Microsoft HoloLens, they would execute five basic menu-related tasks in AR by using bare-hand gestures in mid-air. For each gesture, they would ideate a mid-air haptic sensation that suits this gesture best and feels most 'logical' for that interaction. At the start of each session, participants were presented a set of ten distinct mid-air haptic sensations to (re)acquaint themselves with the different adjustable properties (i.e., location on the hand, shape of the pattern, duration, dynamics and single-point vs. multi-point feedback).

Using the five gestures described above, participants then completed a set of tasks in AR. For each gesture, participants were encouraged to think out loud, depict and describe what mid-air haptic sensation they would want it to be accompanied by. Sessions had a duration of approximately 45 min. Because wearing an AR headset for a prolonged time becomes cumbersome and considering the time required to elicit and explain in detail each mid-air haptic sensation, we deliberately kept the set of tasks relatively small in comparison to other elicitation studies. For each participant, the entire process was video and audio recorded. In addition, the study moderator took notes; inquired for clarification, and reminded participants to go over all the adjustable properties that constitute a mid-air haptic pattern (cfr. supra).

Naturally, the HoloLens is by default unable to detect Piumsomboon's gestures and respond to them. This is why during each session, the proprietary HoloLens gestures were always visited first. After having performed the two 'functional' HoloLens gestures and having ideated a matching mid-air haptic sensation for them, participants were able to more easily perform the 'nonfunctional' gestures from Piumsomboon's user-defined set and imagine them having the targeted effect. No participants indicated having difficulties with this.

As to not curb creativity, we emphasized that technical limitations and feasibility did not have to be considered. Participants were told to simply imagine what they would want to feel and not worry about how the mid-air haptic sensation would be emitted on the hand. For example, we explained that the mid-air haptics could be emitted from the HoloLens itself (as in a similar setup presented by [8]) rather than from a unit standing on the table.

### **3 Analysis**

All elicited sensations were analyzed and categorized based on the researcher transcripts and audiovisual recordings made during the elicitation phase. Because we allowed participants a high degree of freedom, often minor variations on similar ideas and patterns were proposed. Rather than discriminating sensations from each other for each identifiable objective inconsistency, we assessed and categorized them on a conceptual level. For example, to summon the 'home menu tile' in AR by making a 'bloom' gesture, 11/24 participants suggested a single short mid-air haptic sensation felt on the palm of the hand right after finishing the movement. Whether this sensation had the shape of, e.g., a square or a circle was not deemed imperative to the conceptualization and thus all 11 designs in which 'a short sensation on the palm to confirm that the gesture was well registered' was described, were placed in the same conceptual group.

As such, all 120 ideated sensations were classified based on their conceptual similarity. This resulted in 26 different conceptual groups, each representing a distinct conceptual model of mid-air haptic feedback. To understand the degree of consensus on the conceptual groups among participants, the revised agreement rate *AR* by Vatavu & Wobbrock [11] was calculated for each gesture:

$$AR(r) = \frac{|P|}{|P|-1} \sum\_{P\_i \subseteq P} \left(\frac{|P\_i|}{|P|}\right)^2 - \frac{1}{|P|-1}$$

where P is the total amount of elicited sensations per task and Pi is the subset of sensations with a similar conceptual model for that task. To revisit 'bloom' as an example; the conceptual groups contained respectively 11, 8, 2, 2 and 1 user-elicited mid-air haptic design(s). The agreement score for this gesture is then calculated as follows:

$$AR\_{blom} = \left(\frac{|24|}{|23|}\right) \times \left(\frac{|11|}{|24|}\right)^2 + \left(\frac{|8|}{|24|}\right)^2 + \left(\frac{|2|}{|24|}\right)^2 + \left(\frac{|2|}{|24|}\right)^2 + \left(\frac{|1|}{|24|}\right)^2 - \left(\frac{|1|}{|23|}\right) = 0.308$$

#### **4 Results**

Each participant ideated one mid-air haptic sensation for each of the five gestures, resulting in a total of 120 sensations. Table 1 shows the agreement rate for each gesture/task.

**Bloom.** The bloom gesture, used to summon the home menu tile at any given moment in the HoloLens environment requires all 5 finger tips to be pressed together and then opened into a flat hand, palm facing up, with space between

**Table 1.** Consensus set of user-defined mid-air haptic sensations

each finger. Out of 24 participants, 11 suggested to receive a simple, short onetime feedback on the hand palm that immediately disappears again and confirms that the gesture has been registered well (AR = 0.308). The other conceptual groups were a) 'opening circle on the palm of the hand to mimic the menu being *spawned* from the hand (n = 8); b) 'constant sensation on the entire palm that remains present as long as the menu is opened' (n=1); c) 'horizontal line rolling from the bottom of the palm up to the fingertips' (n = 2); and d) 'mid-air haptic sensation on each fingertip that becomes sensible starting from the moment the fingertips are pressed together until the moment the gesture is completed (n = 2).

**Tap and Hold.** The 'tap and hold' gesture is used to scroll through pages and requires the user to pinch together their stretched-out index finger and thumb and maintain this posture in order for the scroll modus to remain activated. Users can then lift or lower the hand to move the page up and down. Scroll speed can be adjusted by moving the hand further up and down in respect to the original location. The conceptual sensation that was elicited most often here (n = 11, AR = 0.308) was 'a constant sensation on the tip of the thumb and index finger that remains sensible as long as the scroll functionality is active, with the intensity depending on the speed at which the user scrolls'. An almost similar design proposed by 8 other participants also suggested a constant sensation on the tips of the index finger and thumb. However, they made no notion of the intensity corresponding to the scrolling speed.

**Swipe.** To rotate through a carousel, HoloLens users normally have to point and click on elements in the far left or far right of that carousel. We, however, asked our participants to imagine that instead, they could use a full-hand 'swipe' to rotate the elements in the carousel. We clarified that this 'swipe' gesture was appropriated from Piumsomboon et al.'s user-defined gesture set for AR, and that it would not actually work with the HoloLens. Nonetheless, participants were asked to perform the gesture multiple times and imagine the carousel actually rotating. None of them indicated having trouble imagining this work and multiple participants spontaneously commented that this would indeed increase the ease of use. With an agreement rate of 0.228, the conceptual group on which there was most consensus contained the design of 10 participants who suggested to feel a 'vertical line that "rolls" over the entire length of the hand (i.e., from the bottom of the palm over to the fingertips) while making the gesture'. The two (out of five) other most popular conceptual groups were 'sensation (static) that decreased in intensity while moving the hand' (n = 4) and a concept similar to the consensus sensation but with the line being fixed to one location instead of 'rolling' over the hand (n = 5).

**Stop.** Using the HoloLens media player, participants watched a short movie and were asked to imagine they could pause it by briefly holding out a flat hand ('stop gesture') in front of them. This was the second non-functional gesture, yet again, none of the participants indicated having trouble to imagine it work. The consensus on elicited sensations for this gesture was 'very high' (AR = 0.576) [11]. Eighteen participants proposed 'a single short sensation to confirm that the gesture has been registered', similar to the consensus sensation for 'bloom'. The other 6 participants (n = 3, respectively) wanted to feel a) 'a sensation on the hand as long as the movie remained paused (as opposed to one short confirmation)'; and b) 'a sensation that enlarged and/or intensified while the hand was brought forward, to only be emitted at full strength when the arm was completely stretched out and the screen paused.

**Spin.** We asked participants to imagine that they could adjust the volume of the AR environment by spinning their stretched index finger clockwise (increase) or counter-clockwise (decrease). Seven different conceptual groups were needed to classify the heterogeneous ideas that were elicited for this gesture. The design on which there was most consensus (n = 12, AR = 0.297) was described as 'a constant sensation on the index fingertip or on the entire index finger, with intensity increasing or decreasing according to the volume'

#### **5 Discussion**

The presented sensations constitute a user-defined consensus set of mid-air haptic sensations that match gestures used to interact with an Augmented Reality menu environment. Despite the quasi-infinite range of possible mid-air haptic sensations and the idiosyncrasy of its features (timing, dynamics, location on the hand, intensity, . . . ), the agreement scores of our final five gestures (between 0.228 and 0.576) can be regarded as medium (0.100–0.300) to very high (*>*0.500) according to [11]. As the consensus set shows, the majority of participants seemed to prefer relatively simple and straightforward sensations to amplify their gestures. In addition to an inclination towards non-complex sensations, also noteworthy was how participants used mid-air haptic *intensity* in their elicited sensations. Depending on the functionality of the associated gesture, in some cases intensity was indeed a key feature. When the gesture actuated a discrete function (e.g., bloom and stop), a short monotone sensation was usually preferred, whereas more continuous menu-actions (e.g., scrolling through a page or adjusting volume) came with more continuous (i.e., changing intensity) sensations.

Administering the end-user elicitation method to generate what we propose to label as *intermediary referents* instead of *symbols*, shows promise but requires some remarks too. Our study suggests this method to be useful for the ideation of user-defined *mid-air haptic* sensations, however, for other sensory modalities it might not be as easy to have non-experts elicit novel designs and forms. Novices may not be familiar enough with the adjustable variables and properties of a sensorial modality in order to ideate variants of it. This contrasts with the end-user elicitation of, e.g., gestures, as people are apprehensive of their own physical possibilities (and limitations) and therefore inherently more capable of expressing themselves physically. Asking untrained participants to ideate, e.g., auditory intermediary referents (sound design), would assumedly require more training and/or facilitating tools. The think-aloud protocol that was used for our study, in combination with the available UltraHaptics kit on which simulations of elicitations could be depicted, was in our case proficient. However, when having end-users elicit other types of intermediary referents, we advise to assess well up front whether a modality is at all suitable for end-user elicitation and to think of the means or tools necessary to allow participants to elicit new variants of it.

Finally, one way to further validate the results of an elicitation study is what Ali et al. [1] describe as end-user *identification* studies. They are the conceptual inverse of elicitation studies: they present participants with a symbol and ask which referent would be invoked by it. In the case of intermediary referents, then, a control group would be asked to identify from a set of mid-air haptic sensations, which one suits a specific symbol (gesture) best. This would be a beneficial topic for follow-up research.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Surface Roughness Judgment During Finger Exploration Is Changeable by Visual Oscillations**

Yusuke Ota1 , Yusuke Ujitoko1,2(B) , Yuki Ban3 , Sho Sakurai1 , and Koichi Hirota1

 The University of Electro-Communications, Tokyo, Japan<EMAIL_ADDRESS>Research & Development Group, Hitachi, Ltd., Yokohama, Japan The University of Tokyo, Tokyo, Japan

**Abstract.** Previously, we proposed a pseudo-haptic method of changing the perceived roughness of virtual textured surfaces represented by vibrational feedback during pen-surface interaction. This study extends the method to finger-surface interactions in order to change the roughness judgment of real surface. Users watched computer-generated visual oscillations of the contact point while exploring the texture using bare fingers. The user study showed that our method could modify roughness judgment of real textured surfaces.

**Keywords:** Human computer interaction · Haptic interface

#### **1 Introduction**

Tactile texture rendering of virtual surfaces is important for providing the user with a highly realistic interaction experience with virtual objects. Such realistic tactile texture rendering of virtual object surfaces usually require some kind of electromechanical devices.

Recently, pseudo-haptics has attracted attention as a light-weight haptic presentation approach. The pseudo-haptic sensation [3,7] is produced by an appropriate sensory inconsistency between the physical movement of the body and the observed movement of a virtual pointer [4]. As for texture perception, some studies attempted to generate texture perception using the pseudo-haptic effects without sophisticated haptic devices [1,2,5,9,10]. Previously, we proposed a pseudo-haptic method for modulating the vibrotactile roughness of virtual surfaces during pen-surface interactions, and our user study showed the effectiveness of the proposed method [9]. In the method, users watched a visually perturbed contact point. However, since the proposed method assumed pen-surface interactions, users could not sense temperature or friction of the virtual surface.

**Electronic supplementary material** The online version of this chapter (https:// doi.org/10.1007/978-3-030-58147-3 4) contains supplementary material, which is available to authorized users.

**Fig. 1.** We propose a method for modulating the roughness perception of real textured surfaces touched by the users using their bare fingers.

In the present study, we attempt to apply the same method to finger-surface interactions. In that environment, the users watch the visually perturbed contact point while exploring real surfaces with their bare fingers (see Fig. 1). The research question of this research is to investigate whether the method can change the roughness judgment of the real surface during finger-surface interaction.

### **2 Concept**

#### **2.1 Pseudo-haptic Method Using Visual Oscillations**

This study adopted the same pseudo-haptic method using visual oscillation as [9]. Please see details there. The pointer translation is calculated as X*vis* = X*origin*+ C∗α∗random(−1, 1)∗abs(V ) and Y*vis* = Y*origin*+C∗α∗random(−1, 1)∗abs(V ).

Here, X*origin*, Y*origin* are the original pointer position along the x-axis and y-axis. X*vis*, Y*vis* is the translated pointer position along the x-axis and y-axis, which are visualized on screen. The translation amount is obtained as a product of α, a random value sampled from the standard normal distribution, and the absolute value of the device's velocity V . C is a constant. α is the coefficient unit, which means the "size of the visual oscillation."

### **3 User Study 1**

The first user study investigated whether visual oscillations modified the roughness judgment during finger-surface interactions. Ten participants, aged from 22 to 25 participated in the experiment. All participants were right-handed.

#### **3.1 Experimental System**

We manufactured texture plates by arranging glass particles (KENIS, Ltd) on acrylic plates following [6,8]'s approach. Glass particles of different sizes were carefully attached to a flat acrylic plate with a double-sided sticky tape. In this study, we prepared nine texture plates with uniform surfaces, shown in Fig. 2. The diameters of the glass particles were 50, 75.5, 115, 163, 210, 425, 605, 850, and 1194 µm.

**Fig. 2.** Nine texture plates covered with glass particles. The diameters of the glass particles were different for different plates. The length and width of the plate were 100 mm.

Figure 3 (a) shows the experimental environment. One of the texture plates was placed about 30 cm in front of the participants. The texture plate was unseen directly from the participants, owing to a mirror. The position sensor of the tracker (Polhemus Fastrak) was attached to the participants' index fingers.

The position of the index finger was updated at 120 Hz and sent to a personal computer (PC). Unity was used for visualizing the position of a finger, according to the sensing position at 60 Hz. The contact point was visualized as a black circle, and the circle diameter was three pixels, which was equivalent to 0.93 mm on display (ASUS VG278HE, 1920 × 1080, 81.589 dpi). From the viewpoint of the participant, the pointer appeared to be positioned as the same position as their index finger. The participants touched the texture plates with their index fingers, while watching two separate areas on the screen, and judged which area felt rougher (Fig. 3 (b)). The haptic information about both areas, acquired by touching the texture plate, was the same, since the glass particles' diameters across the texture plate were the same. The width and height of the texture area displayed were adjusted to be the same as those of the real texture plate.

The pointer oscillated, and the position of the pointer was updated using the expression in section II. We conducted preliminary experiment to narrow down the α value where the pseudo-haptics were effective to some extent. According to the preliminary experiment, the user study used three different α values (α = 1, 1.5, 2) to investigate the effect of size of α values. We refer to the three different α values as different "visual conditions". C was configured so that the translation distance was 4.82 pixels (1.5 mm) when the pointer moved at 200 mm/s and α was 1. C was constant for all three α.

**Fig. 3.** (a) Environment of the user study. Top: Overview. Bottom Left: The sensor was attached to the participants' index fingers. Bottom Right: Participants watched their finger movement through pointer on mirrored display. (b) Experimental window. There were left and right side.

We refer to the nine texture plates (from #1 to #9) as different "texture conditions".

#### **3.2 Task Design**

This experiment used a within-participants design. The participants touched a texture plate under the mirror, watching the pointer in the left or right area on display. In One area, the pointer oscillated while in the other area, the pointer did not oscillated.

The assignment of the visual oscillation to the specific area (left or right) was randomly determined. After freely touching the plate, the participants were asked to report which area they felt to be rougher, by pushing the button. Since we wanted to reproduce the same situation as when touching a surface as usual, we did not define the time limit for each trial. We told participants to focus on the haptic roughness of the plates while watching the pointer on the screen and asked them to judge the roughness based on the haptic roughness instead of visual information. We allowed participants only to slide the pen on the surface and prohibited tapping or flicking gesture with pen. Also, participants were told that the texture plates were homogeneous.

There were nine texture conditions (from #1 to #9) and three visual conditions (α = 1, 1.5, 2). For each condition, the participants repeated the trial four times. Thus, each participant conducted 108 trials (= 9×3×4). The texture and visual conditions were presented in the random order, and were counterbalanced across the participants.

#### **3.3 Result and Discussion**

First of all, to determine whether the participants' answers were dissimilar from chance (50%), a chi-square goodness-of-fit test was performed on the frequency of the participants' selections for two texture plates, against the null hypothesis that the two areas were equally selected. Figure 4 shows the number of times and a percentage a selection was made for the areas with visual oscillation for each visual and texture condition. Out of the 40 selections (10 participants multiplied by 4 times) under each condition for the question, the difference was observed to be statistically significant (p < 0.05) for all conditions, except only one condition (α = 1 and texture #1, p = 0.057). The result of chi-square test and the biased selection to the areas with visual oscillation implied that the participants reported that the vibrotactile texture felt rougher compared with the case in which there was no visual oscillation.

**Fig. 4.** Matrix along texture condition axis and visual condition axis on the frequency and percentage of selection for the areas with visual oscillation.

Next, we conducted an analysis using the generalized linear model with the binomial distribution. We compared the full model and each of the three reduced models which had variables of (1) only visual condition or (2) texture condition or (3) visual and texture condition but without interaction effect. Then, we conducted a likelihood ratio test (L.R.T) against the null hypothesis that each reduced model's deviation and the full model's deviation were same. We generated it using the chi-square test which is equivalent to L.R.T when performing logistic regression.

According to the result, the reduced model without texture condition was significantly different from the full model (p < 0.01) and thus, the texture condition significantly affected the response. On the other hand, the reduced model without visual condition or interaction effect was not different from the full model (p > 0.05). Thus, visual condition and interaction effect was not shown to affect the response.

The result of this user study suggests that visual oscillations may change the roughness judgment of real textures. However, under the experimental conditions, the participants may have answered only the presence or absence of the visual oscillations. Therefore, we conducted a user study 2 to evaluate the physical equivalent point from the viewpoint of tactile roughness, and indirectly confirmed the effect of visual oscillations.

# **4 User Study 2**

The second user study investigated what size of visual oscillation is optimal to provide the desired, modified roughness. Six participants, aged from 22 to 25 participated in the experiment. All participants were right-handed.

#### **4.1 Experimental System**

The experimental environments except for texture plates' arrangement and the experimental window were the same as user study 1. The same texture plates from #1 to #9 were used in this user study. The closest pairs of two texture plates in terms of roughness (e.g. [#1, #2]) were presented to participants at the same time. We refer to the eight pairs of texture plates (from [#1, #2] to [#8, #9]) as different "texture pair conditions".

The visual oscillation was disabled for the rougher texture and was enabled for the smoother texture. Participants adjusted the size of visual oscillations α of smoother texture so that they felt the same roughness from both texture plate. The value of C was the same as previous user study.

**Fig. 5.** (a) Participants touch two different rough texture plates. (b) Experimental window. There were left and right side, each corresponding to two plates.

#### **4.2 Task Design**

This experiment used a within-participants design. The participants touched two texture plates under the mirror. One of the two plates was rougher and another was smoother. Though the assignment of the left or right to the plates was randomly determined, which side was the target of adjustment as illustrated in the experimental window. The pointer visually oscillated when the smoother plate was touched, but the pointer did not oscillate when the rougher plate was touched. The participants were asked to adjust the size of the visual oscillations α of the smoother plate so that they felt the same roughness from both the texture plates. Participants pushed a button from "decrease (−0.04)", "slight decrease (−0.02)", "slight increase (+0.02)", or "increase (+0.04)". The initial value of α was set at 0. There was no time limit for adjustment and participants could push buttons repeatedly.

There were eight texture pair conditions. For each condition, the participants repeated the trial twice. The texture pair conditions were presented in the random order.

#### **4.3 Result and Discussion**

Figure 6 shows the results. The horizontal axis shows the smoother texture plate of texture pairs. The vertical axis shows the adjusted size of visual oscillation for the smoother texture of texture pairs.

If the visual oscillation does not affect the roughness, the adjustment value should be zero. To determine whether the adjustment value is different from zero, we conducted t-tests for each texture pair condition. We conducted the normality by the Shapiro-Wilk test for each texture pair condition in advance. The results of the t-test for all shows that the adjusted size of visual oscillation was significantly different from zero (p < 0.05) and adjusted values ranged from 0.8 to 1.5.

**Fig. 6.** The adjusted visual oscillation size for smoother texture plate for equalizing roughness.

Next, we performed a one-way ANOVA with factors of texture pair condition on the adjusted size. We conducted a Mauchly's test to check the sphericity criteria in advance of the ANOVA test. According to the ANOVA results, the texture pair condition did not significantly affect the adjusted size (F(7, 84) = 2.12, p = 0.61).

# **5 Conclusion**

In this study, we applied the pseudo-haptic method using visual oscillations, to modify the subjective roughness judgment while scanning real textured plates with bare fingers. We manufactured various texture plates, with roughness ranging from several tens of micrometers to several millimeters. We conducted user study where users judged the roughness of plate under conditions with and without visual oscillations. The results showed that visual oscillations can modify the perceived roughness of real textured surfaces.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Identifying Tactors Locations on the Proximal Phalanx of the Finger for Navigation**

Justine Saint-Aubert(B)

Paris, France

**Abstract.** Vibrotactile stimulation has been investigated to provide navigational cues through belts, vests, wrist-bands and exotic displays. A more compact solution would be the use of a ring type display. In order to test its feasibility, a user-centered experiment is conducted. The ability of participants to identify cardinal directions and inter cardinal directions by vibrotactile stimulation on the proximal phalanx is investigated. The results indicate that participants achieved 96% accuracy for cardinal directions and 69% accuracy for cardinal plus inter-cardinal directions using a static stimulation. The identification rates of dynamic stimulation are lower than that of static stimulation.

**Keywords:** Vibrotactile displays · Perception · Navigation · Ring device

#### **1 Introduction**

#### **1.1 Navigation Using Tactile Stimulation**

Navigation refers to the process of finding a way from one place to another. It implies asserting one's position in the environment and planning the next direction. In the following, we take an interest in the planning step only.

We usually rely on visual feedback in order to navigate but holding a map or a mobile phone while walking affect the awareness of the pedestrian [13]. Tactile has shown benefits in substituting vision since information can be conveyed to users without altering their perceptions of the surrounding environment [22]. Among tactile cues, vibrations are one of the most convenient. They can be generated using small actuators (tactors), resulting in easy-to-carry devices. Vibratory displays have shown their efficiency for in-vehicle and pedestrian navigation (e.g [3,10]), but are hardly used in everyday life.

Among existing solutions, hand-held devices have been proposed. Mobile phones that integrate one tactor are the most frequently used and handles composed of an array of tactors have also been advanced [4] but they prevent manual interactions and can induce fatigue. In a different strategy, hand-free systems have then been developed. Vibrotactile matrix have been integrated in gloves [21], body trunk systems (e.g [6]), anklets [2], forearm systems [16] and others. All these solutions have shown to support navigational tasks but are unlikely to be used for everyday life. Furthermore, the body trunk and the head are considered as personal areas and users acceptance can be an issue. An alternative can be found in wristbands displays. Prototypes that include one or two tactors [1,14] or watch with 6 tactors [19] have been proposed1. The wrist is however one of the less sensitive part of the body [23] and a more compact solution can be suggested through a ring device.

A ring display would be easy to carry, discreet and versatile since it can be adapted to any finger. The finger is a socially established part so user acceptance should not be an issue. Finally, it implies in part a tactile stimulation on the glabrous skin that is higly sensitive to vibration [18]. The solution then seems promising but the feasibility of such a device has never been investigated.

#### **1.2 Feasibility of a Ring Device**

A ring display would be made of tactors placed all around the finger and the location of the activated tactors interpreted as directional information. In order to be used for navigation, users have then to correctly localize the tactors, which is not guaranteed. The stimulation would be on the proximal phalanx of the finger and at this level the glabrous skin of the palmar surface is next to the less sensitive skin of the dorsal surface, the perception of vibrations could then be altered. The tactors would also be close from each other while localization rates are better with greater inter-disances (e.g [15,23]).

These drawbacks could be compensated by the spatial repartition of the tactors on the finger that makes information map body coordinates. A dynamic strategy could also be exploited. While a single tactor is activated during static stimulation, several ones are sequentially triggered during dynamic stimulation and recognition rates can be greatly improved using this strategy [20]. To test these hypotheses, a psycho-physical experiment is conducted. Four tactors are placed on the proximal phalanx of participants in order to map body coordinates. Their ability to identify directions using static and dynamic stimulation studied. The identification of cardinal directions is investigated (1). The identification of cardinal plus inter-cardinal directions is also examined (2).

#### **2 Psychophysical Experiment**

#### **2.1 Set up**

Four pancake tactors (*GoTronic* VM834, ∅ 8 mm, 3 V, 1200 rpm) are glued at ring level in the middle of the dorsal, volar and side surfaces of the index finger (Fig. 1 (a)). They are activated with a *Arduino Uno* card2 controlled by a C++ program running on a Linux computer. The overall frequency is up to 3.5 kHz.

1 The latter is not specifically dedicated to navigation but it is a potential application.

2 https://store.arduino.cc/arduino-uno-rev3.

Signals sent to tactors are 300 ms pulse at 3 V. The resulting vibration is about 200 Hz in the range of optimal sensitivity of Pacinian corpuscles (200 Hz ↔ 250 Hz) [11]. The tactors are fixed on the non-dominant hand of the participants, leaving the other hand free to provide feedback via a computer located on a table in front of them. The hand with tactors is placed palm down and the index finger raised using foam in order to prevent tactors/table and tactors/skin contact. The complete set up is shown in Fig. 1 (b).

**Fig. 1.** (a) Pictures and close ups of tactors fixed on the proximal phalanx of the index finger. (b) Picture of the complete set up (tactors and computer) with a particpant.

#### **2.2 Method**

A compass and buttons corresponding to directions are displayed on the computer. Vibro-tactile stimulations are transmitted to the participants who have to identify the encoded directions, using a touch-pad to make responses.

During a session, the order of the stimulations is randomly predefined so it is the same between participants. Stimuli are tested in a row and no breaks are allowed. Once a participant gives an answer, no feedback is provided and the next stimulus appears after 3 s. Pink noise is played to them through headphones and they are not blindfolded but are asked to focus on the computer screen.

#### **2.3 Participants**

Eight participants (2 males), ages 27 to 43 (mean age 31 years ± 5.8), volunteer. They are selected because they do not report any visual impairment or physical issues. Seven of them are right-handed according to Coren's handedness test [5].

# **3 Identifications of Cardinal and Inter-cardinal Directions**

#### **3.1 Static and Dynamic Stimulations of Cardinal Directions**

The ability of the participants to identify cardinal directions is examined. In a first session, static stimulation is tested. The tactors and the compass keep the same orientation (Fig. 2 (a)). The activation of the tactor on the dorsal surface of the finger corresponds to a North command, the tactor on the volar surface describes the South, a tactor on the side encodes the East and so on (Fig. 2 (b)).

**Fig. 2.** (a) Relative directions between the index finger from face and the compass. Static (b) and dynamic (c) stimulation of cardinal directions. Sinusoids represent activated tactors and red color apparent locations of vibrations. Numbers and shades of red indicate a sequence of activation. (Color figure online)

In another session, dynamic stimulation is investigated. The tactors and the compass remains in the same positions, however three tactors are activated sequentially. For instance, the North is encoded by the activation of the tactor on the volar surface, the side and finally on the dorsal surface. The same scheme, always clockwise is chosen for the other directions (Fig. 2 (b)).

All the participants performed both sessions using a counterbalanced design in order to control potential effects of the order. During each session, each cardinal direction is tested 30 times for a total of 120 trials.

#### **3.2 Static and Dynamic Stimulations with Inter-cardinal Directions**

The ability of participants to identify inter-cardinal directions is investigated.

**Fig. 3.** Stimulations of inter-cardinal directions. See the caption of Fig. 2 for details.

While the tactors are only located at cardinal locations, inter-cardinal directions are simulated by activating two tactors simultaneously. The vibrations should be perceived in the middle [12]. Static and dynamic stimulations of inter cardinal directions using such dual tactors activations are explained in Fig. 3.

All participants tested static and dynamic stimulation. A counterbalanced design is again employed. During each session, each stimulation is tested 30 times for a total of 240 trials. This part of the experiment is conducted after the sessions testing only cardinal directions.

#### **3.3 Identification Rates**

Participant answers are entered into a response matrix so that correct responses are analysed. The percentages of correct identifications by participants and over participants are shown in Fig. 4.

**Fig. 4.** Correct identifications of cardinal directions during static (a) and dynamic (b) stimumaltions, and of cardinal/inter-cardinal directions during static (c) and dynamic (d) ones. Individual results are exhibited by markers using one color by participant. Bars and errorbars represent the means and standard deviations over participants. "A" stands for All directions, "N" for North, "*SE*" for South-East and so on. (Color figure online)

Participants are able to identify cardinal directions alone with great accuracy during static stimulation. The percentage of identification is closed to the perfect score with 96% (*std* = 5*.*1%) of correctness. However, the score drops by 24% reaching 72% during dynamic stimulation and a higher variability is exposed (*std* = 5*.*1%). To test whether the difference between static and dynamic simulations is significant, a non-parametric paired Wilcoxon signed-rank test (percentages don't follow a normal distribution) is used. The mean percentages correct over all directions are compared. The null hypothesis states that there is no difference between static and dynamic stimulations is rejected (p = 0.008). The alternative hypothesis then supports that the type of stimulations has an effect on correct identifications.

Other results are related to the tests of cardinal directions plus inter-cardinal ones. In static stimulation, the global identification is uncertain with an overall score of 69% and variability (*std* = 11*.*5%). Inter-cardinal directions are not well recognized. In the dynamic case, the identification rate is even lower with only 30% (*std* = 11*.*2%) of correct identifications. The statistical analysis is performed on these data and the null hypothesis is rejected (p = 0.012), again supporting that the type of stimulations has an effect on correct identifications.

#### **4 Discussions**

#### **4.1 Cardinal Identifications**

Following the results of the experiment, the feasibility of a ring finger display for navigation is discussed. Cardinal directions can be transmitted efficiently using a static stimulation as shown by identification rates closed to the perfect score. A ring display could then be employed to navigate in cities that apply the Hippodamian model (or grid plan) and the tactile feedback will not need to be supplied by vision. Identification rates can reach perfect score during more casual simulations as it is the case during everyday navigation or by using a repeated command strategy.

The feasibility of the system should however be examined during a navigation task since conditions are likely to differ. For instance, results have been obtained while the hand orientation remains constant. According to past studies, wrist motions have a little negative effect on direction identification [17] and while they affect a little bit the detection, factors such as intensity can be adjusted to compensate [9]. Mobility should then not be an issue. However as phalanxes have a higher mobility than the wrist, this point should be further explored.

#### **4.2 Inter-cardinal Identifications**

The use of a ring display in places that do not apply Hippodamian model is however challenging since the transmissions of inter-cardinals directions are approximate. Identification rates of cardinals directions are even lower when intercardinal are displayed than when they are not. Participants have then difficulty to differentiate the activation of one and two tactors. Additional tactors placed at inter-cardinal locations instead of dual tactors activation should lead to the same results, or even wors. The higher the number of tactors, the harder the identification [8], especially since they will no longer be on noticeable loci.

The use of a dynamic pattern does not help. Results for dynamic simulations are associated with poorer identification rates than static simulations in both part of the experiment. The simultaneous activation of three tactors was confusing for participants and this issue may results from the short inter-distance between tactors or a wrong choice of pattern. This pattern was chosen as a compromise between long motion and limited activation of tactors. Based on the results, different patterns should be explored. In the same way different type of dynamic stimulation have been tested on other body parts (e.g the back [7]), the same should be done on the phalanx.

An alternative solution could be to exploit temporal pattern. The identifications of cardinal directions only during static stimulation are accurate and correspond to discrete feedback. Cardinal directions displayed with short breaks could then encode inter-cardinal directions. North-East would correspond to the activation of the North, a few milliseconds break, then the activation of the East only. This solution could also answer the variability issue. Indeed dynamic patterns and direct presentations of inter-cardinal directions both increase betweensubject variability. This issue can be raised because of different phalanx dimensions or type of skin that may have a detrimental effect on vibrations propagation. The static stimulation however minimizes the issue.

# **5 Conclusions and Perspectives**

The feasibility of a ring vibro-tactile display for navigation have been investigated in a user-centered experiment. Participants were able to identify accurately cardinal directions transmitted by static stimulation of four tactors at ring level. The transmission of inter-cardinal directions was more challenging and a dynamic stimulation was not beneficial. A research will work towards the test of patterns that could be recognized accurately as inter-cardinal directions.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Tactile Perception of Objects by the User's Palm for the Development of Multi-contact Wearable Tactile Displays**

Miguel Altamirano Cabrera(B), Juan Heredia, and Dzmitry Tsetserukou

Skolkovo Institute of Science and Technology (Skoltech), Bolshoy Boulevard 30, bld. 1, 121205 Moscow, Russia

<EMAIL_ADDRESS>
**Abstract.** The user's palm plays an important role in object detection and manipulation. The design of a robust multi-contact tactile display must consider the sensation and perception of the stimulated area, aiming to deliver the right stimuli at the correct location. To the best of our knowledge, there is no study to obtain the human palm data for this purpose. The objective of this work is to introduce a method to investigate the user's palm sensations during the interaction with objects. An array of fifteen Force Sensitive Resistors (FSRs) was located at the user's palm to get the area of interaction, and the normal force delivered to four different convex surfaces. Experimental results showed the active areas at the palm during the interaction with each of the surfaces at different forces. The obtained results were verified in an experiment for pattern recognition to discriminate the applied force. The patterns were delivered in correlation with the acquired data from the previous experiment. The overall recognition rate equals 84%, which means that user can distinguish four patterns with high confidence. The obtained results can be applied in the development of multi-contact wearable tactile and haptic displays for the palm, and in training a machine-learning algorithm to predict stimuli aiming to achieve a highly immersive experience in Virtual Reality.

**Keywords:** Palm haptics *·* Cutaneous force feedback *·* Tactile force feedback *·* Wearable display

# **1 Introduction**

Virtual Reality (VR) experiences are used by an increasing number of people, through the introduction of devices that are more accessible to the market. Many VR applications have been launched and are becoming parts of our daily life, such as simulators and games. To deliver a highly immersive VR experience, a significant number of senses have to be stimulated simultaneously according to the activity that the users perform in the VR environment.

**Fig. 1.** a) Sensor array at the user's palm to record the tactile perception of objects. Fifteen FSRs were located according to the physiology of the hand, using the points over the joints of the bones, and the location of the pollicis and digiti minimi muscles. b) Experimental Setup. Each object was placed on the top of the FT300 force sensor to measure the applied normal force. The data from the fifteen FSRs and the force sensor are visualized in real-time in a GUI and recorded for future analysis.

The tactile information from haptic interfaces improves the user's perception of virtual objects. Haptic devices introduced in [1–5], provide haptic feedback at the fingertips and increase the immersion experience in VR.

Many operations with the hands involve more that one contact point between the user's fingers, palm, and the object, e.g., grasping, detection, and manipulation of objects. To improve the immersion experience and to keep the natural interaction, the use of multi-contact interactive points has to be implemented [6,7]. Choi et al. [8,9] introduced devices that deliver the sensation of weight and grasping of objects in VR successfully using multi-contact stimulation. Nevertheless, the proposed haptic display provides stimuli only at the fingers and not on the palm.

The palm of the users plays an essential role in the manipulation and detection of objects. The force provided by the objects to the user's palms determines the contact, weight, shape, and orientation of the object. At the same time, the displacement of the objects on the palm can be perceived by the slippage produced by the forces in different directions.

A significant amount of Rapidly Adapting (RA) tactile receptors are present on the glabrous skin of the hand, a total of 17, 023 units on average [10]. The density of the receptors located in the fingertips is 141 units/cm 2 , which is bigger than the density in the palm 25 units/cm 2 . However, the overall receptor number on the palm is compensated by its large area, having the 30% of all the RA receptors located at the hand glabrous skin. To arrive at this quantity, we should cover the surface of the five fingertips. For this reason, it is imperative to take advantage of the palm and to develop devices to stimulate the most significant area with multi-contact points and multi-modal stimuli. Son et al. [11] introduced a haptic device that provides haptic feedback to the thumb, the middle finger, the index finger, and on the palm. This multi-contact device provides kinesthetic (to the fingers) and tactile stimuli (at the palm) to improve the haptic perception of a large virtual object.

The real object perception using haptic devices depends on the mechanical configuration of the devices, the contact point location on the user's hands, and the correctness of the information delivered. To deliver correct information by the system introduced by Pacchierotti et al. [12], their haptic display was calibrated using a BioTac device.

There are some studies on affective haptics engaging the palm area. In [13], the pressure distribution on the human back generated by the palms of the partner during hugging was analyzed. Toet et al. [14] studied the palm area that has to be stimulated during the physical experience of holding hands. They presented the areas of the hands that are stimulated during the hand holding in diverse situations. However, the patterns extraction method is not defined, and the results are only for parent-child hand-holding conditions.

Son et al. [15] presented a set of patterns that represent the interaction of the human hand with five objects. Nevertheless, the investigated contact point distribution is limited to the points available in their device, and the different sizes of the human hands and the force in the interaction are not considered.

In the present work, we study the tactile engagement of the user's palms when they are interacting with large surfaces. The objective of this work is to introduce a method to investigate the sensation on the user's palm during the interaction with objects, finding the area of interaction, and the normal force delivered. This information is used to the reproduction of tactile interaction, and the development of multi-contact wearable tactile displays. The relation between the applied normal force and the location of the contact area should be found, considering the deformation of the palm interacting with different surfaces.

#### **2 Data Acquisition Experiment**

A sensor array was developed to investigate the palm area that is engaged during the interaction with the different surfaces. Fifteen FSRs were located according to the physiology of the hand, using the points over the bones' joints, and the location of the pollicis and digiti minimi muscles. In Fig. 1 the distribution of the fifteen FSRs is shown.

The fifteen FSRs are held by a transparent adhesive contact paper that is attached to the user's skin. The transparent adhesive paper is flexible enough to allow users to open and close their hands freely. The different shape of the user's hands is considered every time the array is used by a different user. The method to attach the FSRs on the user's palms is the following: a square of transparent adhesive paper is attached to the skin of the users, the points, where the FSR must be placed, are indicated with a permanent marker. After that, the transparent adhesive paper is taken off from the hand to locate the fifteen FSRs at the marked points. The FSRs are connected to ESP8266 microcontroller through a multiplexer. The microcontroller receives the data from the FSRs and sends it to the computer by serial communication.

The hand deformation caused by the applied force to the objects is measured in this study. The force applied to the objects is detected by a Robotiq 6 DOF force/torque sensor FT300. This sensor was chosen because of its frequency of 100 Hz for data output and the low noise signal of 0.1 N in F*z*, which allowed getting enough data for the purposes of this study. The sensor was fixed to a massive and stiff table (Siegmund Professional S4 welding table) using an acrylic base. An object holder was designed to mount the objects on top of the force sensor FT300.

#### **2.1 Experimental Procedure**

Four surfaces with different diameters were selected, three of them are balls, and one is a flat surface. The different diameters are used to focus on the position of the hand: if the diameter of the surface increases, the palm is more open. The diameters of the balls are 65 mm, 130 mm, and 240 mm.

**Participants:** Ten participants volunteering completed the tests, four women and six men, aged from 21 to 30 years. None of them reported any deficiencies in sensorimotor function, and all of them were right-handed. The participants signed informed consent forms.

**Experimental Setup:** Each object was placed on the top of the FT300 force sensor to measure the applied normal force. The data from the fifteen FSRs and the force sensor are visualized in real-time in a graphical user interface (GUI) and recorded for future analysis.

**Method:** We have measured the size of the participant's hands to create a sensor array according to custom hand size. The participants were asked to wear the sensors array on the palm and to interact with the objects. Subjects were asked to press objects in the normal direction of the sensor five times, increasing the force gradually up to the biggest force they can provide. After five repetitions, the object was changed, and the force sensor was re-calibrated.

#### **2.2 Experimental Results**

The force from each of the fifteen FSRs and from the force sensor FT300 was recorded at a rate of 15 Hz. To show the results, the data from the FSRs were analyzed when the normal force was 10 N, 20 N, 30 N, and 40 N. In every force and surface, the average values of each FSR were calculated. The average sensor values corresponding to the normal force for each surface are presented in Fig. 2.

From Fig. 2 we can derive that the number of contact points is proportional to the applied force. The maximum number of sensors is activated when 40 N is

**Fig. 2.** Number and location of contact points engaged during the palm-object interaction. The data from the FSRs were analyzed according to the normal force in each surface. The average values of each FSR were calculated. The active points are represented by a scale from white to red. The maximum recorded value in each surface was used as normalization factor, thus, the maximum recorded value represents 100%. Rows A, B, C, D represent results for a small-size ball, a medium-size ball, a large-size ball, a flat surface, respectively. (Color figure online)

applied to the medium-sized ball, and the minimum number is in the case when 10 N is applied to large-size and flat surface.

The surface dimension is playing an important role in the activation of FSRs. It can be observed that the surfaces that join up better to the position of the hand activate more points. The shape of the palm is related to the applied normal force and to the object surface. When a normal for of 10 N is applied to a smallsize ball, the number of contact points is almost twice the contact points of other surfaces, it increases only to seven at 40 N. The number of contact points with the large-size ball at 10 N is two, and increases to eight at 40 N. Instead, the flat surface does not activate the points at the center of the palm.

#### **3 User's Perception Experiment**

To verify if the data obtained in the last experiment can be used to render haptic feedback in other applications, an array of fifteen vibromotors was developed. Each of the vibromotors was located in the same positions as the FRS, as shown in Fig. 1. The vibration intensity was delivered in correlation with the acquired data from the Sect. 2 to discriminate the applied force from the hand to the large-size ball (object C).

#### **3.1 Experimental Procedure**

The results from the big ball were selected for the design of 4 patterns (Fig. 3). The patterns simulate the palm-ball interaction from 0.0 N to 10 N, 20 N, 30 N and 40 N, respectively. The pattern was delivered in 3 steps during 0.5 s each one. For instance in the 0.0 N - 10 N interaction, firstly the FRS values at 3 N were mapped to an vibration intensity and delivered to each vibromotor during 0.5 s, then similar process was done for the FRS values at 6 N and 10 N.

**Fig. 3.** Tactile presentation of the patterns with the array of vibration motors a) the arrangement of fifteen vibromotors is shown. b) P1, P2, P3, and P4 represent the actuated vibromotors on the palm and the intencity of vibration of each actuator (by red gradient color). (Color figure online)

**Participants:** Seven participants volunteering completed the tests, two women and five men, aged from 23 to 30 years. None of them reported any deficiencies in sensorimotor function, and all of them were right-handed. The participants signed informed consent forms.

**Experimental Setup:** The user was asked to sit in front of a desk and to wear the array of fifteen vibromotors on the right palm. One application was developed in Python where the four patterns were delivered and the answers of the users were recorded for the future analysis.

**Method:** We have measured the size of the participant's hands to create a vibromotors array according to the custom hand size. Before the experiment, a training session was conducted where each of the patterns was delivered three times. Each pattern was delivered on their palm five times in random order. After the delivery of each pattern, subject was asked to specify the number that corresponds to the delivered pattern. A table with the patterns and the corresponding numbers was provided for the experiment.

#### **3.2 Experimental Results**

The results of the patterns experiment are summarized in a confusion matrix (see Table 1).

**Table 1.** Confusion matrix for the pattern recognition.

The perception of the patterns was analyzed using one-factor ANOVA without replication with a chosen significance level of α < 0.05. The p *−* value obtained in the ANOVA is equal to 0.0433, in addition, the F critic value is equal to 3.0088, and the F value is of 3.1538. With these results, we can confirm that statistic significance difference exists between the recognized patterns. The paired t-tests showed statistically significant differences between the pattern 1 and pattern 2 (p = 0.0488 < 0.05), and between pattern 1 and pattern 3 (p = 0.0082 < 0.05). The overall recognition rate is 84%, which means that the user can distinguish four patterns with high confidence.

#### **4 Conclusions and Future Work**

The sensations of the users were analyzed to design robust multi-contact tactile displays. An array of fifteen Interlink Electronics FSRTM 400 Force Sensitive Resistors (FSRs) was developed for the user's palm, and a force sensor FT300 was used to detect the normal force applied by the users to the surfaces. Using the designed FSRs array, the interaction between four convex surfaces with different diameters and the hand was analyzed in a discrete range of normal forces. It was observed that the hand undergoes a deformation by the normal force. The experimental results revealed the active areas at the palm during the interaction with each of the surfaces at different forces. This information leads to the optimal contact point location in the design of a multi-contact wearable tactile and haptic display to achieve a highly immersive experience in VR. The experiment on the tactile pattern detection reveled a high recognition rate of 84.29%.

In the future, we are planing to run a new human subject study to validate the results of this work by a psychophysics experiment. Moreover, we will increase the collected data sensing new objects. With the new dataset and our device, we can design an algorithm capable to predict the contact points of an unknown object. The obtained results can be applied in the development of multi-contact wearable tactile and haptic displays for the palm.

The result of the present work can be implemented to telexistence technology. The array of FSR senses objects and the interactive points are rendered by a haptic display to a second user. The same approach can be used for affective haptics.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# From Hate to Love: How Learning Can Change Affective Responses to Touched Materials

Müge Cavdan1(&) , Alexander Freund1 , Anna-Klara Trieschmann1 , Katja Doerschner1,2, and Knut Drewing1

> 1 Justus Liebig University, 3539 Giessen, Germany<EMAIL_ADDRESS>2 Bilkent University, 06800 Ankara, Turkey

Abstract. People display systematic affective reactions to specific properties of touched materials. For example, granular materials such as fine sand feel pleasant, while rough materials feel unpleasant. We wondered how far such relationships between sensory material properties and affective responses can be changed by learning. Manipulations in the present experiment aimed at unlearning the previously observed negative relationship between roughness and valence and the positive one between granularity and valence. In the learning phase, participants haptically explored materials that are either very rough or very fine-grained while they simultaneously watched positive or negative stimuli, respectively, from the International Affective Picture System (IAPS). A control group did not interact with granular or rough materials during the learning phase. In the experimental phase, participants rated a representative diverse set of 28 materials according to twelve affective adjectives. We found a significantly weaker relationship between granularity and valence in the experimental group compared to the control group, whereas roughness-valence correlations did not differ between groups. That is, the valence of granular materials was unlearned (i.e., to modify the existing valence of granular materials) but not that of rough materials. These points to differences in the strength of perceptuo-affective relations, which we discuss in terms of hard-wired versus learned connections.

Keywords: Haptics Valence Affect Roughness Granularity Learning

# 1 Introduction

We constantly interact with various materials like plastic, fabric, or metal. Haptic perceptual properties of materials have been summarized by five different dimensions [1] that are softness (but cf. [2]), warmness, micro- and macro roughness, friction, and stickiness. In addition to the sensory properties that we experience while haptically exploring a material, we often also have an initial affective reaction to it. Moreover, the

Research was supported by the EU Marie Curie Initial Training Network "DyVito" (H2020-ITN, Grant Agreement: 765121) and Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number 222641018 – SFB/TRR 135, A5.

affective reactions that a material elicits might also influence the duration of our haptic interactions. For example, soft and smooth materials, which cause more pleasant feelings than rough and sticky materials [3–6], might be explored longer.

Previous research on the semantic structure of affective experiences postulates three basic affective dimensions [8]: valence, arousal, and dominance, where each dimension has two opposite poles [9]: arousal ranges from a very calm state and sleepiness (low) to vigilance, which is accompanied by excitement (high). Valence is a continuum from negative to positive and dominance ranges from dominant to submissive. Most of the research in haptic perception has focused on the connection between pleasantness (which can be equated with valence) and the perception of sensory dimensions. One key finding has been that smooth and soft materials are related to more pleasant feelings than rough materials [10], and that the rougher a material is rated, the more unpleasant it feels [10]. In a more recent study [11], all three basic affective dimensions and their relationship with materials' sensory characteristics have been systematically investigated: Drewing et al. [11] used a free exploration paradigm to study the sensory and affective spaces in haptics and tested the generalizability of their results to different participant groups. They found that arousal was related to the amount of perceived fluidity, that higher dominance as associated with increases in perceived heaviness and decreases in deformability, and that greater positive valence was associated with increased granularity and decreased roughness.

It is currently unknown to what extent such perceptuo-affective connections are due to learning experiences and to what extent they are hard-wired, innate mechanisms. Here we investigated directly whether existing relationships between sensory material properties and affective responses can be unlearned, and whether the extent of unlearning depends on the specific perceptuo-sensory relation, for two haptic perceptual dimensions: granularity and roughness. We speculate that hard-wired connections should be more resistant to unlearning than learned ones.

We ran a classical conditioning study that constisted of two phases: learning and experimental phases with two groups each (experimental and control). In the learning phase, participants haptically explored selected materials while watching affective images: In the experimental group rough materials were combined with positive images, granular materials with negative images and distractor materials with neutral images. In the control group participants learned instead associations of fibrous and fluid materials with arousal, which were however not subject of this paper and will not be further discussed. In the experimental phase, participants rated a representative set of 28 materials for 12 affective adjectives. We calculated perceptuo-affective correlations for valence-roughness and valence-granularity relationships per group and compared these correlations between groups. Lower correlations for the participants in the experimental group would indicate an unlearning of the relationship between valence and the respective perceptual dimension.

# 2 Methods

#### 2.1 Participants

Sixty-six students (9 males; age 18–34 years, mean: 23.5 years) from Giessen University participated in our study. Four were excluded from analysis due to misunderstanding the task, technical error, or an increased threshold in the two-point touch discrimination (>3 mm at index finger). All participants were naïve to the aim of the experiment, spoke German at native-speaker level, and none reported relevant sensory, or motor impairments. All procedures were in accordance with the Declaration of Helsinki (2008), and participants provided written informed consent prior to the study.

#### 2.2 Setup, Material, and Adjectives

Fig. 1. Experimental setup from the experimenter's viewpoint.

 Materials were presented in the box (see Fig. 1). Participants reached the materials through the hand opening, which was covered with linen to hinder participants to look inside the box. On a monitor (viewing distance about 60 cm) we presented images (visual angle 14.2°) and adjectives to the participant. Earplugs and active noise cancelling headphones (Beyerdynamic DT770 PRO, 30 O) blocked the noises that can occur from exploring the materials. All materials were presented in 16 16 cm plastic containers embedded in the bottom of the box. A light sensor in the box signaled when the participant's hand was on the front edge of the container, allowing to start picture presentation simultaneously to haptic exploration. Participants gave responses using a keyboard. The experimenter sat on the other side of the table in order to exchange materials guided by information presented on another monitor.

For the learning part of the experimental group, we selected materials from [11] which had a high factor value on one of the target sensory dimensions (either granularity or roughness) but did not show high factor values in any of the other dimensions (fluidity, fibrousness, heaviness, deformability). Bark and sandpaper were selected for roughness, and salt and lentils for granularity. In the control group other materials were used (jute, wadding, water, shaving foam). Additionally, for both groups (experimental and control) we added four distractor materials, which did not have high factor values on the manipulated dimensions: cork, chalk, paper, and polystyrene. We also the sensory and affective adjectives were obtained from [11]; one to two representative adjectives per sensory dimension (rough, granular, moist, fluffy, heavy and light, deformable and hard). In the learning phase, sensory ratings served to draw the participant's attention to the materials.

For establishing affective-sensory associations we used images from the International Affective Picture Systems (IAPS). The IAPS database includes 1196 colorful images of various semantic contents, that have been rated according to valence, and arousal [12, 13]. For the experimental group, we selected sixteen images with high negative valence (<2.5) and sixteen images with high positive valence (>7.5), and as diverse content as possible (excluding drastic injury images). For the control group, we used images with high or low arousal instead. We also selected 32 distractor images, which have average valence and arousal values (between 4.5 and 5.5).

In the experimental phase, participants rated 28 materials (plastic, wrapping foil, aluminum foil, fur, pebbles, playdough, silicon, paper, styrofoam, paper, sandpaper, velvet, jute, silicon, stone, bark, flour, metal, cork, polish stone, oil, shaving foam, soil, hay, chalk, salt, lentil). We assessed affective responses via adjectives and selected four high loading adjectives per affective dimension: valence (pleasant, relaxing, enjoyable, and pleasurable), arousal (exciting, boring, arousing, and attractive) and dominance (dominant, powerful, weak, and enormous/tremendous).

#### 2.3 Design and Procedure

Participants were randomly assigned to either the experimental or the control group. In the learning phase of the experimental group, we coupled the exploration of the two very rough materials with positive images and the granular materials with negative images, in order to manipulate valence. In the control group, different materials were explored and coupled with different images.

The learning phase consisted of 64 trials: in the experimental group, each of the two granular materials was presented eight times coupled with one of the 16 negative images, and each of the two rough materials was coupled 8 times with one of the positive images. Also, each of the four distractor materials (cork, chalk, polystyrene, and paper) was presented eight times with a distractor image. Both, the assignment of corresponding images to materials and the order of presentation, were random.