We propose a method for forming a focus and providing a tactile sensation farther from an AUTD without enlarging the phased array or placing the devices near the focal point. There are several approaches which provide comparable workspace extension without enlarging the phased array [1,5]. In the proposed method, ultrasound waves emitted from an AUTD are reflected and focused using a concave reflector. A technique of reflecting ultrasound waves in order to focus them, has been applied in other studies [7,8]. The technique has also been applied in a previous study in haptics [11], through which the phased array can present a tactile stimulus at an arbitrary position within the workspace, but a focus-formable distance has not been extended owing to the planar reflections. In the proposed method, it is possible to generate a focus farther from the AUTD and provide a tactile sensation while maintaining the ability to steer the focus electrically, by applying a mirror formula [3]. A mirror formula has been applied in another study so as to reconstruct the geometry of arbitrary reflectors using ultrasound phased arrays [12].

The theory supporting the proposed method and its advantages are described in Sect. 2. In Sect. 3, two experiments conducted in this area are described. First, we investigated whether a focus is formed when applying the proposed method and if a sufficient sound pressure that can provide a tactile sensation is obtained. Next, we experimentally confirmed the range of the workspace by focusing at certain points.

#### **2 Proposed Method**

#### **2.1 Mirror Formula**

The procedure to create a focal point at an arbitrary position using a concave reflector is illustrated in Fig. 1. To simplify the following equation, let us define the z-axis as shown in Fig. 1, where f is the focal length of the concave reflector. We assumed a paraxial approximation system and ignored any aberrations. When an acoustic source is placed at A = (a*x*, a*y*, a*z*), the reflected wave is concentrated and forms a focus at B = (b*x*, b*y*, b*z*) using the following mirror formula,

$$\frac{1}{a\_z} + \frac{1}{b\_z} = \frac{1}{f},\tag{1}$$

as well as the magnification formulae,

$$b\_x = -\frac{b\_z}{a\_z} a\_x,\tag{2}$$

$$b\_y = -\frac{b\_z}{a\_z} a\_y. \tag{3}$$

Therefore, a focus can be created at (b*x*, b*y*, b*z*) by driving the AUTD to reproduce the sound wave emitted from the image sound source placed at (a*x*, a*y*, a*z*), as derived from Eqs. (1)–(3).

**Fig. 1.** The process to form a focal point using a concave reflector. AUTD reproduces the propagating waves as if they were emitted from the image sound source, which are reflected by the reflector and concentrated at the focal point.

#### **2.2 Contributions by Concave Reflector**

The focus can be formed farther away from the phased array because the angle of coverage can be changed arbitrarily through an appropriate design of the concave reflector. In the case where focal points are formed by direct incident waves or reflected waves from a planar reflector, the angle of coverage is uniquely determined for each focal position, and becomes narrower as the distance from the AUTD increases. As a result, the focal diameter lengthens and a tactile sensation cannot be obtained; in addition, the phased array has to be enlarged to increase the angle of coverage. By contrast, concave reflectors can have various focal lengths. If we appropriately design the focal length and use the mirror formula, the angle of coverage can be increased even at a position far from the AUTD. Therefore, the diameter of the focal point can be smaller, which produces a tactile stimulus, without enlarging the phased array.

# **3 Experiment**

#### **3.1 Implementation**

Figure 2 shows the experiment setup. An AUTD is composed of 249 transducers. A T4010A1 (developed by Nippon Ceramic Co., Ltd.) was employed as the transducer. The T4010A1 emits a 40-kHz ultrasound at 121.5 dB in sound pressure level (SPL) at a distance of 30 cm. Each ultrasonic transducer was driven by the full power of the AUTD. The phase shift of each transducer was calculated and driven using the wavelength λ = 8.5 mm at a sound speed of c = 340 m. The concave mirror is a symmetrical parabolic dish shape, the diameter of which is 400 mm and the focal length is 180 mm. The distance between the AUTD and the reflector is 669 mm. A standard microphone (Br¨uel & Kjær 4138-A-015) was moved using a 1-axis motorized stage. The sound pressure was estimated by calculating the absolute value of the 40 kHz component of DFT.

**Fig. 2.** Geometry of the proposed system. The position (0*,* 0*,* 180 mm) is the focus of the reflector.

#### **3.2 Experiment 1: Validation of Mirror Formula**

In Experiment 1, the sound pressure distribution around the focus was measured along the x-axis at the depth of designed focal position, and it was confirmed whether the focal point was formed by the mirror formula. The focal points were formed at (0, 0, 180 mm), (20, 0, 180 mm), and (20, 0, 200 mm), where the distances from the AUTD were 489 and 469 mm. As a reference experiment, the sound pressure distribution was also measured, where a focal point was directly formed at (0, 0, 200 mm) without using a reflector. The microphone was moved within the range of *−*100 mm <= x <= 100 mm, where it was fixed vertically upward when using the reflector and vertically downward when not using it, in order to steer the microphone toward the direction of sound wave arrival.

The results of Experiment 1 are shown in Fig. 3. As shown in this figure, it can be confirmed that each designed focus can be presented by the concave mirror. The maximum sound pressure at each focal position with the reflector is 6.34 *×* 103, 5.43 *×* 103, and 5.69 *×* 103 Pa, or 170.0, 168.7, and 169.1 dB SPL, respectively, which is sufficient to provide a tactile stimulus [2].

By contrast, the maximum sound pressure at (0, 0, 200 mm), without the reflector is 1.49 *×* 103 Pa, or 157.4 dB SPL. Furthermore, the focal diameter without the concave reflector is more than twice that with the reflector. Figure 3 shows that proposed method achieves a higher focal sound pressure and a smaller focal diameter than the conventional method, and that the proposed method can present tactile stimuli.

**Fig. 3.** Measured acoustic pressure distribution in the x-axis direction around each focal point. The position of the focus is at (*x, y, z*) = (0*,* 0*,* 180 mm), (20*,* 0*,* 180 mm), and (20*,* 0*,* 200 mm) using the reflector, and at (0*,* 0*,* 200 mm) without the reflector. (b) shows an enlarged view of (a).

#### **3.3 Experiment 2: Range of the Workspace**

The sound pressure at the focal point when the focus was moved was measured in order to confirm the range of the workspace. The focal position was set at y = 0 mm and z = 160, 180, and 200 mm, where the distance from the AUTD was 509, 489, and 469 mm, respectively, and moved in the x-axis direction. The microphone was fixed vertically upward and was moved within the range of *−*100 mm <= x <= 100 mm.

The results of Experiment 2 is shown in Fig. 4. As indicated in this figure, the focal sound pressure under each condition is sufficiently high around x = 0 mm, which enables to present a tactile stimulus, and decreases as the focus moves away from the z-axis.

**Fig. 4.** Measured focal acoustic pressure at each position when the focal point was moved in the x-axis direction. The y coordinate of the focal point is 0 mm.

#### **4 Discussion**

As the authors' comment, we were able to perceive a tactile stimulus at the focal points when entering the hands between the reflector and the AUTD. The hand is an obstacle shielding the incident wave, but the focus remains because the aperture of the reflector is sufficiently larger than the obstacle.

In Experiment 1, when the focal point was formed at (20, 0, 180 mm), the position where the sound pressure became the highest in the x-axis direction was x = 18 mm, which was shifted from the set position of x = 20 mm. This is thought to be because of aberrations and displacement of the device. In this study, we did not consider the difference in the acoustic path length, which causes a decrease in the focal sound pressure. Therefore, the focal sound pressure can be increased if we optimize the phase shift of each transducer.

The results of Experiment 1 showed that a focal point can be formed at an arbitrary point through the proposed method. Figure 5 shows an example

**Fig. 5.** Application example of the proposed method.

application of this system, in which an image and a tactile sensation are simultaneously presented in a large dome screen by a visual and tactile projector.

Although the proposed method theoretically forms a focal point at an arbitrary position, there are cases where a focal point cannot be formed depending on the position of the phased array and the concave reflector, as well as the aperture of the reflector. For example, in Experiment 2, when the focal point was set to (0, 0, 240 mm), the position of the image sound source formed by the AUTD was (0, 0, 720 mm) according to the mirror formula, which is only 51 mm below the AUTD. However, most of the emitted sound waves are not reflected in this case because when simply considering the geometric acoustic model, a concave reflector of approximately 2.57 *×* 103 mm in diameter is required if the position and focal length of the reflector are fixed. In fact, the focal sound pressure at (0, 0, 240 mm) was 1.20 *×* 103 Pa. Moreover, when the irradiation direction of the incident wave is sufficiently shifted from the direction toward the concave mirror, a focal point can be formed at an unexpected position. This is because the grating lobe generated focuses at unintended positions accompanying with the main focus. In fact, in Experiment 2, when the focal point was set at a position sufficiently far from the z-axis, the focus formed by the grating lobe was confirmed. In addition, in this paper we did not consider the attenuation during propagation, by which the amplitude of sound waves exponentially decreases with distance. Therefore, if the propagation distance is too long, the focal sound pressure can be attenuated significantly [2,4].

#### **5 Conclusion**

We proposed a method for focusing on an arbitrary point farther from the AUTD using a concave mirror. Our experiments showed that the proposed method can form a focus whose diameter is approximately the wavelength of ultrasound at a point 49 cm away from the phased array with an 18 *×* 14 cm2 aperture and present a tactile stimulus. The focus was electrically steerable within several centimeters in the lateral direction.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Movement-Free Virtual Reality Interface Using Kinesthetic Illusion Induced by Tendon Vibration**

Satoshi Tanaka(B) , Keigo Ushiyama, Akifumi Takahashi, and Hiroyuki Kajimoto

The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo, Japan<EMAIL_ADDRESS>
**Abstract.** In current virtual reality (VR) systems, the physical movement of the body is required, which creates problems of safety, cost, and accessibility. To solve those problems, we propose a system that fixes a user's body, detects force when a user tries to move, and generates the sensation of movement using kinesthetic illusion caused by tendon vibration. We implemented a system limited to simple motion, and conducted an experiment to evaluate operability, body ownership, and agency. Although we could not statistically verify the effect of kinesthetic illusion, the results suggested that it may be possible that kinesthetic illusion could increase ownership and decrease agency.

**Keywords:** Kinesthesia · Tendon vibration · Virtual reality

#### **1 Introduction**

An interface for manipulating avatars is an essential component of virtual reality (VR). In science fiction works, VR is often described as a system in which users can subjectively move freely while their physical body is lying on a bed. However, present VR systems reflect the movement of the physical body to the avatar using devices such as position tracking controllers.

In VR, the movement of the physical body causes various problems. For example, the risk of collision and injury; the requirement for a large space and equipment; and the problem that people with limited mobility cannot use VR. The development of a VR system that does not require physical movement would lead to the solution of such problems of safety, cost, and accessibility.

A brain-computer interface (BCI) can be used to implement a movement-free VR system. However, BCI has technical difficulties, such as the requirement for expensive devices, and invasive means (e.g., anesthesia) are required to block motor commands.

**Electronic supplementary material** The online version of this chapter (https:// doi.org/10.1007/978-3-030-58147-3 35) contains supplementary material, which is available to authorized users.

By contrast, a simpler method may exist. Even if the user's body is physically fixed, by altering user's kinesthesia (sense of body movement), the user may be able to feel moving as if not fixed. Fortunately, our kinesthesia can be modulated relatively easily. For example, the sense of motion can be generated even if the body is not actually moving if vibration stimulus is applied to the tendon [6]. This phenomenon is called kinesthetic illusion.

In this study, we propose a system that detects the force exerted by a user's body fixed on a rigid frame to control an avatar, and presents kinesthetic illusion using tendon vibration. In this paper, We implement a system limited to one degree-of-freedom motion as a proof of concept, and evaluate its operability and effects on body ownership and agency.

#### **2 Related Work**

When vibration is applied to the tendon of a muscle, kinesthetic illusion, in which the vibrated muscle is stretched, is generated [6]. This phenomenon is considered to be caused by the activation of the muscle spindle by vibration [6], and it has been reported that the nerve fires at a frequency that agrees with the vibration frequency up to a certain vibration frequency [4,15].

Research has also been conducted on the presentation of complex kinesthetic illusions using this phenomenon. Albert et al. [1] converted recorded nerve firing patterns to vibration and presented kinesthetic illusions that reproduced the movement in the recording. Additionally, Thyrion et al. [16] successfully presented three-dimensional kinesthetic illusion by predicting nerve firing the movement trajectory and converting it into vibration.

This phenomenon has also been applied to VR and human-computer interaction. For example, Hagimori et al. [7] combined tendon vibration with a visual stimulus using a head-mounted display (HMD) to make small physical movements perceived as large movements in VR. Barsotti et al. [3] proposed a system that combined kinesthetic feedback using tendon vibration with a BCI based on motor imagery.

Among the above techniques, in the BCI-based approach, the user does not need to move physically at all. However, in those BCI systems, the user has to perform motor imagery, rather than trying to move physically. This may cause a sense of unnaturalness in VR applications.

Other methods exist to modulate kinesthesia in addition to tendon vibration. Okabe et al. [13] reported that the illusion of finger movement was generated by presenting the flow field of tactile sensation according to the shearing force of the fingertip. Similarly, Heo et al. [8] generated the illusion of bending an object using vibrotactile stimuli according to the force applied to the object.

The modulation of a kinesthetic sensation can also result from visual stimuli alone. Pseudo-haptic feedback proposed by Lecuyer et al. [10] is a technique to present a sense of force only using visual stimuli. However, it was reported that a force-sensing stationary device, similar to our study, was perceived like moving in their experiment [10].

As a method similar to our study, Mochizuki et al. [12] proposed a VR interface that does not require physical movement by measuring the torque exerted by joints of the physically fixed user. However, their current system used visual stimulus only, and techniques that directly alter kinesthesia were not tested.

# **3 System**

#### **3.1 Operating Principle**

The operation of the proposed system is shown in Fig. 1. In this system, the user's body (arms, legs, etc.) is fixed to a rigid frame. When the user tries to move the body, a force sensor attached at a fixing position detects the force exerted by the user (Fig. 1a). In response to the detected force, the computer simulates the user's intended motion and renders a virtual arm on the HMD (Fig. 1b). Simultaneously, a vibration stimulus is applied to the user's tendon in response to the simulated motion to present kinesthetic illusion (Fig. 1c). Thus, vision and kinesthesia are presented as if the user is moving the body, despite the user's body being fixed.

**Fig. 1.** Operation of the system (a) user tries to move and exerts a force. (b) Computer simulates movement and controls the HMD and vibrators. (c) Vibration induces illusory movement.

#### **3.2 Implementation**

As a proof of concept of the proposed method, we implemented a system limited to one degree-of-freedom motion. Although this technique may be applicable to the motion of various joints, we chose the extension and flexion of the forearm following the experiment of Roll et al. [14].

**System Configuration.** The system detects force using a load cell attached to the wrist fixing component, and connected to the PC through the front-end circuit. The game engine Unity (Unity Technologies) was used for the simulation and rendering, and visual stimuli were presented using an HMD Vive Pro (HTC Corporation) via an external graphics processing unit. The vibration waveform was generated using a waveform generator circuit, amplified using an audio amplifier (MUSE M50), and presented using two vibrators (Vp210, Acouve Laboratory). The vibrators were attached near the right elbow using an elastic fabric supporter, to vibrate distal tendons of the biceps brachii (BB) and triceps brachii (TB) muscles. To make the initial position of the virtual arm coincide with the physical arm, a position tracking device (Vive Tracker, HTC Corporation) was attached to the frame. Additionally, to enhance the ownership of the arm, the actual movement of the user's fingers was captured and reproduced in the display using Leap Motion (Ultraleap).

**Algorithms.** The angular velocity of the virtual elbow joint was proportional to the force applied to the load cell; that is, when the force applied to the load cell was F [N] (assuming F = 0 at system startup), the commanded value of the angular velocity was ωcommand = 50F [deg/s]. The angle of the elbow joint θ [deg] was obtained by integrating the angular velocity, but was limited to −45 to 45◦. For the angle and angular velocity, the extension direction was positive. Vibration waveforms were generated by frequency modulating a sine wave between 0 and 100 Hz. Although the amplitude changed according to the frequency because of the frequency response of the system, and kinesthetic illusion is likely to be diminished in lower frequencies, a simple linear mapping was used for the sake of simplicity. This algorithm is based on the knowledge that nerve firing corresponds to the vibration frequency [4,15] and its applications [1,16]. The vibration frequencies fBB [Hz] (for BB) and fTB [Hz] (for TB) were

$$f\_{\rm BB} = \begin{cases} 0 & (4\omega < 0) \\ 4\omega & (0 \le 4\omega < 100) \\ 100 \text{ (otherwise)} \end{cases} \tag{1}$$

$$f\_{\rm TB} = \begin{cases} 0 & (-4\omega < 0) \\ -4\omega \left( 0 \le -4\omega < 100 \right) \\ 100 & \text{(otherwise)} \end{cases} \tag{2}$$

where ω [deg/s] was the angular velocity of the virtual elbow joint (ω = 0 when the angle θ reaches positive or negative limit). However, the coefficients used in the above algorithms were set empirically and not determined theoretically.

#### **4 Experiment**

To evaluate the operability, body ownership, and agency of the avatar using the proposed system, we conducted an experiment in which the participants used the system under the following three conditions.

**Tendon** All elements of the proposed method were incorporated.

**None** Only visual stimulus was used, without vibration stimulus.

**Tactile** Only cutaneous cues were presented using high-frequency vibration that was unlikely to cause kinesthetic illusion. Based on the method of Bark et al. [2], an amplitude-modulated sine wave of 250 Hz was used for the vibration waveform, and the amplitudes of the waveform input to the BB and TB vibrators were ABB = 0.1 × 10θ/45 and ATB = 0.1 × 10−θ/45 (from 0 to 1), where θ [deg] is the virtual elbow angle.

#### **4.1 Methods**

Initially, 13 laboratory members that specialize in VR and/or haptics participated in the experiment, but because malfunctions of Leap Motion occurred during experiments involving three of the participants, their data were excluded from subsequent analyses. Finally, data from 10 participants (22 to 25 years old, average 23.6 years old, all right-handed, one female, nine male) were analyzed.

First, the acceleration amplitude of the BB vibrator attached to the participant's arm driven with a 100 Hz sine wave was adjusted to approximately 130 m/s2 (which was determined to stably evoke kinesthetic illusion), using an accelerometer (LIS331HH, STMicroelectronics). We also presented 100 Hz sine wave stimuli and confirmed orally that the kinesthetic illusion occured in both the extension and flexion directions. When the illusion was not sufficiently obtained, the position of the vibrators was re-adjusted until the illusion occurred.

The participants then performed a task to evaluate the operability of the system based on Fitts' law [5,11] for each condition. The task consisted of 50 trials. In each trial, the participants controlled the virtual forearm and kept the angle aligned with the target for 1 s. The center position of the target was randomly generated from −30◦ to 30◦ and the width was randomly generated from 5◦ to 15◦. White noise was presented using the built-in headphones of the HMD during task execution.

After the task for each single condition was complete, the participants answered the questions shown in Table 1 using the seven-point Likert scale (−3: totally disagree, +3: totally agree), to evaluate body ownership and agency. This questionnaire was a modified version of the questionnaire used in the study on rubber hand illusion by Kalkert et al. [9] and consisted of four categories: Ownership, Ownership Control, Agency, and Agency Control. The questions were translated into Japanese.

To cancel the order effect, the order of the conditions was counterbalanced as much as possible. However, due to the aforementioned malfunctions, among 3! = 6 permutations, orders of conditions None-Tendon-Tactile and Tactile-Tendon-None was used only once, and the other orders were used twice.

#### **4.2 Results**

**Operability Evaluation Using Fitts' Law.** In the task results for each participant and each condition, the equation of the modified Fitts' law MT = a +

**Table 1.** Questionnaire for evaluating ownership and agency (based on Kalckert et al. [9])

b log2(A/W + 1) [11] was fitted, where MT [s] is the time required for the trial, A [deg] is the difference between the angle at the start of the trial and the target angle, and W [deg] is the width of the target. Additionally, the index of performance IP was calculated [11]. Figure 2 compares the average IP for all participants under each condition. The repeated measures analysis of variance showed no significant differences between the conditions (p = .898). As a result of multiple comparisons using the Bonferroni correction, no significant differences were found (p = 1.000 for all pairs). Additionally, the correlation coefficient r of the fitting ranged from 0.139 to 0.851 with an average of 0.532.

**Questionnaires.** In the same manner as [9], the answers to questions belonging to the same category were averaged, and four scores (Ownership, Ownership Control, Agency, and Agency Control) were calculated (Fig. 3) and further analyses were done using these scores. The Wilcoxon signed-rank test between Ownership and Ownership Control, Agency and Agency Control was performed in each condition and significant differences in both Ownership-Ownership Control and Agency-Agency Control were found in all conditions (p<.05). Also, the Friedman test was performed between conditions for each score, but there was no significant difference (p = .393 for Ownership, p = .087 for Ownership Control, p = .607 for Agency, p = .098 for Agency Control).

#### **4.3 Discussion**

**Operability.** There was no significant difference in IP between the conditions, and as shown in Fig. 2, the average IP for the Tactile and Tendon conditions were almost the same or slightly lower than that of the None condition. Hence, we consider that operability was not improved by presenting kinesthetic illusion in the proposed system.

**Fig. 2.** Comparison of the average *IP* by condition (error bars indicate the standard deviation)

**Fig. 3.** Comparison of the questionnaire scores by condition

**Body Ownership and Agency.** For the results of the questionnaire, there was no significant difference in scores between the conditions, possibly due to large variability between participants. However, in Fig. 3, ownership tended to increase in the Tendon condition in comparison with other conditions. Therefore, while we cannot conclude that tendon vibration was effective in the movement-free VR interface, it may contribute to the generation of ownership.

Additionally, in Fig. 3, a small decrease in Agency and an increase in Agency Control in the Tendon condition were observed. In fact, some participants' comments suggested a lack of agency, such as "a feeling of being moved by others" for the Tendon condition. We consider that the loss of agency was because movement simulation and the method for the vibration presentation were imperfect, and the kinesthetic illusion was different from the intended motion of the user.

# **5 Conclusion**

In this paper, we proposed and implemented a VR system that requires no physical body movement by detecting the force exerted by the user's fixed body and presenting kinesthetic illusion. As a result of the experiment, an improvement in operability caused by kinesthetic illusion was not confirmed. Additionally, although not statistically verified, the results suggest that the kinesthetic illusion may lead to an improvement of ownership and decrease of agency. As future work, a more precise verification of usefulness and improvement of the presentation method are required.

**Acknowledgements.** This research was supported by JSPS KAKENHI Grant Number JP18H04110.

### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Haptic Display Using Fishing Rod

Daiki Naito(&) and Hiroyuki Kajimoto(&)

The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo, Japan<EMAIL_ADDRESS>
Abstract. This paper proposes a grounded haptic display that can widely present a force sensation using a fishing rod. The proposed haptic display incorporates a fishing rod to present the force sensation at the user's fingertip using a thread being reeled from the tip of the fishing rod. The multidirectional force sensation is presented by driving the base supporting the fishing rod using a pan-tilt mechanism, and wending the thread using the reel. The fishing rod is lightweight and moves at high speed. Additionally, it is possible to widely present the force by bending the fishing rod. This device requires conversion from the position of the user's fingertip and magnitude of the force to the necessary winding force of the thread and the posture of the fishing rod. This study developed a control method and evaluated it by considering a case wherein the force to be presented is exerted in the vertical direction.

Keywords: Virtual reality Human interface Haptic device Fishing rod

# 1 Introduction

In the field of virtual reality, many studies are developing sensory presentation devices that reproduce sensations felt in the real world. Presenting a force sensation is an important element with regard to improving the perception of objects in virtual space [1].

A haptic display is a system that accurately tracks the movement of the user and presents the required force to their fingertips and tools [2]. Users can perceive the sensation of touching virtual objects created by a computer using the haptic display [3– 5]. To present a force in an arbitrary direction and at an arbitrary point in space, the first basic structure to be considered is a structure using the robot arm of a serial link such as PHANToM [4]. One drawback of this type is that, to realize a large workspace, a rigid bar with the same length as the workspace moves within the space, which increases the risk and requires a motor with a large output at the base to generate a force at the end of the rigid bar. Another possible structure is the use of thread traction, as in the case of

Electronic supplementary material The online version of this chapter (https://doi.org/10.1007/ 978-3-030-58147-3\_36) contains supplementary material, which is available to authorized users.

SPIDAR [5]. This device type has a problem that numerous winding mechanisms must be installed to cover the entire workspace. Hence, a large-scale device is required to realize a wide workspace.

To this end, we propose a new type of haptic display that can present a force sensation to the fingertips using a fishing rod shown in Fig. 1. The force sensation is presented to the finger by attaching a thread to the finger and controlling the winding force of the thread. The base of the fishing rod has a pan-tilt mechanism. Because the fishing rod is lightweight, even a motor with a relatively low output can control its attitude at high speed. Although there is a restriction whereby the pulling direction of the thread cannot be set in a perfectly arbitrary manner (for example, a downward force cannot be generated), this might serve as a technique with a wide workspace, in the sense of range.

This paper presents the configuration of the proposed system and the development

Fig. 1. Overview of device: (left) fishing rod is not bent; (right) fishing rod is bent.

and evaluation of a control method in the case wherein the force direction is vertical.

#### 2 Related Work

Here, the haptic displays using a thread are described. The thread-based haptic displays can be approximately divided into wearable type and grounded type displays.

The wearable type uses a technique for presenting the force sensation by mounting the device on the user's body. Hirose developed HapticGEAR [6] and Hosseini developed a similar device using twisted-thread actuators [7]. In this wearable type of haptic displays, the range wherein the force sensation can be exerted is wide, and the user can use it freely. However, wearing the device is time-consuming and imposes a burden to the user owing to the weight of the device. Additionally, the reaction force is inevitably generated in the mounting part when the force sensation is presented.

In contrast, a grounded thread-based display type, such as SPIDAR [5], fixes the device to the wall or floor. The device essentially comprises a cube type frame and four motors at the vertices, and exerts the force sensation to the finger using winding threads. The grounded type can generate a strong and accurate force because the device is grounded and not affected by the reaction force. However, there is a problem whereby the range of the force to be presented is limited within the cube. Additionally, although this device type can apply a multi-directional force using multiple threads, the operating range of the fingertip is more limited owing to interference amongst the threads. SPIDAR has been under development for a long time and comes in various types [8]. SPIDAR-S [9], which presents a force sensation to the user's finger with a single thread, and SPIDAR-MF [10], which corresponds to wrist twisting using a rotatable frame structure, have been proposed to solve the workspace problem. However, SPIDAR-S does not assume a multiple degree-of-freedom (DoF) force sensation, and SPIDAR-MF focuses on preventing the interference of threads corresponding to multiple fingers.

Based on these considerations, a haptic device using a thread currently has three requirements: suppressing the generation of the force to a position except for the intended position; preventing the interference between the body and the threads; presenting a force sensation within a wide range. The proposed haptic device satisfies these three requirements at the cost of sacrificing the accuracy of force direction. The problem of the repulsive force has been solved by developing the device as a grounded type. The interference between the body and the threads is prevented by using one thread from the fishing rod. Finally, the presentation of the force sensation within a wider range is realized using a method for driving the base of the fishing rod.

# 3 Device Configuration

The proposed device is shown in Fig. 1. A thread coming out of a fishing rod is attached to the user's fingertip end-effector. To wind up the thread, a DC motor (Maxon, 135079, 10 W) is attached to the reel and controlled using PWM control. To present the multi-directional force using the fishing rod, two servomotors (JX Servo, C 70 Digital Servo, 92 W) are mounted onto the base supporting the fishing rod, and a pan-tilt mechanism is adopted. The fingertip position is detected by an optical motion capturing system (OptiTrack, V 120: Trio) installed around the device, and optical markers (retroreflective balls) are attached to the fingertip end-effector. With the abovementioned device configuration, the winding of the thread attached to the endeffector and the posture of the fishing rod are controlled to present the force sensation.

Three effects are expected when using the fishing rod as a haptic display. First, compared with the case of multi-threads, the calculation of the force is simpler because the tension direction of the thread matches the direction of the force presented to the fingertip. Second, because the fishing rod is lightweight, it is possible to control the posture of the fishing rod at high speed, even when using a motor with relatively low torque. Third, the fishing rod can innately present a strong force to the fingertip. Although the necessary torque is calculated from the length of the fishing rod and thread tension, it can be suppressed when a strong force is required because the bending of the fishing rod has the effect of equivalently shortening the length of the fishing rod such as right figure of Fig. 1. In this case, however, the accuracy of the force direction is reduced.

#### 4 Control Method

In this study, we considered a control method for presenting a directional force. Accordingly, we limited the investigation to the case of a vertical force; that is, the thread attached to the fingertip always exerts a straight upward pull. The optical motion capturing system detects the position of the fingertip and the proposed device vertically pulls the thread attached to the fingertip by controlling the winding of the thread and the posture of the fishing rod.

In this case, the fingertip position becomes the input information of the control system. When introducing this device into a virtual reality application, it is necessary to assess whether it touches the object in the virtual space or not, and to change the presenting magnitude of the force. Additionally, it is important to change the magnitude of the force to express the softness suitable to a virtual object. In other words, the direction and magnitude of the force presented to the fingertip are the targets that should be realized by controlling the device.

In the case of a serial link type haptic device, and particularly with regard to the control of an impedance device type such as PHANToM [4], the position of the end effector is the input, as in the present situation, while the direction and magnitude of the presented force are the targets to be realized. Because a conventional serial link haptic device does not deform the link mechanism itself, it is easy to measure the position of the end effector by carrying out forward kinematics calculations to obtain the torque required by the motor through inverse dynamics calculations. In our case, it is currently difficult to derive a calculation formula that considers the bending of the fishing rod in relation to the force presented at the fingertip using the proposed device.

Thus, we initially attempted to perform feedback control using the difference between the target direction of the force and the current direction of the thread such as Fig. 2(a). The optical marker was not only attached to the fingertip end-effector, but also to the thread (Fig. 2(b)), and the direction of the thread was obtained.

Fig. 2. Overview of feedback control; (a) consideration: (b) actual situation.

However, when the device was actually operated through feedback control, the problem of thread vibration occurred. One of the causes for this is that the sampling rate of the optical tracking was 120 Hz, which is close to the resonant frequency of the thread itself and increased the control difficulty. Additionally, the resonant frequency of the thread dynamically changed with the change of its length and tension.

Based on the abovementioned trial, we considered that it is currently difficult to control the fishing rod using feedback control. Thus, we decided to control the fishing rod using feedforward control. As mentioned above, the measurement information in the proposed device is the fingertip position, and the target values are the magnitude and direction of the force presented at the fingertip. This time, the target direction of the force was fixed to be vertical. Moreover, it is desirable that the winding force of the thread and the angle of the fishing rod are obtained as output information. To this end, it was necessary to investigate the relationship between the position of the fingertip and the winding force of the thread, the posture of the fishing rod, and the force generated by the winding force of the thread and posture of the fishing rod. In the next section, we present the measurement and fitting of the relationship between the input and output information.

#### 5 Measurement

#### 5.1 Measurement Method

In the measurement, the relationship between position x of the fingertip and the duty ratio command value d to the motor used for winding the thread, the attitude h of the fishing rod, and the force f generated by the winding force of the thread and the attitude of the fishing rod were measured. The measurement environment is shown in Fig. 3. We attached a 1.2-kg load instead of a finger. The position x was the distance from the device to the load. The angle of the fishing rod is defined as 0° when it is parallel to the ground. To measure the force of the thread pulling the load, the load was placed onto an electronic balance. The weight of the load was recorded by starting the electronic balance before the measurement, and the traction force was calculated from the change of the electronic balance when the load was pulled by the thread.

The load was placed at 22.5 cm, 45.0 cm, 56.25 cm, 67.5 cm, 78.75 cm, and 90.0 cm with the device position as the origin. The duty ratio of the motor winding the thread at each position was changed by 10%, from 10% to 100%. Then, the angle of the fishing rod was adjusted such that the thread pulled the load exactly in the vertical direction. Additionally, because the motor of the reel started to rotate when the duty ratio was 5% or more, measurements were also made when the duty ratio was 5% at

Fig. 3. Measurement environment.

each position. The measurement of the fishing rod's angle and the traction force to the 11 values of the duty ratio was repeated five times while the position was fixed, and the mean and the standard deviation were calculated.

#### 5.2 Measurement Result

The measured means and the standard deviations were plotted in Fig. 4. The relationship amongst location x of the load, duty ratio d of the motor controlling the winding of the thread, and angle h in the pitch axis direction of the fishing rod is shown in Fig. 4 (a). Figure 4 (b) shows the relationship amongst location x of the load, the duty ratio d of the reel motor, and the force f of the thread pulling the weight.

In Fig. 4(a) and (b), only one data point existed when the load was set to 90 cm. The total length of the fishing rod used in this study was 90 cm. Additionally, the duty ratio of 5% was the minimum duty ratio at which the motor of the reel started to rotate (the fishing rod began to bend).

The measurement data obtained when the load was placed at 22.5 cm and 78.75 cm was smaller in the number of plots than the data obtained when the load was placed from 45.0 cm to 67.5 cm. When the load was set to 78.75 cm, the posture of the fishing rod for pulling in the vertical direction was not observed with a duty ratio higher than 40%. Moreover, when the load was installed at 22.5 cm and the duty ratio exceeded 20%, the fishing rod began to bend not only in the pitch axis direction, but also in the roll axis direction. Therefore, the duty ratio over 30% was not measured.

According to Fig. 4 (a), the fishing rod's angle increases as the position of the load approaches the device. Similarly, in Fig. 4 (b), the traction force also increases. Increasing the fishing rod's angle is the same as increasing its whip. Therefore, it can be confirmed that the traction force also increases, when the whip of the fishing rod increases.

Fig. 4. Measurement results: (a) relationship between duty ratio of motor winding the thread and attitude of fishing rod; (b) relationship between duty ratio of motor winding the thread and pulling force of the thread.

In the proposed feedforward control, the position x of the fingertip and the magnitude f of the presented force are given, and the duty ratio d required by the motor winding the thread and the angle h of the fishing rod must be obtained. From Fig. 4(b), the duty ratio d of the motor winding the thread was derived by inputting the position x of the fingertip and the strength f of the force to be presented. Then, based on Fig. 4 (a), the posture h of the fishing rod was determined from the position x of the fingertip and the duty ratio d of the motor.

To realize the abovementioned procedure, the regression equation obtained by the least squares method was derived from the measurement data presented in Fig. 4. The regression curves based on the regression equation are shown for several locations in Fig. 5 and Fig. 6. Figure 5 shows the regression curve obtained from x and f to d, using a second order polynomial, while Fig. 6 shows the regression curve obtained from x and d to h using a first order polynomial, which indicates that the regression curve adequately represents the data obtained by measurement.

Our control scheme was confirmed by performing actual feedforward control. The amount of force was fixed and the fingertip was moved. As shown in Fig. 7, the thread always pulled the fingertip straight upward, even when the fingertip was moved, which indicates that the desired feedforward control was realized.

Fig. 5. Comparison of regression curves with measurements of thread pulling force and force winding the thread: (a) x = 45.0 cm; (b) x = 56.25 cm; (c) x = 67.5 cm.

Fig. 6. Comparison of regression curve with measurements for force winding the thread and posture of fishing rod: (a) x = 45.0 cm; (b) x = 56.25 cm; (c) x = 67.5 cm.

Fig. 7. Validation of control; the fingertip is continuously pulled in vertical direction with the same amount of force.

# 6 Conclusion

This paper proposes a haptic device using a fishing rod and describes the proposed control method that pulls the fingertip in the vertical direction. The relationship between the data was investigated by measuring the position of the fingertip, winding force of the thread, posture of the fishing rod, and force of the thread pulling the fingertip. The regression curves were obtained based on the measured data and incorporated into the control of the device. Thus, it was always possible to pull the fingertip in the vertical direction, even if the fingertip was moved.

However, the proposed control method is obviously a simplification, and must be expanded to all force directions and fingertip positions. The vibration of the fishing rod by quick motion is another issue that must be solved. Additionally, we must clarify the space wherein an accurate force direction can be reproduced, and the space wherein the force direction cannot be reproduced but the force amplitude can be reproduced. This should be coupled with investigations into how much the wrong force direction affects the perception of force sensation and the related task performance.

Furthermore, a user study within a virtual reality environment should be also conducted, both with regard to subjective evaluation, such as softness presentation, and task performance. From now on, we will evaluate the performance of the device with a focus on the user studies.

Acknowledgment. This research was supported by JSPS KAKENHI Grant Number JP18H04110.

# References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Confinement of Vibrotactile Stimuli in Periodically Supported Plates**

Ayoub Ben Dhiab(B) and Charles Hudin

CEA, LIST, 91191 Gif-sur-Yvette, France<EMAIL_ADDRESS>
**Abstract.** For multitouch and multiuser interactions on a touch surface, providing a local vibrotactile feedback is essential. Usually, vibration propagation impedes this localization. Previous work showed that narrow strip-shaped plates could allow the confinement of vibrotactile stimuli to the actuated area. Adding to this principle, periodically supported plates also provide a non-propagative effect at low frequencies. Using both geometrical properties, we provide a device allowing a multitouch interaction through an array of piezoelectric actuator. Experimental validation show that vibrations are well confined on top of actuated areas with vibration amplitude over 2 µm.

**Keywords:** Surface haptics · Confinement · Vibrotactile stimuli · Narrow plate · Ribbed plate · Non-propagating · Evanescent wave · Vibrations

# **1 Introduction**

Wave propagation in today's surface haptic interfaces limits the user to a single point of interaction whereas most exploratory procedures benefit from the use of multiple fingers to properly process surface information [7]. Solving this problem requires to localize vibrations. There are several methods related to technologies that allow localized haptic feedback on continuous surfaces. For spatial localization of vibrations, i.e., the creation of local deformation at a point or area of the plate, there are two techniques: *Time-Reversal* [5] and *Modal Superimposition* [4]. For these two techniques, the wavelength of the vibrations to obtain a local deformation at the centimeter or millimeter scale necessarily involves high frequencies. The *Time-Reversal Wave Focusing* technique uses the propagation of elastic waves to generate constructive interference at a given time and position on the plate. The acceleration peak created by the focusing of the waves causes the ejection of a static or moving fingerpulp, giving a haptic feedback smaller than the fingertip. However, this approach needs a calibration procedure and is subjected to external parameters (e.g. temperature, finger interaction), which can impede the tactile stimulation. As for *Modal Superimposition*, it uses a truncated modal decomposition to focus a deformation shape and control its position. The high frequencies produced by an array of piezoelectric actuators allow for the appearance of a tactile lubrication phenomenon, i.e. a variation in friction that is only felt when the finger is moved [10]. Both techniques rely on propagation of high frequency waves. During user interaction, the non-linear behavior of the finger significantly modifies the vibrational behavior of the plate. Such modifications are particularly striking for high frequencies thus leading to many complications (incl. flexural waves scattering, mode damping, mode translation). In order to avoid these limitations and provide a haptic feedback both to static and moving fingers, the *Inverse-Filter* approach [8] proposes to dynamically control low-frequency waves, which are less impacted by finger interaction, to provide the user with the desired stimuli only at the contact points. Nonetheless, this technique is based on (consequent) computation, signal processing and calibration procedures. As this technique uses waves within the tactile frequency range (0–1 kHz), it induces a global movement of the plate and therefore limits the interaction to five control points. Generally, all the previously mentioned methods depend on wave propagation and thus, are in need of calculations and signal processing in order to achieve localization. Therefore, we have investigated in our previous work [1] another method that overcomes such requirements by relying on geometry features and wave evanescence instead. We have shown that it is possible to obtain localized deformations above the actuators for low frequency signals in narrow thin plates (1D). A particularity of this technique is that it allows the spatial localization of low-frequency stimuli i.e. a local deformation of the plate even if we use low-frequencies with long wavelength. In addition, finger interaction does not attenuate low-frequency evanescent waves. No matter the surface in contact, i.e. finger, hand, arm, foot... the behavior of the vibrating plate is not impacted and a localized stimuli can be provided. Although interesting, this approach is limited to narrow plates. However, for a rich multitouch interaction, extending this approach to an arbitrary 2D plate becomes necessary and is the focus of this paper. Because waves tend to directly propagate in 2D plates, the confinement i.e. the suppression of propagative waves in favor of evanescence, was not easily achieved. In order to do so, we used a theoretical result raised by Mead [6]. He states that a periodically supported beam exhibits a non-propagating low-frequency band up to the first resonant mode of isolated segments. Meaning in our case that if we have a periodically bounded plate and if each formed areas have each a waveguide geometry, a cutoff frequency exists and we can have the spatial localization of low frequency stimuli. Therefore, in this paper, we verify this assumption experimentally with a novel setup and discuss the various possibilities it opens up.

#### **2 Principle**

#### **2.1 Confinement by the Ribs**

In this section we explain how vibrations can be confined along both axis of a ribbed plate depicted in Fig. 1. On the x-axis, we find ourselves solving a problem of an infinite beam supported periodically (Fig. 1). Regularly supported beams can be seen as a set of individual supported beam portions applying

**Fig. 1.** Plate bounded by a set of equidistant ribs forming *n* areas which theoretically behaves like narrow-plate bounded on their longer edges to a rigid frame.

bending moments (represented by rotation spring in Fig. 1) to their neighboring beam portions. In such systems, vibrations propagate from one portion to another through bending moments produced at the junction between two portions. Mead [6] has shown that for frequencies bellow the first resonance of an individual portion, the propagation from one portion to another attenuates exponentially. This means that in our system, the cut off frequency f1 that marks the transition between propagative and evanescent waves along the waveguide dimension, also corresponds to the transition from propagative to evanescent wave in the transverse dimension. This enables the extension of the waveguide approach describes in [1] to 2D surfaces regularly supported by thin ribs.

#### **2.2 Evanescence and Waveguide Geometry**

The localization along the y-axis of low frequencies is implied by the waveguide geometry of the propagation medium. In such medium, it was shown that bending waves cannot propagate below the cut-off frequency of the first propagation mode. This non-propagation effect allows the confinement of vibrotactile stimuli and thus can be used for multitouch haptic interactions. If we suppose a perfect bounding of the ribs to a rigid frame, each of the n areas possesses their first cut-off frequency equal to [1]:

$$f\_1 = \frac{\pi}{4\sqrt{3}} \frac{h}{W^2} \sqrt{\frac{E}{\rho(1-\nu^2)}}\tag{1}$$

with ρ the mass density, E the Young's modulus, ν the Poisson's ratio, h the plate thickness, W the distance between ribs. When an area is submitted to a vibration of frequency f<f1, waves propagating along the y-axis and the x-axis are evanescent and dies exponentially with the distance.

# **3 Experimental Validation**

#### **3.1 Apparatus**

To verify the spatial localization provided by our method, we have set up a system designed to allow a user to freely explore a surface. The size of the device thus corresponds to the size of an adult hand, and the number of areas formed corresponds to the number of fingers. The only requirement associated with the implementation of this method is having a plate correctly bounded periodically to a rigid frame. If each created area have a waveguide geometry, a cut-off frequency exists and a localized vibration can be obtained.

**Mechanical Components.** The apparatus consists in a glass plate, measuring 150 × 130 × 0.5 mm3, bounded on a set of 6 equidistant ribs of width 4 mm forming 5 areas. In order to provide haptic feedbacks, piezoelectric actuators (muRata 7BB-20-3) of circular geometry, composed of two part: a plate of diameter 20 mm, thickness of 0.1 mm; and a piezoelectric transducer of diameter 12.8 mm, thickness of 0.11 mm, were glued on the bottom side of the plate between each ribs. The out of plane displacement of the surface is measured by a laser vibrometer (Polytec MLV-100/OFV-5000) mounted on a motorized 3 axis platform. This setup is illustrated on Fig. 2.

**Fig. 2.** Experimental setup. A glass plate is bounded onto a set of 6 equidistant ribs with epoxy resin. The actuators are glued to the bottom of the plate, 5 per area between each rib, for a total of 25 actuators.

**Driving Components.** Because vibration localization is induced by the propagation medium geometry, no specific calculation nor wave control implementation are needed. Signal amplification for each actuators was assured by a Piezo Haptics Driver DRV8862 from Texas Instrument. A voltage output module NI-9264 was used to send requested analog signals to each driver for amplification. Communication with the NI-9264 voltage output module and signal design was made with Python using the PyDAQmx package.

#### **3.2 System Frequency Response Function (FRF)**

The frequency response function or FRF (ratio of the displacement at a given point to the voltage applied to an actuator in the frequency domain) for the actuator was measured by sending a linear sweep signal of 40 V amplitude going from 0 Hz to 5 kHz sampled at 10 kHz and is represented on Fig. 3. We search with this FRF frequencies where the amplitude at the center of the actuator (in light gold) presents noticeable differences with the amplitude outside the actuator area (in dark blue). We can notice that, from 0 to 1.2 kHz, vibration amplitudes inside the actuator area are 30 dB higher in average than outside the actuator area which shows that elastic waves do not propagate throughout the plate for vibration frequency in this range. The FRFs at other points were also measured and show the same behavior, though with some differences. We supposed in Sect. 2 that clamping conditions were perfect thus having the same cut-off frequency for every area. But the FRF in other areas actually present different cut-off frequencies meaning that the clamping quality of each rib was not assured. The boundary condition being made with epoxy resin we can assume that eventual air bubbles or gaps could have been left during the mounting process thus giving slight behavior differences between each area and locally within each area.

**Fig. 3.** System Frequency Response Function (FRF) *HdB* = 20 log10 *|H|* at the center of the activated actuator (in light gold) and 20 mm away outside the actuator inside the same area (in dark blue). (Color figure online)

#### **3.3 Stimuli Confinement**

To illustrate the multitouch capability of our device we mapped plate amplitudes when submitted to signals sent with multiple piezoelectric actuators. We chose a 250 Hz (tactile sensitivity peak) 5 cycles burst excitation and a driving voltage amplitude of a 100 V for every actuator used. Figure 4 shows the maximum displacement throughout the plate when using 1, 2, 3, 5, 6 and 7 actuators at the same time. We can notice that waves are confined above activated actuators with amplitudes reaching 3µm inside the actuator area. Outside the actuated area, waves propagating along the y-axis are evanescent and die exponentially. Along the x-axis, vibration propagation is impeded by the clamped ribs surrounding the actuator. The energy representation with a logarithmic scale of Fig. 5 shows certain areas of interest. Apart from localized energy area above each actuator, we can note leaks with a 15 dB amplitude difference between vibrations inside actuator areas and leaked vibrations while other areas present a 30 dB difference. This leak is actually due to clamping imperfection caused during the prototype assembly. Another area which is less visible (squared in green in Fig. 5), asses the evanescent behavior along the x-axis due to the set of supports. This important loss of energy (around 25 dB loss) comes from the evanescence behavior and the width of the ribs which, by providing strong bending moments, reduce the transmitted energy. Thinner ribs would still provide a localized effect but with a reduced energy loss.

**Fig. 4.** Maximum out of plane displacement in *µ*m of the plate for 1, 2, 3, 5, 6 and 7 actuators activated at same time. Vibration sources are localized and reach a vibration amplitude above 3 µm while the rest of the plate stays at rest.

**Fig. 5.** Energy representation with a logarithmic scale of the plate submitted to 5 activated actuators. Most of the energy is localized above each piezoelectric actuator.

#### **4 Discussion and Perspective**

The device shown here can provide localized vibrotactile feedback in several areas of a continuous plate. The confinement is achieved along both axis by low frequency evanescence phenomenon: one based on the geometry of the propagation medium; the other based on the periodically placed supports. This system is particularly unique and there is to our knowledge no setup with such capabilities and we believe that our device offers new possibilities for studying the tactile perception and to develop new interactions. The interface is simple and robust, i.e. it does not require any special signal processing or computation to work and is not impacted by user interaction. However, since evanescent waves are used, the localization of stimuli is only done in the area of each actuator. In order to increase the resolution of such a device, the number of actuators must be multiplied as well as the number of ribs. It raises questions about the control electronics, the power consumption and the available amplitude (sufficient amplitudes cannot be guaranteed with small actuators). Another point that can also be addressed is concerning possible use case of such device. The authors believe that the method in question is easily applicable to medium to large surfaces such as car dashboards. Breitschaft and al. [2] indicates that in the automotive domain four tasks must be performed by tactile interiors: exploration, detection, identification and usage. These tasks are already achievable as is if the method is applied to a thin touch screen (<2 mm). The user can explore the surface freely with one or more fingers or by sliding or tapping or even by placing his hand entirely on the surface without changing its behavior. The detection of the localized stimulus and its identification is ensured by the shape and frequency content of the signal used, which can also be varied according to the position of the fingers if one wants to create areas with different vibratory behaviors. Finally the confirmation action is done by the selected zone by sending a signal with a strong frequency dynamic to simulate a button (this function can be improved by adding a force sensor). Future works will focus on two axes, one dealing with the possibility of thinner ribs and the other on an interface use case, in particular the localization of a vibratory targets on a flat continuous surface.

# **5 Conclusion**

At low frequency, bounded ribbed-plates allow for the localization of vibrotactile stimuli. A piezoelectric actuator bounded to such medium mostly produce evanescent waves which are confined to the actuator covered area as long as the driving frequency is lower than the first cut-off frequency of our system. No control strategies nor signal processing are needed, directly sending the desired signal to the desired actuator suffice in order to obtain a localized vibrotactile stimuli. Multitouch and multiuser interactions can therefore be implemented with ease. Application such as tactile fingerspelling [3], tactile keyboard, tactile memory games and applications using exploratory procedures on a plates can be implemented here. Although the presence of actuators underneath the plate limits applications to surfaces where transparency is not necessary, recent breakthroughs shows that transparent piezoelectric actuators are at hand [9] making our technology compatible with screen surfaces.

**Acknowledgment.** The revised version of this article benefited from insightful comments from Dr. Sabrina Pan¨eels.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **2MoTac: Simulation of Button Click by Superposition of Two Ultrasonic Plate Waves**

Pierre Garcia1,2(B), Fr´ed´eric Giraud1, Betty Lemaire-Semail1, Matthieu Rupin2, and Michel Amberg1

1 Univ. Lille, Centrale Lille, Arts Et Metiers Institute of Technology, Yncrea Hauts-de France, ULR 2697 L2EP, 59000 Lille, France *{*pierre.garcia,frederic.giraud<EMAIL_ADDRESS>2 Hap2u, 20 Rue du Tour de l'Eau, 38400 Saint-Martin-d'H`eres, France<EMAIL_ADDRESS>
**Abstract.** Recent studies have shown that a button click sensation could be simulated thanks to ultrasonic vibrations. In this context, a travelling wave may enhance the simulation because it creates internal lateral stresses that are released during the stimulation. However, this solution is difficult to integrate on plates. We present 2MoTac, a method which superpose a longitudinal and a bending mode simultaneously on a plate, in order to create a pseudo-travelling wave. We present the design, and a psychophysical study to deduce the optimal ratio between the bending and longitudinal mode amplitudes, in terms of detection threshold and robustness.

**Keywords:** Haptic display · Tactile perception · Ultrasonic vibration · Vibration mode · Keyclick · Button click

# **1 Introduction**

Most of new human-machine interaction devices now rely on touch screens. They have become so cost effective that physical buttons and knobs are removed and replaced by virtual buttons displayed on the flat and hard surface of the display panel in vending machines, car dashboards, and so on. However, touch screens do not involve the sense of touch in the interaction with the machine: they do not provide information through this channel, unlike physical buttons for instance. Therefore, sight is predominant for whom wants to interact with the aforementioned machines, leading to many disadvantages: elderly and visuallyimpaired individuals struggle to use touch screens, and drivers must look away from the road, which is a major safety issue in automobile. Therefore, technological improvements to touch screens are needed, in order to involve touch when interacting with a machine through its touch screen.

To create the illusion of touching a button on a panel, lateral displacement [1] or vibrotactile stimulation [11] can be used. The low frequency vibration of the touch screen directly stimulates the skin mechanoreceptors; because this vibration is difficult to contain, [3] identifies the frequency response of the touch screen (FRF) and then uses the frequency that maximises the vibration displacement at a specific point. In [7], the authors invert the FRF and can control the vibration at three different positions where actuators are located. However, using vibrotactile stimulation to create the simulation of a button click has the disadvantage to produce audible noise. To cope with this issue, [10] uses the rapid vibration of a standing wave at ultrasonic frequency, to reduce the internal stresses inside the finger pulp due to the friction that appears when pressing the touchscreen. More robust results are obtained if a net tangential force is produced, by using electroadhesion force synchronized on ultrasonic lateral displacement [15] or by using a travelling wave [6]. These solutions however seem difficult to be integrated on plates, without obstructing the view by actuators.

This paper introduces a new concept of tactile stimulation that uses two orthogonal modes to simulate a button click. The elliptical motion of the particles in contact with the fingertip creates a lateral force as if a travelling wave was used. After presenting the technological principle that derives from [2], a study has been conducted in order to define the ratio between the two modes that optimizes the feeling of a single click when reversing the lateral force direction.

#### **2 Presentation of the Two Modes Stimulation**

#### **2.1 Theoretical Background**

We consider a plate, which length is L, thickness b and height H. Its vibration can result in the superposition of different types of modes: the longitudinal modes are characterized by the out-of plane translation of the plate's cross-sections while bending modes are characterized by an in-plane translation and an out-of plane rotation of the cross-section, as described in Fig. 1.

**Fig. 1.** Vibration mode a) Longitudinal, b) bending and c) Elliptical motion in the middle of the top surface for several values of Λ

When the two modes are produced at the same vibrating frequency, then the point in the middle of the top surface undergoes a displacement which has two components:

In this paper, the two modes are energized at the same frequency f, with a phase shift of ±π/2. We then write:

$$a\_L(t) = \frac{A\_L}{2}\sin(2\pi ft) \qquad a\_B(t) = \frac{A\_B}{2}\cos(2\pi ft) \tag{1}$$

with A*L* and A*B* the vibration amplitude of each mode (peak-peak); if A*B* < 0, the phase shift is π/2. Hence, by combining the two vibration modes, the points at the top surface follow an elliptical motion, which is able to create a net lateral force, as described in [4]. By changing the ratio Λ = *AL AB* , the shape of the elliptical trajectory is modified, as described in Fig. 1c), promoting the normal displacement over the longitudinal one for Λ < 1 and vice versa for Λ > 1. In the remaining of this paper, we introduce the maximum vibration speed U*L* = 2πfA*L* and U*B* = 2πfA*B* because they combine frequency and displacement.

#### **2.2 Related Work**

To produce a button click sensation, [10] uses a rapid change in friction between the fingertip and a plate by increasing or decreasing ultrasonic vibration. The detection threshold of the click is found to be lower if the friction is decreased compared with a friction increased. A fraction of the elastic energy stored in the finger pulp is due to some tangential contact forces that appear during touch. When it is suddenly released, it creates a stimulus that is detectable by the skin's mechanoreceptors. However, the detection thresholds of this click are higher compared with those measured to detect a friction change in presence of a lateral motion of the fingertip [12]. Therefore, [6] suggests to use a travelling wave instead of a standing wave. Indeed, the results show a lower perceptual threshold for travelling waves, and participants press less times per trial and exert smaller normal force on the surface in order to perceive click effect.

Travelling wave can increase the tangential contact forces due to the elliptical motion of the particles that are in contact with the fingertip, thus increasing the stimulus when they are released. Two theories can explain this phenomenon. In [14], the tangential force derives from the contact conditions between a vibrating plate and an elastic medium. In [9], the authors introduce the lateral viscous forces that are generated by a travelling wave, a phenomenon which is compatible with the existence of the squeeze-film. For both theories, the lateral force is a complex function of the normal and tangential displacements of the particles.

In a travelling wave, the ratio Λ between the normal and lateral vibration amplitude is fixed by the geometry of the vibrating plate. To cope for this issue, [2] presents a prototype that can produce a longitudinal and a bending mode which resonant frequencies are equal. However, the actuator for the longitudinal mode is bulky, while the vibration amplitude of each mode are not controlled. With 2MoTact, the two components U*B* and U*L* can be independently set, offering the possibility to optimize the tuning of Λ in order to improve the lateral contact forces produced to enhance the click sensation. A study is then conducted in order to determine which ratio is optimal in terms of psychophysical threshold. To compare the thresholds between each other, the stimulation level is defined as U = -U2 *L* + U2 *B*, which combines the vibration speed for the longitudinal and the bending mode. We include in the study the case of a pure bending mode displacement (U*L* = 0, Λ = 0).

**Fig. 2.** The Experimental setup a) and typical example of 2 interlaced psychophysical staircases, which targets the 50% perceptual threshold by a one-up one-down test. At each trial, the implemented staircase is chosen at random with a probability of 1/2.

#### **2.3 Design of the Plate**

Our prototype consists of a 148×18×2 mm3 aluminium plate, actuated by nine piezoelectric ceramic actuators (5 mm × 7 mm × 0.5 mm from Noliac, Denmark) as presented Fig. 3c). The plate was designed such that both modes (longitudinal and bending) can be excited at the same frequency (f = 34 kHz). To energize the longitudinal mode, eight actuators placed on the upper and lower side of the plate are used. For the bending mode, only one ceramic placed in the middle of the plate on the lower side is used. Two other ceramic plates (14 mm × 2 mm × 0.3 mm, Noliac Denmark) are added as sensors. Two voltage amplifiers (WMA-300 from Falco, The Netherlands) can apply a voltage up to 300 V peak to peak to the actuators' terminals.

The deformation shape for each mode is presented in Fig. 3. Due to the deformation shape for each mode, the ratio Λ is not constant over the top surface, and its value is specified in the middle of the plate. The speed and phase of the vibration of each mode are controlled independently, using the vector control method [4,5]. From the vector control method we can define:

$$U\_L = \sqrt{U\_{Ld}^2 + U\_{Lq}^2} \qquad U\_B = \sqrt{U\_{Bd}^2 + U\_{Bq}^2} \tag{2}$$

For instance, an elliptical motion of the particle with the same vibration speed along x and y of 0.3 m/s is simply obtained by setting the references U*Ld* = U*Bq* = 0.3 and U*Lq* = U*Bd* = 0. The closed loop control is embedded into a DSP (STM32F405 from ST Microelectronics). The response time of the vibration speed for both modes is set to about 2 ms as presented in Fig. 3b).

**Fig. 3.** Measurements on the device; a) deformation mode shape for the longitudinal mode (up) and the bending mode and b) transitory response of the vibration when Λ is switched from *−*1 to 1 at U*B* = U*L* = 0.3 m/s. c) Side view of the aluminium plate. (1) longitudinal sensor, (2) bending sensor, (3) longitudinal actuator (4) bending actuator

#### **3 Materials and Methods**

#### **3.1 Participants**

Data were collected from seven healthy volunteers aged between 18 and 40 (3 females). Participants were wearing noise-cancelling headphones in order to prevent noise disturbance. All participants gave written informed consent. The investigation conformed to the principles of the Declaration of Helsinki and experiments were performed in accordance with relevant guidelines and regulations.

#### **3.2 Experimental Set-Up**

The plate is mounted on a force sensor that measures the normal force at which a participant presses on the plate as shown in Fig. 2a). The keyclick rendering was performed as in [6]; two normal force thresholds were defined f1 and f2 = f1 + 0.3N. When the user reaches f1, we turn on the two modes, with specific vibration speeds U*L* and U*B* leading to a value for Λ. When f2 is reached, the direction of the elliptical motion is reversed simply by inverting U*B*.  The force is sampled at 10 kHz by the DSP, and a Laptop PC is used to send the threshold value f1 and the amplitude set points U*B* and U*L* to the DSP through USB.

#### **3.3 Experimental Procedure**

Before starting the experiment we ask the participants to wash their hands and we clean the plate to standardize the surface.

After a training period during when participants could discover the haptic feedback by testing the device at maximum intensity, at different Λ and at two different activation force levels, they had to press once on the middle of the plate with their index finger as shown in Fig. 2. They were asked to say whether they could feel the virtual click or not. The estimation of the psycho-physical threshold was performed with a simple one-up one-down staircase procedure, which targets the 50% performance level on a psychometric function, corresponding to the level at which the probability of a detectable click equals the probability of an undetectable one [8].

As in [6], two force levels f1 = [0.1N, 0.4N] are interleaved in order to avoid the prediction of the next stimulation intensity. Five different values for Λ = [0.5 2/3 1 1.5 2 ∞] were tested. The order of the conditions was pseudo-randomized across participants to avoid learning curve effects. The experiment ended when 6 turnovers or 60 trials were achieved.

## **4 Results**

For all force criteria (f1 = [0.1N, 0.4N]) and vibration ratio Λ= [0 0.5 2/3 1 1.5 2 ∞] we have computed the median level thresholds which are presented Fig. 4 and in Table 1.

The result shows very different threshold levels, depending on the value of Λ: for Λ ∈ [1/2 2/3 1], the threshold is close to 0.7 m/s, and doesn't change much with the force level. For Λ ∈ [1.5 2 ∞], the threshold level is higher (around 0.9) and decreases when the force level f1 increases. Overall, the averaged threshold level is minimal for Λ = 1/2 if f1 = 0.1N and for Λ = 1 when f1 = 0.4N.

**Table 1.** Experimental results: 50% module threshold for different Λ and different activation pressure levels

Interestingly, we also compare in Fig. 4c) and d) the variance for each condition. A F-Test (α = 0.05, F*crit* = 4.28) have been performed on the variance of threshold level for Λ = 1 against the others. The results are: for f1 = 0.1N {Λ = 0.5, F = 4.68; Λ = 2/3, F = 7.34; Λ = 1.5, F = 5.5; Λ = 2, F = 8.83} and for f1 = 0.4N {Λ = 0.5, F = 9.73; Λ = 2/3, F = 9.42; Λ = 1.5, F = 4.08; Λ = 2, F = 7.69}. We observe that for both force conditions, the condition Λ = 1 produces less variance than other values.

# **5 Discussion**

Our study shows that the elliptical motion of particles can indeed decrease the amount of vibration amplitude that is needed to give the illusion of a button click. Therefore, we can suggest that lateral displacement helps to increase the internal lateral stresses that are first stored when participants touch the plate, before they are released by inverting the direction of the particles' motion. Moreover, we have not seen many differences between the values for Λ ≤ 1 on the detection threshold, and this threshold doesn't change with the force level; the condition Λ = 0 has been tested but the averaged detection threshold has been found to be higher than the capability of the device. Therefore, we hypothesize that the value of Λ does not change the lateral stresses when in the range 0.5 ≤ Λ ≤ 2/3. We also show that the variance is minimal for Λ = 1. Therefore, this gives rise to the optimal value at which the plate should operate. However, for now we do not give an explanation for this specific behaviour; a modelling that takes into account the intermittent contact could be used for that aim [13].

**Fig. 4.** Experimental results: 50% module threshold, compute for different Λ and different activation pressure levels : a) f1 = 0.4N, b) f1 = 0.1. The error bars, the whisker boxes and the horizontal bars show respectively the min. and max. values, their interquartile range and the median value and corresponding variance c) and d) respectively.

### **6 Conclusion**

In this paper we used two modes simultaneously to produce the simulation of a click on a static finger. To create a simulation with the lowest detection threshold as well as the lowest variance between participants, an equal amount of normal and lateral displacement should be set. This finding will be useful to design and control new tactile interfaces.

**Acknowledgement.** This work is supported by IRCICA (Research Institute on software and hardware devices for information and Advanced communication, USR CNRS 3380).

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **A Proposal and Investigation of Displaying Method by Passive Touch with Electrostatic Tactile Display**

Hirobumi Tomita1(B), Satoshi Saga2, Shin Takahashi1, and Hiroyuki Kajimoto3

1 University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, Japan<EMAIL_ADDRESS><EMAIL_ADDRESS>2 Kumamoto University, 2-39-1, Kurokami, Chuo-ku, Kumamoto, Japan<EMAIL_ADDRESS>3 The University of Electro-Communications, 1-5-1, Chofugaoka, Chofu, Tokyo, Japan<EMAIL_ADDRESS>
**Abstract.** In the field of tactile displays, electrostatic force displays have been developed for presenting tactile stimulation on a screen. However, with the conventional electrostatic force displays, the user cannot feel the stimulation without rubbing on the display. In this paper, to solve this problem, we propose a new method to present a tactile sensation without moving a finger by creating a small space between an electrode and an insulating surface. Moreover, we evaluate the perceived threshold of the proposed display through the experiment.

**Keywords:** Electrostatic tactile display *·* Passive touch *·* Conductive thread

# **1 Introduction**

In the field of tactile displays, electrostatic force displays have been developed for presenting tactile stimulation on a screen. The electrostatic tactile display consists of a high-voltage generator, an electrode, and an insulator. In this method, the user touches the insulating film on the electrode. If high voltage is applied to the electrode, he/she feels the tactile stimulation by rubbing on the film. In the 1950s, Mallinckrodt et al. discovered that a phenomenon such as vibration-like friction was generated through an insulating film by electrostatic force [5]. Related surveys, dealing with parameters like the effect of input frequencies, waveforms, or amplitude modulations were conducted by many researchers [1,3,4,7,9].

However, in this method, the tactile stimulation is not presented when the user stops moving the finger, and there is little research on displaying the method by passive touch with an electrostatic tactile display. Pyo et al. developed the tactile display, which generates electrovibration and mechanical vibration by using the electrostatic parallel plate actuator [6]. This device enables passive touch with electrostatic force, however, it requires a voltage higher than 1 kV.

In this paper, we propose a new method to present tactile sensation without moving a finger; we also implement and evaluate a device applying the proposed method. If passive tactile presentation becomes possible by this technique, the range of applications of tactile presentation using electrostatic force will be widely expanded.

# **2 Implementation of Device for Passive Touch with Electrostatic Tactile Display**

The electrostatic tactile display consists of a high-voltage generator, an electrode, and an insulator. In the conventional method, an insulating film covers a flat electrode shown in Fig. 1 (a), and a user moves his/her finger on the insulator. When a high voltage is applied to the electrode, the dielectric polarization is generated in the finger. In this state, the electrode applies an attractive static force to the finger. When the user slides his/her finger on the display, s/he feels a tactile stimulation. However, when the user does not slide his/her finger on the display, s/he cannot perceive the tactile sensation.

Here, we proposed a method that employs a parallel-plate electrostatic actuator mechanism [2,6] for displaying tactile stimulation to a stationary finger without significantly changing the configuration of the electrostatic tactile display. Figure 1 (b) shows the schematic sketch of the parallel-plate electrostatic actuator to generate mechanical vibration. The stationary electrode and nonstationary electrode are set in parallel. When high-voltage is applied to the stationary electrode, the attractive electrostatic force occurs on the non-stationary electrode. We considered that a passive tactile presentation was possible by adopting Pyo et al.'s method [6]. The finger is considered as the non-stationary electrode. We created a small space by using cylindrical electrodes, which have a small diameter, such as a conductive thread or wire, as shown in Fig. 1 (c, d). In this paper, we used 120 μm diameter stainless steel fiber as a conductive thread. These threads were arranged lengthwise without any gaps between them on PET resin (Fig. 2 (b)). The width of these threads arranged is 25 mm to allow enough space for one finger. All of these threads are conductive. They are connected to the high voltage generator. Then, we covered them with a polyvinylidene chloride insulation film. This insulating film is inexpensive and easy to replace. The thickness of this insulating film is 11 µm.

We considered that the slight vibration of the insulating film and finger surface became possible by applying periodic high-voltage from the high-voltage generator to the cylindrical electrode, and the vibration could be perceived by the finger. The high-voltage generator shown in Fig. 2 (c) is developed by Kajimoto laboratory (The University of Electro-Communications, Tokyo). The device includes an mbed LPC1768 microcontroller, which controls the output voltage at a maximum of 600 V by modifying the firmware.

**Fig. 1.** (a) Conventional electrostatic tactile display: An insulating film is placed on a flat electrode. (b) Schematic sketch of the parallel plate electrostatic actuator [2,6]. (c) Proposed method: A soft insulating film is placed on a nonplanar electrode, such as a conductive thread or wire. (d) Enlarged view of finger surface with the proposed method.

**Fig. 2.** (a) Overview of our display device, (b) a conductive thread arranged lengthwise and (c) the high-voltage generator.

# **3 Evaluation Experiment of Proposed Method**

We held an evaluation experiment to explore whether it is possible to feel tactile stimulation with the proposed method. Thresholds of the voltage amplitude at which the user feels the tactile sensation for various input waveforms were investigated. In this manner, we can confirm the input waveform that displays tactile sensation and how the voltage threshold changes in each input waveform.

#### **3.1 Evaluation Method Using Method of Constant Stimuli**

In this experiment, we investigated the perception thresholds of each voltage amplitude by using constant stimuli. As shown in Fig. 3 (a), we prepared three waveforms: sine wave, square wave, and a delta function that occurs periodically. Then we changed the voltage amplitude of these waveforms and asked the participants to answer whether they felt the stimulus presented on the tactile display. After collecting evaluation results from the participants, we calculated the perceived tactile sensation (vertical axis of Fig. 3 (b)) and psychometric function with voltage amplitude on the horizontal axis. This function is calculated by approximating the logistic curve shown in the following equation to the perceived tactile sensation in each voltage amplitude.

$$R = \frac{1}{1 + ae^{-bv}}\tag{1}$$

R shows the perceived tactile sensation, v shows a voltage amplitude, a and b show constant parameters. By using this function, we estimate the threshold voltage at which the user feels the tactile sensation. In this experiment, a voltage where the psychometric function went across 50% of the perception rate was set as a threshold. An example of a psychometric function obtained from the averaged results of all participants was shown in Fig. 3 (b). By calculating the thresholds of every waveform and frequency of the input waves, we can estimate the trend of the voltage thresholds.

**Fig. 3.** (a) Three prepared waveforms and (b) an example of a psychometric function.

#### **3.2 Outline of Evaluation Experiment**

We conducted an evaluation experiment of constant stimuli with 9 university students (2 female), 22 to 25 years old.

We prepared the display device, a personal computer, a polyethylene terephthalate (PET) resin stand, and a pressure sensor (FSR406) for the experiment. The display device is put on the PET resin stand to isolate it from the floor and to make it easier to place the finger. The pressure sensor is installed under the PET resin stand (Fig. 2 (a)) to measure the pressing force of the finger. We fed back the value of the pressure sensor to the participants via an LCD monitor and instructed them to keep it constant.

We prepared three waveforms and 10 types of dominant frequencies from 10 Hz to 630 Hz of each waveform. Furthermore, we prepared 9 voltage amplitudes from 200 V to 600 V for calculating the voltage threshold, and each condition was displayed ten times. These conditions were selected in random order. In addition, we investigated whether the voltage threshold changes by the difference of the pressing force of the finger. The participants were requested to perform the experiments under the conditions of a weaker pressing force (between 35 gf and 44 gf) and a stronger one (between 75 gf and 84 gf). From these conditions, participants conducted 5400 trials evaluation (3 waveforms *×* 10 frequencies *×* 9 amplitudes *×* 2 pressure *×* 10 trials for each condition).

First of all, we explained the outline of the experiment and obtained written informed consent for participation in the study (based on ethical guidelines of University of Tsukuba) from all participants. For masking the external sound, white noise was applied to participants using a headphone during the experiment. We instructed them to place their right index finger on the display device and to keep the pressure condition during the experiment. After a waveform was inputted to the display device, participants were instructed to select either "feel something" or "feel nothing" by using the installed keyboard. After participants finished the evaluation with all the waveforms, we instructed them to keep the other pressure condition and repeat the evaluation. 5-minute breaks were taken three times during the experiment. After the experiment, we collected the participants' answers and calculated a voltage threshold for each input waveform.

#### **3.3 Results and Discussion**

The results of the evaluation experiment are shown in Fig. 4 and Fig. 5. The horizontal axis of these graphs shows the frequency of the input waveform. The blue line indicates the condition of a weaker pressure, and the orange line indicates the condition of a stronger pressure in these graphs. Figure 4 shows the number of participants who perceived some tactile sensation for each input waveform. Figure 5 shows the result of the voltage thresholds in each input waveform. These thresholds are derived from the intersection of the 50% line and the logistic curve. From Fig. 4 and Fig. 5, we observed that most of the participants could perceive the displayed tactile sensation in all conditions of square waves and limited frequency conditions of sine waves or delta functions. From the result, the

**Fig. 4.** Results of the number of participants who feel tactile sensation for each input waveform. (Color figure online)

**Fig. 5.** Results of the threshold of voltage amplitude for each input waveform.

availability of the tactile display without moving a finger is confirmed by using our proposed method. Furthermore, the voltage thresholds are observed to be less than 500 V in most waveforms. In addition, we found the following two characteristics; the evaluation results differed according to the pressing force, and the outline of the graph differed according to the waveform.

Regarding the voltage thresholds, the results of 350 V to 500 V are very high compared to general tactile display devices. However, it is possible to reduce the voltage thresholds by making the insulation film thinner, and the advantage of this method is that there is almost no current flow and power at the display side. Then, it is possible to install many tactile displays or large area tactile displays. Therefore, even if a high voltage is required, we consider that this method of tactile presentation is effective for some specified applications.

Regarding the pressing force, we confirmed that the evaluation result differs depending on the pressing force. In Fig. 4, there is a significant difference in the number of participants who perceived the tactile sensation between the two pressing conditions in the sine waves and delta function. From Fig. 4, we observe that the number of participants who perceived the tactile sensation is larger under the weak pressure condition than that of under the strong pressure condition in most sine waves and delta functions. Most participants perceived the tactile sensation under both conditions in the square waves. In Fig. 5, there was a significant difference between the two pressure conditions at all waveforms. Furthermore, the averaged voltage threshold appeared to be smaller under the weak pressure condition than that under the strong pressure condition in all of the square waves, some sine waves and some delta function. We considered that the factor of this result is based on the ratio between the pressing force of the finger and a just noticeable difference. We focused on Weber's law and showed it in the following equation.

$$\frac{\Delta R}{R} = \text{Const} \tag{2}$$

where ΔR is a just noticeable difference in the tactile sensation, and R is a pressing force in this evaluation experiment. From Eq. (2), the more pressing force, the more a difference of perceived amount is needed. Therefore, we considered that the threshold of input voltage amplitude had to be high for decreasing a just noticeable difference when the pressing force was strong.

Regarding the trend difference of graphs between waveforms (Fig. 4), when square waves were displayed, most of the participants could perceive the displayed tactile sensation. However, the participants could not perceive the tactile sensation in some frequencies of sine waves and delta functions. As seen in Fig. 5, the graphs in the case of sine and square waves are downward convex with the minimum value around 200 Hz. The graph in the case of the delta function is gentle and upward right. The reason for the shapes is considered to be related to the frequency components of each input waveform and the frequency responses of human mechanoreceptors. Vardar et al. investigated how displayed waveforms affect haptic perceptions of vibration under electrostatic tactile display [8]. Then they observed that the participants were more sensitive to square wave stimuli than sine-wave stimuli for a dominant frequency lower than 60 Hz. Furthermore, they discussed that a low dominant frequency square wave still contains highfrequency components that stimulate the Pacinian channel. Our experimental results were similar to the results of theirs; voltage thresholds are higher in the case of sine wave stimuli than in the case of square one under 40 Hz frequencies. In the case of the delta function, these waves have many frequency components, however, they are smaller in amplitude than the other two waveforms. This could be because the waveforms did not have sufficient amplitude to stimulate mechanoreceptors at low dominant frequencies.

#### **4 Conclusions**

In this paper, we proposed a method that can display tactile stimulation without moving a finger with an electrostatic tactile display, and we conducted an evaluation experiment. The proposed method is based on the vibration of a thin insulating film, which is realized by creating a small space between the electrode and the insulator. We use a conductive thread as an electrode and place a plastic film on the conductive thread. We explored whether the user can feel tactile stimulations under several conditions. We collected the evaluation results of the participants under the conditions of waveforms, frequencies and pressing forces, and calculated the voltage threshold. From the results, the possibility of a tactile display without moving a finger is confirmed by using our proposed method.

Besides, we also revealed the following fact: the evaluation results differed according to the pressing force and the outline of the graph differed according to the waveform. Regarding the pressing force of the finger, we considered that the ratio between the pressing force and the suction force induce the difference of perceptibility. Regarding the difference in the shapes of the graphs, we considered that the frequency components of the input waveform and the frequency responses of mechanoreceptors is the reason.

In the near future, we plan to clarify the modeling of tactile perception using our proposed method and the difference in the width of tactile expression compared with the conventional method. Further, we will perform more precise experiments to observe the physical phenomenon between the surface of the finger and the display.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Sensing Ultrasonic Mid-Air Haptics with a Biomimetic Tactile Fingertip**

Noor Alakhawand1,2(B) , William Frier3, Kipp McAdam Freud1,2, Orestis Georgiou3, and Nathan F. Lepora1,2

 Department of Engineering Mathematics, University of Bristol, Bristol, UK<EMAIL_ADDRESS>Bristol Robotics Laboratory, Bristol, UK Ultraleap Ltd., Bristol, UK

**Abstract.** Ultrasonic phased arrays are used to generate mid-air haptic feedback, allowing users to feel sensations in mid-air. In this work, we present a method for testing mid-air haptics with a biomimetic tactile sensor that is inspired by the human fingertip. Our experiments with point, line, and circular test stimuli provide insights on how the acoustic radiation pressure produced by the ultrasonic array deforms the skin-like material of the sensor. This allows us to produce detailed visualizations of the sensations in two-dimensional and three-dimensional space. This approach provides a detailed quantification of mid-air haptic stimuli of use as an investigative tool for improving the performance of haptic displays and for understanding the transduction of mid-air haptics by the human sense of touch.

**Keywords:** Tactile sensors *·* Biomimetic *·* Mid-air haptics

#### **1 Introduction**

Ultrasonic phased arrays can generate haptic sensations in mid-air. They focus acoustic radiation pressure in space, which deflects the skin to induce tactile sensation [1]. To evaluate whether the array is producing the desired haptic sensations, we need to understand how focal points of pressure interact with compliant skin to cause it to deform. In this paper, we propose a method for sensing mid-air haptics with a biomimetic tactile fingertip inspired by the human sense of touch. Using the data obtained from the sensor, we are able to visualize the different patterns produced by the haptic array.

Efforts to measure the haptic output from a phased ultrasonic array range from quantitative to qualitative. Quantitative methods include microphones to measure the sound pressure level of the generated focal points [1,8], directly measuring the ultrasonic output of the system without considering its interaction with other material. On the other hand, to quantitatively consider the interaction of the sensations with skin-like materials, Laser Doppler Vibrometry (LDV), a tool commonly used for non-contact vibration measurement, can give insight on how the haptic stimuli would interact with human skin at high frequencies [2]. Alternatively, qualitative methods include pulsed schlieren imaging, which was used to visualize the pressure field produced by a focal point as it interacts with external materials [5]. Additionally, by projecting the focal points onto the surface of an oil bath, it can be used to visualize the patterns generated by the haptic array [6]. New research has used a microphone-based tactile sensor array to evaluate the vibrations of its surface due to ultrasonic haptic sensations [9], highlighting the potential of tactile sensors for testing the output of a haptic system.

In this work, we propose a method to sense and evaluate mid-air haptics using the TacTip, a biomimetic tactile fingertip. The TacTip is biologically inspired by glabrous (hairless) human skin, which has an intricate morphology of layers, microstructures, and sensory receptors that contribute to its functions [3,10]. We present a method for analyzing mid-air haptic sensations with a tactile sensor, allowing us to quantitatively test ultrasonic arrays with a method inspired by the human sense of touch.

### **2 Experimental Setup and Method**

This work aims to develop a method for testing mid-air haptics with a biomimetic tactile sensor. We carried out experiments with the tactile fingertip mounted on a robot arm and an ultrasonic array (Fig. 1).

**Fig. 1.** The TacTip, a biomimetic tactile sensor (left): the flat-tipped model used in this study; the skin of the TacTip with 127 inner nodular pins (middle); and the experimental setup with the tactile sensor mounted on a robot arm to collect data over the ultrasonic phased array (right).

#### **2.1 Biomimetic Tactile Sensor**

The TacTip (Fig. 1, left panel) is a biomimetic tactile sensor developed at the Bristol Robotics Laboratory [3,10], based on the structure of glabrous skin. The human fingertip has *dermal papillae* where the dermis interdigitates with *intermediate ridges* in the epidermis. These ridges and papillae focus strain from the skin surface down to mechanoreceptors within the dermis. The TacTip mimics this structure with an outer rubber-like skin which connects to inner nodular pins (Fig. 1, middle panel). As the soft sensor interacts with objects, its skin deforms and the nodular pins transmit surface strain into inner mechanical movements, similar to human skin. An internal camera tracks the movement of its artificial papillae, making it possible to detect the shear deformation of the skin. The sensor has been used in many tasks in robot touch such as object exploration and slip detection [10]. The TacTip is manufactured using dual-material 3D printing, which prints both the sensor's plastic base and the soft rubber-like material for the skin. This allows for low-cost and rapid prototyping of different designs as well as its integration with robotic grippers and hands. Additionally, the design of the TacTip is modular, allowing for different tips to be used, such as varying the shape or texture of the skin or varying the layout of the nodular pins [10]. The tip of the sensor can be filled with gel to affect its compliance or be left unfilled. Since this is the first time the TacTip has been used to detect small forces on the order of millinewtons, we needed a more compliant tip; after testing tips with these variations, we found the flat-tipped TacTip without gel (Fig. 1, left panel) to be more sensitive, and thus suitable for this work.

#### **2.2 Ultrasound Phased Array**

To generate the mid-air haptic stimuli for the experiments, we used the Ultrahaptics Evaluation Kit (UHEV1) from Ultraleap. The array has a 16 by 16 grid of ultrasonic transducers which operate at 40 kHz to generate focal points in midair, with an update rate of 16 kHz. The device is accompanied by software which allows us to modulate these focal points so that they can be felt by users [1] and to generate various shapes and textures [6].

#### **2.3 Experiment**

We used a 6-DOF robotic arm (ABB IRB120) to move the tactile sensor over the haptic array. The robot arm moved the sensor in 10 mm increments over an 80 mm by 80 mm grid at a height of 200 mm above the haptic array. At each position, 30 frames were captured from the camera to image the TacTip's inner nodular pins at 30 fps. This was done for a focal point generated by the array, as well as two shapes (a line and a circle). The shapes were generated by the array using Amplitude Modulation (AM) and Spatiotemporal Modulation (STM), to see whether the sensor distinguishes between these two standard modulation techniques. AM generates focal points in the path of the desired pattern and modulates their intensity over time, while STM generates one focal point and moves it rapidly along the path.

**Fig. 2.** Analysis method. We capture an image from the tactile sensor as it interacts with the mid-air haptic stimulus (1) and extract each pin position (2). Voronoi tessellation is generated with pin positions as the center point for each cell (3); the change of area of each cell compared to an unstimulated sensor, ΔA, is used as a measure of the stimulus intensity (4). This is repeated for readings over a grid (5). Gaussian Process Regression combines the data sets to produce detailed visualizations (6).

#### **2.4 Analysis**

In this work, we developed an analysis method to sense mid-air haptics with a biomimetic tactile fingertip (Fig. 2). The images captured from the tactile sensor as it interacts with the mid-air haptic stimulus were processed to find the positions of the nodular pins at each time step. Then we used the pin positions to generate a bounded Voronoi tessellation, shown by Cramphorn et al. to transduce a third dimension to the sensor data [4]. Voronoi tessellation partitions a plane based on the distance between points on that plane; each point along an edge is equidistant from two points, and each vertex is equidistant from at least three points. The areas of the cells give us information for tactile perception; increasing areas indicate a compression of the skin. Thus, the areas of each cell in the Voronoi tessellation were compared with a data set in which the sensor was not stimulated, and the difference between the two areas, ΔA, was used as a measure of the intensity of the stimulus as felt by the sensor. This was done for every time step, and then averaged over the 30 frames of data. The process was repeated for readings in a grid over the haptic display to populate a twodimensional plane. Then we trained a Gaussian process regression (GPR) model for the measured intensity, represented by ΔA (using the MATLAB function *fitrgp* with the default squared exponential covariance function). The output values were then scaled between zero and one, to represent the relative intensity of the stimulus as felt by the tactile sensor.

**Fig. 3.** The mid-air haptic shapes, line and circle, as felt by the biomimetic tactile sensor. The Amplitude Modulation (AM) cases use four focal points modulated with a 200 Hz sine wave. The Spatiotemporal Modulation (STM) cases use one focal point moving at a repeat frequency of 100 Hz along the path of the shape.

# **3 Results**

In this work, we used a biomimetic tactile fingertip to sense mid-air haptics, to develop a method for testing the output of a haptic display. We measured the response of the tactile sensor to a focal point of pressure generated by an ultrasonic haptic array as well as two haptic shapes, each generated by Amplitude Modulation (AM) and Spatiotemporal Modulation (STM).

#### **3.1 Sensing Mid-Air Haptics on a Two-Dimensional Grid**

The experiments showed that the biomimetic tactile fingertip used in this study, the TacTip, is able to sense the mid-air haptic stimuli produced by the ultrasonic phased array using our developed method (Fig. 2). The focal points of pressure generated by the ultrasound caused the skin-like surface to deform, expanding the areas of the cells in the Voronoi tessellation, allowing us to identify the location and intensity of contact. Voronoi tessellation was a valuable tool in transducing a third dimension in the data, which allowed us to visualize the sensations produced by the ultrasonic array.

Our analysis methods enables us to produce detailed visualizations of the mid-air haptic sensations (Fig. 3), allowing us to distinguish between different patterns produced by the ultrasonic array. Additionally, the variation in the strength of the focal point can be clearly seen. The visual representation of the focal point shows that it creates a localized region of increased displacement (Fig. 2, lower right panel). The point is much stronger in the center, and then decreases in intensity as you move radially outwards. This is similar for the other shapes; the center path of the shape has increased intensity, which decreases as you move away (Fig. 3).

The visualizations produced by our method allow us to compare the shapes generated by the ultrasonic array using different modulation techniques. We see that the tactile sensor is able to distinguish the four focal points that make up the amplitude modulated line (Fig. 3, top left panel). A user of the ultrasonic array would not distinguish the points as the distance between them is small [1], and so it creates the illusion of a continuous line. The sensor can discriminate between the points because our analysis method is measuring the deformation of the tactile sensor's surface, which would correspond to the deformation of the user's skin rather than their perception of the sensation. On the other hand, the spatiotemporally modulated line (Fig. 3, lower left panel) is felt as a continuous line by the sensor. The focal point used to generate the line is moved along its path at very small increments. The distance between the points in this case is too small to be distinguished by the sensor, making the output more similar to how a user would sense the stimulus.

#### **3.2 Sensing a Focal Point in Three-Dimensional Space**

In the previous section, we presented our results when sensing various shapes over a two-dimensional grid. In this section, we extend our method to sense mid-air haptic stimuli in three dimensions. When a person interacts with the haptic array to sense the shapes it generates, they naturally move their hand around the display surface, which includes moving their hands up and down as they process the sensations they feel. Thus, understanding how the generated sensations vary with height is important to determine whether the desired effect is being produced and to check that there are no undesired artefacts. We repeated the data collection process described earlier for the point stimulus at different heights over the array, at 10 mm increments. This results in a three-dimensional grid on which we applied our presented analysis method.

This experiment allows us to see how the shape is sensed over a threedimensional surface, looking at how the shape varies by height. The focal point is generated by the array at a specific height in space; however, there are still sensations at other points due to the interaction of the ultrasonic waves [8]. The point stimulus is sensed by the tactile fingertip as an elongated spheroid, with a localized region of increased intensity (Fig. 4). It appears the lower the intensity of the stimulus felt, the more elongated it is. As the sensor moves away from the center height, the stimulus becomes fainter.

**CROSS-SECTIONAL VIEWS OF A FOCAL POINT IN 3D**

**Fig. 4.** The three-dimensional view of a focal point as felt by the biomimetic tactile sensor presented as cross sections at z = 0 (left), which corresponds to 200 mm above the array; y = 0 (middle); and x = 0 (right).

#### **4 Discussion and Conclusion**

Tactile sensors can further our understanding of the human sense of touch. Our experiments have shown that we can use a biomimetic tactile fingertip to sense the mid-air haptic stimuli produced by an ultrasonic phased array, providing insights on the deformation of the skin-like material of the TacTip due to ultrasonic mid-air haptic sensations. This allows us to produce detailed visualizations of the sensations produced by the device. Using our analysis methods, we were able to see the difference between shapes that are amplitude modulated versus spatiotemporally modulated by the ultrasonic array. Additionally, we were able to sense and visualize a focal point in three-dimensional space, providing insights on how the focal point varies by height.

The visualizations of the stimuli produced in our study are similar to those in other works which use alternative methods. For example, Laser Doppler Vibrometry was used to measure the deformation of skin-like material due to ultrasonic mid-air haptics [2]; it measured the high frequency vibrations (50 Hz and above) of the skin surface, and the root mean square (RMS) of the deformation was used to visualize the sensations. While we do not measure the high frequency vibrations, we get similar results. This could indicate that the data we collect is similar to the RMS of the skin deformation. Additionally, our three-dimensional measurements of the focal point look very similar to simulations of the same stimulus [8]. The elongated spheroid felt by the sensor looks like the higher values of acoustic field pressure in the simulation, suggesting that the tactile fingertip is able to sense the ultrasound when it crosses a threshold pressure.

This work has provided insights into the measurement of haptic stimuli, but it has areas for improvement. At this point, we have measured the intensity of the stimulus without relating it to a specific physical value. Further work is planned to determine the relationship of the measured stimulus intensity to the skin deformation, which would allow us to compare our results with other quantitative experiments. Additionally, we do not measure the skin deformation at high frequencies. While the results we get are similar to those which use vibrometry, studying the vibrations of the artificial skin could determine whether the sensor does behave similarly to human skin. One approach is to modify the sensor with a higher frame rate camera which could allow us to see the highfrequency deformations of the skin; another approach would be to add another high-frequency tactile sensing modality to the TacTip [7].

Our work has shown promising results for sensing mid-air haptics with a biomimetic tactile fingertip. The developed approach could be used as an investigative tool for evaluating and improving the capabilities of haptic displays. The insights gained from this work could also be used to investigate human perception. In the future, we could apply our methods to intelligent exploration of the haptic stimuli. This could allow us to develop an autonomous robotic system that is able to feel and interact with the sensations similar to how a person would explore mid-air haptic stimuli.

**Acknowledgements.** We thank A. Stinchcombe, K. Aquilina, and J. Lloyd for their help. N. Alakhawand was supported by an EPSRC CASE award with Ultraleap. N. Lepora was supported by an award from the Leverhulme Trust on 'A biomimetic forebrain for robot touch' (RL-2016-39).

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Soft-Wearable Device for the Estimation of Shoulder Orientation and Gesture**

Aldo F. Contreras-Gonz´alez(B) , Jos´e Luis Samper-Escudero , David Pont-Esteban , Francisco Javier S´aez-S´aez , Miguel Angel S´ ´ anchez-Ur´an , and Manuel Ferre

Centre for Automation and Robotics (CAR) UPM-CSIC, Universidad Polit´ecnica de Madrid, 28006 Madrid, Spain<EMAIL_ADDRESS>https://www.car.upm-csic.es/?portfolio=exoflex-flexible-exoskeleton

**Abstract.** This study presents the development of a wearable device that merges capacitive soft-flexion and surface electromyography (sEMG) sensors for the estimation of shoulder orientation and movement, evaluating five natural movement gestures of the human arm. The use of Time Series Networks (TSN) to estimate the arm orientation, and a pattern recognition method for the estimation of the classification of the gesture are proposed. It is demonstrated that it is possible to know the orientation of the shoulder, and that the algorithm is capable of recognising the five gestures proposed with two different configurations. The study is performed on people who reported healthy upper limbs.

**Keywords:** Soft robotics · Wearable sensors · UpperLimb · sEMG.

#### **1 Introduction**

There have been many attempts to identify the movement of the human body in a virtual way by monitoring the behaviour of the extremities for haptic interfaces [7], teleoperation tasks [1] and assistive and rehabilitation devices [4,16]. Robots increase the number of repetitions performed in a rehabilitation session, thus improving patient morale and motivation [24]. In recent years, rehabilitation devices use sEMG as main source of feedback [15] for control [8].

Several sEMG techniques are used for the identification and classification of movements [3], some of the most relevant being the Detrended Fluctuation Analysis (DFA) for the identification of low-level muscle activation [19], the sEMG signal decomposition into Motor Unit Action Potential Trains (MUAPTs) [18], the Tunable-Q factor Wavelet Transform (TQWT) based algorithm proposed for the classification of physical actions [2], and Convolutional Neural Network (CNN), recently confirmed as as a powerful tool for the classification of operator movements [25]. These methods, combined with appropriate signal filtering techniques [5], are useful for estimating the movement of the human body. Data fusion [13] using sEMG sensors is widely used in rehabilitation. Movement recognition algorithms generally combine sEMG signals with the Inertial Measurement Unit (IMU) [11], or with force sensors [10]. There are particular cases where flexion sensors [22] are used to avoid the accumulated error on the measurement.

This paper focuses on the development of a soft-compressive jacket with a network of soft-flexion sensors, merged with sEMG sensors attached to it for movement detection of the upper limbs. This device allows the user to quickly start estimating shoulder orientation without the need for prior calibration.

# **2 Materials and Methods**

Using a configuration of seven one-axis sensors, as in previous work [21], it is possible to obtain 95% of the variance of the principal components for the shoulder gestures. The configuration proposed in this paper places only an array of four flexion sensors in the intermediate positions due to the fact that they provide flexion measurements in two axes.

The array of four flexion sensors *Sx* (being 'x' the sensor number) was placed over a compression jacket (see Fig. 1). The capacitive flexion sensors are the *Two Axis Sensor of Bendlabs* [12] and its operation is explained and well detailed in [20]. The sensors have been attached to a compression jacket by sewing two small rigid pieces which hold and guide the sensor in the arm movement direction and to neglect properties such as wrinkles and stretching. The first support (FxA) (see Fig. 1b) holds the sensor in a fixed position while the second one (FxB) allows it to slide inside it and guides it over the arm (see Fig. 1c).

**Fig. 1.** Soft Sensor Device. sEMG location (1a): Trapezius Descendens (CH3), Deltoideus Medius (CH2) and Pectoralis major (CH1). Markers location: one over the shoulder Acromion bone, two on the arm (1b) and two vertically over the base (1c).

The sensor arrangement allows shoulder movement to be measured in a sixdegree-of-freedom (DoF) work-space where arm rotation around its longitudinal axis is not included, this measurement is converted into two angle XY and YZ given by the conversion of the position of the ground truth. sEMG sensors [23], are allocated following the recommendations of Surface Electromyography for the Non-Invasive Assessment of Muscles (SENIAM) [9]. The electrodes are placed on the user (as shown on Fig. 1); then, the user puts on the compression jacket over the electrodes (not shown on Fig. 1b nor Fig. 1c). The design of this device allows the deformation and stretching of the fabric to be disregarded due to the small rigid pieces, in addition to not limiting the user's mobility on daily tasks.

The gestures performed were simplified to cover the natural range of arm movement [14] for daily tasks, and were assigned a number for further identification: **1**. Abduction/Adduction of the shoulder until the arm reaches 120◦ inclination; **2**. Flexion/Extension of the shoulder from 0◦ to 120◦; **3**.  Closing/Swing drill movement of the arm inwards from 0◦ to 120◦; and, **5**. Opening/Swing drill movement, starting with a flexion of 120◦ to 0◦. The method developed in this study was evaluated in four healthy subjects; tests were spread over three different days to avoid exhaustion of the muscles. Each subject performed a total of five repetitions of each of the five gestures, continuously and without interruptions. Participants' ages ranged from 24 to 30 years old. All the gestures made by the subjects were performed in a chair facing a screen.

#### **2.1 Data Acquisition**

To start data collection, the sEMG sensors and the flex sensor compression jacket are placed on the subject's right arm. Then, OptiTrack [17] markers are located as shown in Fig. 1, in order to obtain the real pose of the subject's arm.

Both flexion and sEMG sensors are connected to a custom acquisition board based on the LAUNCHXL-F28379D development board. On the one hand, the sEMG sensors provide an amplified, rectified and integrated analogical signal

**Fig. 2.** The EMG signals and angles of the user's movements (first box on the left) were acquired using visual feedback generated by the interface on the Jetson Nano, which also stores this data. The OptiTrack system stores the position of the markers. In the end, both files are merged into one.

(AKA the EMG's envelope), which is obtained by the micro-controller at a rate of 1kHz. On the other hand, the flexion sensors communicate with the microcontroller via I2C protocol at a frequency of 200 Hz.