0 Research

# Design requirements to improve laparoscopy via XR

Ezequiel R. Zorzal
INESC-ID, Instituto Superior Técnico
ICT/UNIFESP, Universidade Federal de São Paulo
Brazil e-mail<EMAIL_ADDRESS>Maurício Sousa
INESC-ID e-mail<EMAIL_ADDRESS>University of Toronto
Canada  Pedro Belchior
INESC-ID e-mail<EMAIL_ADDRESS>Instituto Superior Técnico
Portugal
João Madeiras Pereira
INESC-ID e-mail<EMAIL_ADDRESS>Instituto Superior Técnico
Portugal  Nuno Figueiredo
Colorectal Surgery e-mail<EMAIL_ADDRESS>Hospital
Lusíadas  Lisbon
Portugal  Joaquim A. Jorge
INESC-ID e-mail<EMAIL_ADDRESS>Instituto Superior Técnico
Portugal

###### Abstract

Laparoscopic surgery has the advantage of avoiding large open incisions and
thereby decreasing blood loss, pain, and discomfort to patients. However, on
the other side, it is hampered by restricted workspace, ambiguous
communication, and surgeon fatigue caused by non-ergonomic head positioning.
We aimed to identify critical problems and suggest design requirements and
solutions. We used user and task analysis methods to learn about practices
performed in an operating room by observing surgeons in their working
environment to understand how they performed tasks and achieved their intended
goals. Drawing on observations and analysis from recorded laparoscopic
surgeries, we have identified several constraints and design requirements to
propose potential solutions to address the issues. Surgeons operate in a dimly
lit environment, surrounded by monitors, and communicate through verbal
commands and pointing gestures. Therefore, performing user and task analysis
allowed us to understand the existing problems in laparoscopy better while
identifying several communication constraints and design requirements, which a
solution has to follow to address those problems. Our contributions include
identifying design requirements for laparoscopy surgery through a user and
task analysis. These requirements propose design solutions towards improved
surgeons’ comfort and make the surgical procedure less laborious.

Human-centered computingVisualizationVisualization design and evaluation
methods

Introduction

Laparoscopy - a minimally invasive surgical approach to the abdomen and pelvis
- allows all types of resectional, staging and diagnostic procedures,
especially when ultrasounds, CT scans and MRI scans cannot provide enough
clinical information. Compared to open surgery, its major benefits are smaller
and fewer incisions, with less tissue damage and pain following the operation,
which in turn results in lower demands for analgesics and shorter hospital
stays inducing a faster recovery time. However, Laparoscopic surgery affords
diminished tactile feedback and also the loss of direct visual contact with
organs, as surgeons rely on an endoscopic camera, which captures and feeds
images onto a display [2]. This results in a limited and restrictive
experience compared to open surgery, as the surgeon dexterity and haptic
feedback are reduced by the laparoscopic rigid instruments [12, 1].

Furthermore, it presents other challenges that can induce surgeon discomfort
by non-ergonomic positioning and hamper the surgical workflow. A paramount
problem in minimally invasive surgery is hand-eye coordination, as surgeons
have to look at screens placed outside the field of operation, which results
in discomfort [1], affecting the surgeon efficiency due to a visual-motor axis
disconnection. One cannot look simultaneously at their instruments, hands and
surgical field. To become proficient, high level training and experience is
required to adapt to this condition, as extra mental effort must be applied
[12]. In addition, almost all these display screens are limited in sense that
they do not support techniques to improve visual collaboration with the rest
of the surgical team [5].

Walczak et al. [26] evaluated whether the positioning of the monitor has an
impact on laparoscopic performance. Results show the time taken to perform the
task was shorter when the screen was placed downwards, which corresponded to
the position participants most preferred. This position allows users to flex
the head at 15 to 45 degrees below eye level, which is the most comfortable
position, as looking down improves eye lens accommodation and reduces eye
weariness and headaches. [14] also evaluated the effect of the monitor,
conducting a study to determine whether wearing a head-mounted display (HMD)
improves task performance, or at least reduces muscle fatigue, comparing it in
an operating scenario against the use of a traditional monitor. While the HMD
made for smoothmotion, but performance in general was not clearly superior.

Batmaz et al. [1] compared visualisation, direct vision, 2D fish-eye,
undistorted and 3D stereoscopic views, and studied their performance effects
on a laparoscopic training exercise where subjects were to place a small
object in the centre of five targets, in a specific order. The results show
that 3D stereoscopic imaging does not have any performance edge over 2D, with
objects being selectively coloured, facilitating depth perception. However,
straight ahead monitor positioning did show performance benefits, as subjects
took less time to perform the tasks as they felt less neck strain and more
comfort. Prescher et al. [24] also assessed 3D viewing, and conducted a study
to determine whether stereoscopic 3D displays with glasses improved
performance in trainees. The 3D display seemingly reduced the time taken to
complete the test as well as the number of dropped objects, while being
generally preferred by subjects.

Kihara et al. [10] developed a virtual reality (VR) system for use in real-
world operation, combining a HMD with a 3D endoscope to provide the surgeon
with high quality imaging right in front of him. The 3D HMD gives the feel of
an open surgery and allows the visualization of content regardless of head
position, while direct vision is allowed by lowering the angle of sight.
Finally, [7] worked on a mixed reality headset which integrates the image from
the laparoscopic camera, a navigation system and diagnostic imaging,
complemented by an audio feedback system. VR can be successfully used for
laparoscopic training curriculum as it not only helps to reduce the physical
and mental workload on surgeons but also improves their surgical performance
in operation theaters. It can also be used to train surgeons to cope up with
other technical problems encountered during surgery simultaneously [9].

Unlike previous work, we aim to identify potential design challenges in
laparoscopic procedures and suggest design requirements to approach these
problems. In this paper, we present a study on the conditions in which the
laparoscopic surgeons perform their tasks. We analyze areas of improvement and
contribute design requirements to appropach these issues.

##  1 Method

We conduct our work according to the methodology used by [17]. The examples
presented in this paper come from the Champalimaud Foundation in Lisbon, PT
[27]. At that site, we observed laparoscopic surgeries on one of the
Foundation’s surgical rooms and talked to surgeons before and after the
procedures, who explained what was about to happen or what took place. Also,
during the surgery, nurses provided insight into the several stages of the
surgery, or what was happening at that time. We also had the opportunity to
ask questions of the surgeons at appropriate moments during the surgery.
Besides that, we video recorded for further analysis in addition to our field
notes. A total of five (5) laparoscopic surgeries were observed for a total of
approximately 10 hours.

The observed cases included the surgical team is composed of mostly male
members, with only one female surgeon in it, and their ages range from 34 to
42 years of age. During surgery, there are at least six people involved in the
procedure:A head surgeon, who coordinates the entire procedure, 1 or 2
assistant surgeons, who mostly observe but also participate in parts of the
surgery, a nurse solely responsible for passing the surgeons tools they may
require throughout the operation, an anesthetist keeping track of the
patient’s vital signs, a nurse supporting the anesthetist and a circulating
nurse. Additionally, a senior surgeon may come in and serve as an advisor,
providing insight and making remarks about what is being seen on camera.

We collected images and videos using smartphones and tablets, as well as the
notes made during the observations. The inductive bottom-up approach to data
analysis was used in which the authors analyzed their field notes and videos.
The findings and the discussion are presented in the subsequent section.

##  2 Results and Discussion

Performing user and task analysis allowed us to better understand the existing
problems in the procedure of laparoscopy, while identifying several
constraints and design requirements, which a solution has to follow in order
to address those problems. For the following, we will discuss the design
requirements identified through the analysis. Problems statements,
requirements, and the proposed design solutions also are summarized in Table
1.

Table 1: Problem statements, design requirements and design solution for a prototype that supports laparoscopy Problem Statements |  Design Requirements |  Design Solution
---|---|---
Visualizing the laparoscopic video during extended periods of time is exhausting for the neck. |  The solution should allow the user to adopt more comfortable neck postures instead of forcing the user to look to the side to see what the other surgeons are seeing. |  Following display: The laparoscopic video follows user head movement, so users can look around and assume a neck posture that is more comfortable for them.
Current interactions surgeons have, such as pointing or consulting patient data, require them to let go of their tools, which interrupts the procedure. |  Surgeons should have hands-free interactions in order to operate in an uninterrupted fashion. |  Hands-free interaction: Every interaction is either done with the head or using the feet.
Browsing patient data interoperatively takes too long because it requires to call in an assistant, who browses the images for the surgeon. |  Users should be able to look at patient data by themselves, without interrupting and adding extra time to the surgery. |  Patient data image browser: users can look to the side to see and browse magnetic resonance images from the patient.
Users may have to move around the patient in order to adopt better positions to hold their tools. |  Interaction using the foot should not rely on pedals, as these would need to be moved around to cope with user movement. |  Foot browsing: Users can use the foot to navigate the patient images, rotating it on its heel to change images faster or slower.
Pointing is unclear and ambiguous: different users have different interpretations of where a surgeon is pointing at. |  Users should be able to point precisely and understand where other users are pointing at, regardless of position in the operating room. |  Pointing reticle: users can place a reticle on both laparoscopic video and patient images, controlling it with head motion. This cursor is visible on other users’ headsets.
Surgeons operate in a crowded area, as they are usually very close together. |  Augmented space should present information close to the surgeon to prevent it from appearing intersected with a colleague. |  Close quarters: Positioning of interface elements is no further than at an elbow’s reach.

###  2.1 Following display

Laparoscopy is an intensive process, not just mentally but physically as well.
The procedure is already very demanding in itself due to surgeons having to
expend extra mental effort thanks to a lack of hand-eye coordination that is
caused by indirect visualization (Fig. 1). That effort extends to the physical
plane when we consider that they have look at the screen all the time, which
places a continuous strain on their necks. Laparoscopy currently faces the
glaring problem of monitor positioning. During surgery, screens are usually
placed far away and at an uncomfortable angle, causing neck and eye strain
over the course of a surgery, especially if it drags for longer periods of
time. Given this, it was important to allow the surgeons some freedom in how
they want to see the video, and then the video, while visible, should follow
user head movements so users do not have to reposition it in the augmented
space, should they feel the need to assume another posture with the neck.

Figure 1: All doctors look to the same screen and, sometimes, this means
having to assume an uncomfortable position. Visualizing the laparoscopic video
during extended periods of time is exhausting for the neck.

The surgeons have to look at screens placed outside the field of operation,
which results in discomfort [1], affecting the surgeon’s efficiency due to a
disconnect between the visual and motor axis, because the surgeon cannot look
at the instruments or hands and the field of surgery simultaneously. To be
successful, more training is required to adapt to this condition, as extra
mental effort must be applied [12]. In addition, almost all display screens
are limited in sense that they do not support techniques to improve
communication and visual collaboration with the rest of the surgical team [5,
4].

Muratore et al. [21] suggest for the future, the ideal display system would be
a 3D high definition image HMD, citing the comfort of looking at the
endoscopic image in any preferred head position, improving ergonomics and
reducing neck strain. The use of an HMD is also seen as beneficial in the
sense that it alleviates equipment clutter in the operating room. It is
further noted the usefulness of individualized image manipulation features
like zooming, which allows each surgeon to see the endoscopic video in the way
they find most comfortable. Also, the works of [26, 14, 1, 10] seem to support
the usage of a HMD for laparoscopic surgery, with the video following the
user’s head movements. With this, users can assume their preferred head
position instead of being forced to look sideways in order to see the video.

###  2.2 Hands-free interaction

From our field observations, we noticed that surgeons place down their tools
to perform some secondary tasks, which interrupts the procedure (Fig. 2). We
suggest that projects avoid this type of situation, with a completely hands-
free approach, using both head gaze and foot movement as sources of input.

Figure 2: Surgeons can’t look at their hands, thus losing hand-eye
coordination.

Other studies [11, 7] looked at using head movements, and gaze to select
targets, especially when the content follows head movements [6]. A more
elaborate approach takes the form of the eye gazing [3, 25], which was well-
received by users, but may not transition well onto the surgical operating
field: these controls would have to be displayed continuously and right in
front of the user, unlike in the presented works, which could be distractive
for users, but more importantly, it would take valuable space from the HMD’s
already limited field of view. In terms of feet, two different approaches
emerge: using a foot pedal as a means to activate a selected control [7] and
using foot movement to select and activate controls [20]. After comparing the
two, we conclude using foot movement would be a more flexible choice, as it
does not rely on extra hardware that is situated in a given position in space.

###  2.3 Patient data image browser and foot browsing

Consulting patient data intra-operatively, such as MRIs and computed
tomographies, may be unfeasible. On the one hand, the data are extensive and
may require some time to identify the required set. On the other, surgeons
must abandon the operating table to sit at a computer and browse the desired
images. The surgeon then gives directions on where to look and when to stop
while an assistant handles the computer. Surgeons usually do not browse the
images themselves when sterilized. Indeed each interaction with non-sterile
equipment such as the keyboard and mouse would entail a new sterilization
procedure, wasting additional resources. This has been discussed in [22, 13,
8]. This is a design opportunity for AR as discussed in [28].

In the work of [21], the authors emphasize the importance of the HMD, stating
preoperative imaging could be individually manipulated through its use, as
well as grant surgeons extra comfort by allowing them to see the laparoscopic
image regardless of head positioning. Hands-free interaction is again
considered, with the suggestion of using foot pedals instead. Also discussed
is the issue of paradoxical imaging, which occurs when the camera faces the
surgeon, causing movements with the tool to appear inverted compared to the
hand movements. However, in an interview with the surgeons of the Champalimaud
Foundation, this did not appear to be an issue, since the surgeon can move
around the operating table, which ensures the camera always faces the opposite
direction. This freedom to move around also impacts the practicality of using
foot pedals to ensure hands-free interaction, as the authors suggest, as the
surgeon would have to either have the same pedals on multiple sides, or move
the pedals around. In this case, exploring foot movement, as proposed by [20],
could be more useful.

###  2.4 Pointing reticle and close quarters

Although they are close quarters, surgeons also currently face problems in
communication. In fact, according to the inquired surgeons, just as they
complain about difficulty in maintaining proper posture, so do they complain
about not being able to let other surgeons know what part of the video they
are pointing at, or to understand what others are pointing at as well. Several
works have approached this by looking at proxemics [18] and embodied vision
[19].

Figure 3: Doctor point at the screen to communicate. Communication is unclear
and ambiguous: different users have different interpretations of where a
surgeon is pointing at.

The instructor can point at the screen for the other surgeons to understand
what anatomical structure he/she is referring to and use gestures for others
to understand the motion of the tools better and envision cutting lines.
Sometimes, pointing can also be done with the tools themselves, but even
though it may be effective, it is not always correct because if both hands are
occupied, it implies letting go of a structure to point with the tool or
asking someone else to hold it. Additionally, pointing from a distance with
the hand is ambiguous at best, as there is no clear way to tell where
precisely a surgeon is pointing at, as can be seen in Fig. 3.

For Prescher et al. [23], the impact of the pointer in a real operating
scenario may be lessened because target selection is not random but rather
contextual, meaning that the following targets may be located through the
description of what is being displayed on-screen. Also, the pointer is
embedded in the laparoscope. Thus the camera must be displaced to move the
pointer, causing the view plane to change and forcing the surgeon to readjust
to the new perspective, losing perceived depth. Therefore, it would be more
useful if the cursor moved independently of the camera, controlled via
gestures or head tracking for a hands-free approach, as suggested by 3D
interactions performed above a table [15, 16].

##  3 Conclusion

This paper presents the requirements and design solutions for laparoscopy
through a user and task analysis. During a task analysis, we map out the
sequence of activities surgeons go through and the actions required to achieve
that goal. Drawing on observations and analysis of video recordings of
laparoscopic surgeries, we identify several constraints and design
requirements, which a solution will have to follow in order to address those
problems. These requirements propose to inform the design solutions towards
improved surgeons’ comfort and make the surgical procedure less laborious.

## Acknowledgments

This work was supported by national funds through FCT, Fundação para a Ciência
e a Tecnologia, under project UIDB/50021/2020. The authors would like to thank
the Champalimaud Foundation for its collaboration and support in the
development of the study.

## References

  * [1] A. U. Batmaz, M. De Mathelin, and B. Dresp-Langley.  Seeing virtual while acting real: Visual display and strategy effects on the time and precision of eye-hand coordination.  PLoS ONE, 12(8):1–18, 2017. doi: 10 . 1371/journal . pone . 0183789
  * [2] S. Bernhardt, S. A. Nicolau, V. Agnus, L. Soler, C. Doignon, and J. Marescaux.  Automatic localization of endoscope in intraoperative CT image: A simple approach to augmented reality guidance in laparoscopic surgery.  Medical Image Analysis, 30:130–143, 2016. doi: 10 . 1016/j . media . 2016 . 01 . 008
  * [3] A. Esteves, E. Velloso, A. Bulling, and H. Gellersen.  Orbits: Gaze Interaction for Smart Watches using Smooth Pursuit Eye Movements.  Proc. UIST ’15, (1):457–466, 2015. doi: 10 . 1145/2807442 . 2807499
  * [4] Y. Feng, K. Li, A. Semsar, H. McGowan, J. Mun, H. R. Zahiri, I. George, A. Park, A. Kleinsmith, and H. M. Mentis.  Communication cost of single-user gesturing tool in laparoscopic surgical training.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19. ACM, New York, NY, USA, 2019. doi: 10 . 1145/3290605 . 3300841
  * [5] H. Fuchs, M. A. Livingston, R. Raskar, et al.  Augmented Reality Visualization for Laparoscopic Surgery.  Federation bulletin / Federation of State Medical Boards of the United States, 50:316–322, 2006. doi: 10 . 1007/BFb0056282
  * [6] A. Grinshpoon, S. Sadri, G. J. Loeb, C. Elvezio, S. Siu, and S. K. Feiner.  Hands-free augmented reality for vascular interventions.  In ACM SIGGRAPH 2018 Emerging Technologies, SIGGRAPH ’18. ACMGrinshpoon, New York, NY, USA, 2018. doi: 10 . 1145/3214907 . 3236462
  * [7] J. Jayender, B. Xavier, F. King, et al.  A Novel Mixed Reality Navigation System for Laparoscopy Surgery, vol. 2878.  Springer International Publishing, 2018. doi: 10 . 1007/b93811
  * [8] R. Johnson, K. O’Hara, et al.  Exploring the potential for touchless interaction in image-guided interventional radiology.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, p. 3323–3332. Association for Computing Machinery, New York, NY, USA, 2011. doi: 10 . 1145/1978942 . 1979436
  * [9] K. Kantamaneni, K. Jalla, et al.  Virtual reality as an affirmative spin-off to laparoscopic training: An updated review.  Cureus, Aug. 2021. doi: 10 . 7759/cureus . 17239
  * [10] K. Kihara, Y. Fujii, H. Masuda, et al.  New three-dimensional head-mounted display system, TMDU-S-3D system, for minimally invasive surgery application: Procedures for gasless single-port radical nephrectomy.  International Journal of Urology, 19(9):886–889, 2012. doi: 10 . 1111/j . 1442-2042 . 2012 . 03044 . x
  * [11] T. Kim, B. Saket, A. Endert, and B. MacIntyre.  VisAR: Bringing Interactivity to Static Data Visualizations through Augmented Reality.  2017\.
  * [12] M. Leite, A. F. Carvalho, P. Costa, et al.  Assessment of laparoscopic skills performance: 2D versus 3D vision and classic instrument versus new hand-held robotic device for laparoscopy.  Surgical Innovation, 23(1):52–61, 2016. doi: 10 . 1177/1553350615585638
  * [13] D. Lopes, F. Relvas, S. Paulo, Y. Rekik, L. Grisoni, and J. Jorge.  Feetiche: Feet input for contactless hand gesture interaction.  In The 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry, VRCAI ’19. ACM, New York, NY, USA, 2019. doi: 10 . 1145/3359997 . 3365704
  * [14] S. K. Maithel, L. Villegas, N. Stylopoulos, S. Dawson, and D. B. Jones.  Simulated laparoscopy using a head-mounted display vs traditional video monitor: An assessment of performance and muscle fatigue.  Surgical Endoscopy and Other Interventional Techniques, 2005. doi: 10 . 1007/s00464-004-8177-6
  * [15] D. Mendes, F. Fonseca, B. Araùjo, A. Ferreira, and J. Jorge.  Mid-air interactions above stereoscopic interactive tables.  In 2014 IEEE Symposium on 3D User Interfaces (3DUI), pp. 3–10, 2014\. doi: 10 . 1109/3DUI . 2014 . 6798833
  * [16] D. Mendes, F. Relvas, A. Ferreira, and J. Jorge.  The benefits of dof separation in mid-air 3d object manipulation.  In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, VRST ’16, p. 261–268. Association for Computing Machinery, New York, NY, USA, 2016. doi: 10 . 1145/2993369 . 2993396
  * [17] H. M. Mentis, A. Chellali, and S. Schwaitzberg.  Learning to see the body: Supporting instructional practices in laparoscopic surgical procedures.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, p. 2113–2122. ACM, New York, NY, USA, 2014. doi: 10 . 1145/2556288 . 2557387
  * [18] H. M. Mentis, K. O’Hara, A. Sellen, and R. Trivedi.  Interaction proxemics and image use in neurosurgery.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, p. 927–936. Association for Computing Machinery, New York, NY, USA, 2012. doi: 10 . 1145/2207676 . 2208536
  * [19] H. M. Mentis and A. S. Taylor.  Imaging the body: Embodied vision in minimally invasive surgery.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, p. 1479–1488. Association for Computing Machinery, New York, NY, USA, 2013. doi: 10 . 1145/2470654 . 2466197
  * [20] F. Müller, J. McManus, S. Günther, M. Schmitz, M. Mühlhäuser, and M. Funk.  Mind the Tap.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, pp. 1–13, 2019. doi: 10 . 1145/3290605 . 3300707
  * [21] C. S. Muratore, B. A. Ryder, and F. I. Luks.  Image display in endoscopic surgery History of surgical endoscopy.  pp. 349–356, 2007.
  * [22] K. O’Hara, G. Gonzalez, et al.  Touchless interaction in surgery.  Communications of the ACM, 57(1):70–77, 2014. doi: 10 . 1145/2541883 . 2541899
  * [23] H. Prescher, D. E. Biffar, C. A. Galvani, J. W. Rozenblit, and A. J. Hamilton.  Surgical navigation pointer facilitates identification of targets in a simulated environment.  In Simulation Series, vol. 46, pp. 246–252, 2014.
  * [24] H. Prescher, D. E. Biffar, J. Rozenblit, and A. J. Hamilton.  The comparison of high definition versus stereoscopic display on standardized fundamental laparoscopic skill procedures.  Summer Computer Simulation Conference, SCSC 2014, Part of the 2014 Summer Simulation Multiconference, SummerSim 2014, July 6, 2014 - July 10, 46(10):346–351, 2014.
  * [25] E. Velloso, M. Wirth, et al.  AmbiGaze: Direct Control of Ambient Devices by Gaze, 2016. doi: 10 . 1145/2901790 . 2901867
  * [26] D. A. Walczak, D. Pawełczak, P. Piotrowski, et al.  Video display during laparoscopy – where should it be placed?  Videosurgery and Other Miniinvasive Techniques, 1:87–91, 2015. doi: 10 . 5114/wiitm . 2014 . 47434
  * [27] E. R. Zorzal, J. M. C. Gomes, M. Sousa, P. Belchior, P. G. da Silva, N. Figueiredo, D. S. Lopes, and J. Jorge.  Laparoscopy with augmented reality adaptations.  Journal of Biomedical Informatics, 107:103463, July 2020. doi: 10 . 1016/j . jbi . 2020 . 103463
  * [28] E. R. Zorzal, M. Sousa, D. Mendes, R. K. dos Anjos, D. Medeiros, S. F. Paulo, P. Rodrigues, J. J. Mendes, V. Delmas, J.-F. Uhl, J. Mogorrón, J. A. Jorge, and D. S. Lopes.  Anatomy studio: A tool for virtual dissection through augmented 3d reconstruction.  Computers & Graphics, 85:74 – 84, 2019. doi: 10 . 1016/j . cag . 2019 . 09 . 006


#  EarDA: Towards Accurate and Data-Efficient Earable Activity Sensing

Shengzhe Lyu, Yongliang Chen, Di Duan, Renqi Jia, Weitao Xu1 * Corresponding
author. Department of Computer Science, City University of Hong Kong
Hong Kong SAR, China

###### Abstract

In the realm of smart sensing with the Internet of Things, earable devices are
empowered with the capability of multi-modality sensing and intelligence of
context-aware computing, leading to its wide usage in Human Activity
Recognition (HAR). Nonetheless, unlike the movements captured by Inertial
Measurement Unit (IMU) sensors placed on the upper or lower body, those motion
signals obtained from earable devices show significant changes in amplitudes
and patterns, especially in the presence of dynamic and unpredictable head
movements, posing a significant challenge for activity classification. In this
work, we present EarDA, an adversarial-based domain adaptation system to
extract the domain-independent features across different sensor locations.
Moreover, while most deep learning methods commonly rely on training with
substantial amounts of labeled data to offer good accuracy, the proposed
scheme can release the potential usage of publicly available smartphone-based
IMU datasets. Furthermore, we explore the feasibility of applying a filter-
based data processing method to mitigate the impact of head movement. EarDA,
the proposed system, enables more data-efficient and accurate activity
sensing. It achieves an accuracy of 88.8% under HAR task, demonstrating a
significant 43% improvement over methods without domain adaptation. This
clearly showcases its effectiveness in mitigating domain gaps.

###### Index Terms:

IMU, earable, domain adaptation

##  I Introduction

Earable devices, which significantly enhance people’s everyday listening
experience, have experienced a noticeable surge in recent years. With
technological advancements, many newly launched devices have been integrated
with various sensors (e.g., AirPods Pro 2, Bose QuietControl 30 and Sony
WF-1000XM5), giving rise to an emerging research area known as earable
sensing. Compared to other sensing modalities, earable sensing holds a
significant advantage as it seamlessly integrates into users’ daily lives.
Consequently, it has attracted researchers’ attention and has been shown to be
effective in various tasks, including health care [1, 2], speech enhancement
[3, 4] and activity recognition [5, 6, 7].

Existing works in this realm are either developed based on dedicated or
commercial off-the-shelf (COTS) devices. The first type requires specially
designed devices, such as eSense [8] and OpenEarable [9]. Despite their
effectiveness, the requisite highly customized hardware significantly
restricts their practicability. On the other hand, with the emergence of
sensor-rich earable products like Apple Airpods, an increasing number of
researchers have proposed plausible solutions based on these COTS devices [3,
10]. However, the obtained sensory data is highly heterogeneous across
different manufacturers, and there are no unified datasets applicable to all
devices, which suggests a dilemma where developing new sensing applications
often necessitates collecting the whole dataset from scratch. As such, how to
develop an accurate earable sensing methodology in a data-efficient manner
based on COTS devices remains an unsolved ad-hoc question.

To fill this research gap, in this paper, we propose EarDA, an earable human
activity recognition (HAR) system based on domain adaption. To the best of our
knowledge, we are the first to propose such a system that can utilize
knowledge learned from other sensing modalities (e.g., smartphones) with
abundant training data to enable effective earable-based HAR with limited
data. However, directly applying well-trained frameworks from other modalities
to earable data is impracticable due to the significant domain gap that exists
between these two modalities (e.g., smartphones are always carried in pockets
while earable devices are attached to the user’s head). Thereafter, we
specially design a domain adaptation training framework to leverage HAR
features learned from public datasets with substantial sensory data to boost
the performance on new earable devices. It effectively reduces the required
human labor intensity on collecting abundant earable training data due to the
absence of unified datasets. In addition, we encounter another challenge that
head movements introduce distinct signal variations, which overwhelm the
informative signal for activity recognition. To mitigate this, we carefully
investigate the obtained signals in different scenarios and devise a filter-
based solution that effectively mitigates the resulting effects.

We have built a prototype of EarDA atop Apple AirPods Pro 2, and it is
evaluated on four common daily activities (walking, upstairs, standing, and
jogging) based on IMU measurements. The training is conducted with a small
volume of earable data (i.e., 5 minutes) with the assistance of four public
smartphone-based IMU datasets. It recognizes activities at an accuracy of 89%,
with a significant increment of 43%, compared with cases only training with
the public dataset and adopting the model on earables without domain adaption,
demonstrating the proposed system’s effectiveness. Incorporating a filter-
based data processing method to mitigate head movement enhances the overall
accuracy by 4%, highlighting the system’s ability to cooperate with the head
interference. In short, we make the following contributions:

  * •

We take the first step toward understanding the domain gap existing in the
public wearable datasets, especially smartphone-based datasets and IMU data
from COTS earables.

  * •

We propose EarDA with novel techniques that mitigate the domain gap between
COTS earable devices and public datasets and influences introduced by
irregular head movements.

  * •

We conduct extensive evaluations and showcase the superiority of the proposed
system. The overall accuracy achieves 88.8%, and the ablation study
demonstrates the effectiveness of the proposed components.

##  II Feasibility Study

###  II-A Impact of Location Diversity

Figure 1: IMU measurements by smartphone (left pocket) and earable (head) for
different activities and head movements.

The disparity among sensory data collected by different modalities mainly
stems from the different sensing locations. To better illustrate such
discrepancies, we collect IMU data from a smartphone placed in the pocket of
the trousers and a naturally-worn earable device when performing different
actions. We visualize their sensory data in Figure 1. It can be observed from
Figure 1(a) and Figure 1(b) that there are notable differences in both of the
acceleration and gyroscope measurements. The smartphone’s sensor records the
movements and vibrations of the lower body, reflecting the motion patterns and
gestures associated with leg and hip movements. On the other hand, the earable
device captures a combination of head and body movements, providing a
different perspective on overall body motion. For example, the maximum range
of acceleration during jogging is observed to be twice that of earables, as
shown in Figure 1(d, e). When measuring acceleration signals in the lower
body, the amplitude of the signals tends to be larger. Nevertheless, apart
from all the challenges, the presence of similar patterns, such as periodic
peaks in both domains, underscores the potential to transfer the knowledge
across domains.

###  II-B Impact of Head Movement

Head movement is unavoidable when performing daily activities. However, the
introduced signal fluctuations pose extra challenges in sensing tasks. As
shown in Figure 1(c), even when the lower body and upper body are experiencing
periodical movements during walking, unpredictable interference appears in IMU
signals collected by the ear-worn device due to the head movement. In
addition, the impact of head movements is still obvious while the users tried
their best to control their heads during the data collection, as shown in
Figure 1(b). This can significantly affect the HAR tasks because the noise and
variations distort the IMU data patterns, making it more difficult to
accurately classify the user’s activity. Fortunately, due to the filtering
function of the upper body, the frequency components of earable IMU data
always fall in a certain range, suggesting the usage of filter-based data
processing methods might help to mitigate the interference from head
movements.

##  III System Design

Figure 2: System overview.

###  III-A System Overview

The overview of EarDA is shown in Figure 2. We design an effective
adversarial-based domain adaptation module that can extract domain-invariant
features across the source domain with substantial data (i.e., public
datasets) and the target domain with limited data (i.e., earable data).
Moreover, we specially design a filter-based data pre-processing pipeline to
mitigate the influence caused by head movements.

###  III-B Domain-invariant Feature Extraction

##### Feature Extractor

A bi-directional LSTM model is designed to construct a generalized feature
extractor due to its architecture, which is adapted to capturing the complex
temporal dynamics and dependencies inherent in sequential data. To boost the
model’s robustness against device orientation and data heterogeneity, the
acceleration and gyroscope magnitudes are calculated as inputs. It serves as a
pre-processing step to abstract away the device’s orientation, providing a
more orientation-invariant representation of the motion. This approach
enhances the model’s generalization ability across different devices and
domains. It can be seen in Figure 1 that the acceleration readings are always
steadier than gyroscope readings during the same activity. The values derived
from the accelerometer and gyroscope always fall on different ranges as the
gyroscope is more sensitive to movement. To mitigate the impact, accelerations
are normalized through the division by gravity, after which the range
differences between acceleration readings and gyroscope readings can be
narrowed.

##### Domain Adaptation

As mentioned previously, the normal LSTM classifier trained on the source
domain cannot be directly applied to the target domain. As a result, the model
should retain the capability to extract domain-invariant features regardless
of which domain the input data falls on. In this work, we present a domain
adaptation module that takes substantial data from the source domain and
limited data from the target domain as input. Then, two distinct pathways are
derived from the feature extractor, one label predictor, and one domain
classifier. The label predictor is tasked with the objective of predicting the
correct activity label for the input IMU sequence. In contrast, the domain
classifier, aided by the gradient reversal layer, attempts to determine the
domain of the input data.

Inspired by [11], we integrate a Gradient Reversal Layer (GRL) to assist in
learning domain-invariant features of the two domains. Specifically, during
the forward pass, the GRL acts as an identity function, allowing data to pass
through unchanged. However, during the backward pass, it reverses the gradient
sign before passing it back to the preceding layers. This effectively
encourages the LSTM layers to learn representations that are indistinguishable
from the domain classifier. Consequently, this process leads to the
minimization of domain discrepancy, resulting in the capability of domain-
invariant feature extraction. More formally, the loss function for the network
utilizing GRL can be written as:

$\displaystyle E(\theta_{f},\theta_{y},\theta_{d})$
$\displaystyle=\sum_{\begin{subarray}{c}i=1..N\end{subarray}}L_{y}\left(G_{y}(G_{f}(x_{i};\theta_{f});\theta_{y}),y_{i}\right)$
$\displaystyle-\lambda\sum_{i=1..N}L_{d}\left(G_{d}(G_{f}(x_{i};\theta_{f});\theta_{d}),y_{i}\right)$
$\displaystyle=\sum_{\begin{subarray}{c}i=1..N\end{subarray}}L_{y}^{i}(\theta_{f},\theta_{y})-\lambda\sum_{i=1..N}L_{d}^{i}(\theta_{f},\theta_{d})$

Here, $G_{f},G_{y},G_{d}$ are the functions representing the feature
extractor, label predictor, and domain classifier over the parameters of
$\theta_{f},\theta_{y},\theta_{d}$, respectively. $\lambda$ is a trade-off
parameter that controls the importance of the domain adaptation loss relative
to the label prediction loss, which is selected to be 0.3 in our work. $L_{y}$
is the label predictor loss and $L_{d}$ is the domain classifier loss. This
loss function aims to minimize the label prediction error while also
encouraging the feature extractor to learn domain-invariant features. The
presence of the GRL is implied in the negative sign before the domain
classifier loss term, which reflects the gradient reversal during the
backpropagation process.

###  III-C Mitigation of Head-movement

The head movement interference has been a challenge in performing HAR tasks
with earables because the signals recorded by ear-worn devices are usually
unpredictable with the interference added. One approach to mitigate the impact
of head movements on HAR tasks is to employ signal processing like applying
filters. It must be admitted that eliminating the motion caused by head
movements from raw IMU data is not easy, as the frequencies of body or head
movements spread over a big range and overlap each other in a specific range.
As studied in [12], human activity frequencies are always between 0 and 20 Hz
and 98% of the frequency spectrum is below 10 Hz. Nonetheless, due to the
filtering function of the body structure, the signals recorded on the head are
usually smaller, always below 5.8 Hz, presented in [13]. This leads to the
potential of mitigating head movement by low-pass filtering. We observed
similar frequency component distribution by comparing the frequency spectrum
under different kinds of head movements. As illustrated in Figure 4, the
dominant frequency components of gyroscope data are concentrated in low-
frequency bins (e.g., below 5 Hz). However, when the same activity is
accompanied by more rigorous head movements, noticeable lobes appear at higher
frequency ranges. Considering previous literature on regular human activities
[12, 13] and the observed sensory data, we empirically devise a low-pass
filter with a cut-off frequency of 5 Hz. It can effectively filter out
unwanted disturbances occupying high-frequency ranges, including those
introduced by erratic head movements, while simultaneously retaining
informative signals related to body movements.

Figure 3: Frequency spectrum of gyroscope measurements collected by earables
during jogging.

Figure 4: Confusion matrix on the test set with and without applying domain
adaptation.

##  IV Evaluation

###  IV-A Implementation

To better understand the impact of sensor locations and head movements on the
performance of HAR tasks, we collected a dataset with Apple Airpods Pro 2. The
Airpods are chosen because: 1) the APIs are provided to access accurate and
real-time 6-axis motion data captured from the accelerometer and gyroscope
embedded in the earbuds, 2) IMU signals from each earbud in a pair are fused
together automatically in API to represent the motion signals of the head, and
3) it is widely utilized in HAR tasks on earable devices as it is one of the
most popular COTS earable products. The sampling rate of Airpods was set to
the highest value allowed by API, which is 25 Hz.

A two-layer bidirectional LSTM model with a hidden size of 16 is implemented
as the feature extractor. The model takes two-dimensional inputs with a
sequence length of 100 from each domain. Fully connected layers with ReLU
activation are employed for both the label predictor and the domain
classifier. The gradient reversal layer is only implemented as part of the
domain classifier. The model was trained with a batch size of 32 for 200
epochs on the computer with an NVIDIA GeForce RTX 2080Ti GPU.

###  IV-B Experimental Setup

####  IV-B1 Earable Data Collection

Figure 5: Three evaluated head movements in the experiment. TABLE I: Performance of the model with different head movements after mitigation of head movements. Head movement | Slight | Random | Roll | Yaw | Pitch | Average
---|---|---|---|---|---|---
Metric | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
Walking | 0.90 | 0.82 | 0.76 | 0.83 | 1.00 | 0.63 | 0.92 | 0.92 | 0.88 | 0.92 | 0.82 | 0.82
Upstairs | 0.77 | 0.81 | 0.89 | 0.79 | 0.62 | 0.76 | 0.80 | 0.89 | 0.86 | 0.92 | 0.78 | 0.81
Standing | 0.92 | 0.96 | 1.00 | 1.00 | 1.00 | 0.96 | 1.00 | 0.86 | 1.00 | 0.86 | 0.97 | 0.94
Jogging | 1.00 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.99
Overall | 0.90 | 0.89 | 0.91 | 0.91 | 0.90 | 0.84 | 0.93 | 0.92 | 0.94 | 0.93 | 0.89 | 0.89
TABLE II: Performance of the model with different head movements. Head movement | Slight | Random | Roll | Yaw | Pitch | Average
---|---|---|---|---|---|---
Metric | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
Walking | 0.78 | 0.86 | 0.76 | 0.80 | 0.86 | 0.39 | 0.87 | 0.67 | 1.00 | 0.59 | 0.80 | 0.73
Upstairs | 0.95 | 0.82 | 0.83 | 0.78 | 0.52 | 0.69 | 0.52 | 0.68 | 0.50 | 0.67 | 0.64 | 0.72
Standing | 0.98 | 0.99 | 1.00 | 1.00 | 1.00 | 0.88 | 1.00 | 0.70 | 1.00 | 0.74 | 0.99 | 0.91
Jogging | 1.00 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.99
Overall | 0.93 | 0.92 | 0.90 | 0.90 | 0.84 | 0.74 | 0.85 | 0.76 | 0.88 | 0.75 | 0.86 | 0.84

During the data collection with Airpods, the participant was asked to perform
four different activities (walking, walking upstairs, standing, jogging) while
wearing the Airpods as usual. The total time for each activity is 14 minutes,
constituting 4 minutes of slight head movement, 4 minutes of random head
movements, and 2 minutes for each of rolling, yawing, and pitching. During the
collection period, the participant was first asked to control their head in a
steady state for 4 minutes while doing all four activities, leading to slight
head movement interference on captured data. After that, the participant
performed all activities casually for another 4 minutes. Head movements are
unavoidable in daily life, so random interferences are added to IMU data
collected. Rolling (tilting), yawing (waving), and pitching (nodding) are
chosen as specific head movements, as depicted in Figure 5. We want to
evaluate the impact of each kind of head movement on the performance of cross-
domain HAR tasks separately.

The collected dataset is segmented into non-overlapping windows, each
encompassing sequences of 4 seconds. Each sequence corresponds to a single
activity label. In consideration of the variability introduced by device
orientation, we compute the magnitudes of acceleration and gyroscope data to
serve as a more stable input feature set for the deep learning model.
Subsequently, the processed 840 data samples are randomly partitioned into
distinct subsets for model training (10%), validation (10%), and testing
(80%). As a result, the training set contains only 84 samples while the
testing set contains 672 samples.

####  IV-B2 Public Datasets

The source domain refers to smartphone-based publicly available IMU datasets.
Thus, four datasets are chosen as they are widely utilized in previous works
[14, 15]. The four publicly available datasets comprise IMU signals gathered
from individuals engaging in various activities while wearing smartphones.
These datasets vary not only in the model of the smartphones used but also in
the range of activities performed by the participants. Additionally, some of
these datasets are further diversified by sensor locations. By incorporating
these datasets into the experiment, the adaptability and effectiveness of the
algorithm across diverse domains can be facilitated.

##### MotionSense

The MotionSense dataset [16] was collected by accelerometer and gyroscope
sensors of Apple iPhone6s in 50 Hz. There are six different activities
designed for the dataset (walking, walking upstairs, walking downstairs,
standing, jogging and sitting), and the smartphones were kept in the
participants’ pockets of trousers.

##### HHAR

The Heterogeneity Human Activity Recognition (HHAR) dataset [17] contains IMU
signals captured between 50-200 Hz for six different activities (walking,
walking upstairs, walking downstairs, standing, biking and sitting). There are
six models of smartphones in the dataset (three models of Samsung Galaxy and
one model of LG), carried by the users around their waist.

##### UCI HAR

The UCI HAR dataset [18] was obtained from a waist-mounted Samsung Galaxy S2
smartphone. 6-axis IMU signals were collected under six different activities
(walking, walking upstairs, walking downstairs, standing, lying down and
sitting) at 50 Hz.

##### Shoaib

The Shoaib dataset [19] consists of the motion data of seven daily activities
(walking, walking upstairs, walking downstairs, standing, jogging, biking and
sitting). During data collection, five Samsung Galaxy SII models were placed
in five body positions. The sampling rate is 50 Hz.

We randomly extract samples corresponding to four activities of interest:
walking, walking upstairs, standing, and jogging. A balanced dataset
comprising 10,000 instances is constructed, ensuring an equal representation
of 2,500 samples for each activity to prevent class imbalance during the
training process. In line with the sampling frequency of the IMU signals from
the Airpods, the data is down-sampled to 25 Hz. The pre-processing steps are
consistent with those applied to the Airpods IMU signals, involving the
segmentation of the data into non-overlapping sequences of 100 samples each.
The prepared dataset is then divided, allocating 80% for training, 10% for
validation, and the remaining 10% for testing. Finally, the training set
comprises 8,000 samples, which is 100 times larger than the training samples
collected from earables.

###  IV-C Overall Performance

Table I summarizes the system’s overall performance on the earable data test
set, including both accuracy and F1-score. Figure 4(a) further visualizes the
confusion matrix. The proposed framework demonstrates an exceptional ability
to classify all four activities, regardless of the original sensor domain. The
model achieves an average accuracy of 88.8% and an F1-score of 0.89. While
excelling in classifying jogging activities with a 99% accuracy, the model
exhibits lower performance for upstairs activities, sometimes misclassified as
less-strong activities like walking.

###  IV-D Effectiveness of domain adaptation

To evaluate the effectiveness of the proposed domain adaptation component,
keeping all the settings the same as EarDA, we discard the domain classifier
and only use the 8,000 data samples from public datasets to train a
classifier. The trained model is then applied to test the AirPods samples. As
shown in Figure 4(b), the model’s performance shows a significant decline,
achieving an accuracy of only 46%. This decline in performance clearly
indicates that the model trained on public IMU datasets cannot be directly
applied to earable IMU data due to a substantial domain gap across different
sensor positions. In Figure 4(b), we present the confusion matrix derived from
the model’s predictions. It can be observed that all jogging samples were
wrongly classified as less intense activities such as walking or going
upstairs.

In comparison, after adding the Gradient Reversal Layer in the back-
propagation path of the domain classifier, the model exhibits strong potential
in recognizing various activities, according to the confusion matrix in Figure
4(a). The model’s performance on standing and jogging activities underlines
its capability to discern static and highly dynamic activities with high
precision. As mentioned, the amplitudes and patterns of IMU sequences while
jogging show significant differences between earbuds and smartphones, which is
addressed by domain adaptation. This significant improvement in performance
highlights the effectiveness of bridging the domain gap and transferring
knowledge even with limited earable domain data.

###  IV-E Effectiveness of head movement mitigation

To investigate the impact of pre-processing on system performance, we compare
the results before and after applying low-pass filtering. The accuracy and
F1-score without head movement mitigation are presented in Table II, while
Table I displays the results with head movement mitigation. The test set is
further divided into 5 groups based on the magnitude of head movements: slight
head movement, random head movement, roll, yaw, and pitch.

From Table II, it is evident that the model’s accuracy decreases when specific
types of head movements are present, particularly under the investigation of
rolling, yawing, or pitching. When participants make an effort to keep their
heads steady during data collection (as seen in the slight head movement
column), the accuracy reaches up to 93%. However, when larger magnitude head
movements are present, the accuracy degrades to approximately 85%.

As expected, the filter effectively eliminates high-frequency components of
the Airpods IMU data while preserving frequency components below 5 Hz, which
are relevant to body activities. As shown in Table I, the model’s performance
in classifying all four activities under various head movement scenarios
improves, with an overall accuracy increment of around 4%. Notably, after
applying the 5 Hz low-pass filter, there is no degradation in accuracy,
regardless of the magnitudes or kinds of head movement interference. It is
crucial to note that the utilization of more advanced techniques or components
could potentially isolate and mitigate even low-frequency head movements from
the captured motion signals. However, those approaches would inevitably
introduce a higher computational overhead and might necessitate a larger
amount of data for training, which is not desired here.

##  V Related Work

Until now, deep learning models deployed in wearable devices facilitate many
ubiquitous applications. Various sensors like IMUs, microphones, and wireless
modules are widely embedded in mobile devices, and there are existing works
focusing on human activity recognition or human interaction detection based on
sensor signals captured by mobile devices, especially smartphones [20, 21, 22,
23, 24, 25, 26]. In industry or academia, ear-worn devices have been a
promising wearable technology for areas like speech enhancement [3, 4], eating
episodes detection [1], respiration rate measurement [2]. Placed on the ear,
earbuds equipped with the accelerometer and gyroscope enable the motion
tracking of the body while they are less susceptible to motion disturbance as
the upper body acts as a natural filter. They are widely utilized in the areas
like human body movement classification or detection [6, 7, 27, 5, 28].
Furthermore, compared to traditional IMU sensors placed in other positions,
earables can sense subtle head or facial movements [29, 30, 31].

Domain adaptation, a specialized branch of transfer learning, aims to train a
model using the data of the source domain and achieve high accuracy on a
target dataset, which differs significantly from the source. Domain adaptation
has been studied in the realm of sensing systems. It has been proven to be
effective in addressing the domain shift issue in wireless sensing [32, 33,
34, 35, 36, 37, 38, 39, 40, 41] and wearable sensing [42, 43, 44]. Few
attempts have been made to utilize domain adaptation in earable sensing.
Nguyen et al. [45] addressed the lack of publicly available datasets in
Respiratory Symptom Detection using earables. A domain adaptation layer was
implemented in [46] to guide the model to learn common feature representations
by aligning the distribution of both the source and target domains. Zhang et
al. [47] sought to use domain adaptation to minimize the training overhead
while maximizing the model’s adaptability to individual user characteristics.

##  VI Conclusion

In this work, we introduce EarDA, a novel system for earable-based human
activity recognition (HAR) that effectively addresses the challenges of domain
adaptation and mitigates the impact of head movements. Our system leverages
the rich datasets available from smartphone-based IMU sensors and adapts them
for use with COTS earable devices, which are becoming increasingly prevalent.
Through extensive experiments, we have demonstrated that the system can
improve the classification accuracy to 88.8% in activity recognition tasks,
even with limited earable training data. The system’s performance highlights
the potential of domain adaptation in the realm of earable sensing and sets
the stage for future work in this area.

## Acknowledgment

The work was supported by the Research Grants Council of the Hong Kong Special
Administrative Region, China (Project No. CityU 21201420 and CityU 11201422),
the Innovation and Technology Commission of Hong Kong (Project No.
PRP/037/23FX and MHP/072/23), NSF of Shandong Province (Project No.
ZR2021LZH010).

## References

  * [1] A. Bedri, R. Li, M. Haynes, R. P. Kosaraju, I. Grover, T. Prioleau, M. Y. Beh, M. Goel, T. Starner, and G. Abowd, “Earbit: using wearable sensors to detect eating episodes in unconstrained environments,” _Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies_ , vol. 1, no. 3, pp. 1–20, 2017.
  * [2] T. Röddiger, D. Wolffram, D. Laubenstein, M. Budde, and M. Beigl, “Towards respiration rate monitoring using an in-ear headphone inertial measurement unit,” in _Proceedings of the 1st International Workshop on Earable Computing_ , 2019, pp. 48–53.
  * [3] D. Duan, Y. Chen, W. Xu, and T. Li, “Earse: Bringing robust speech enhancement to cots headphones,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)_ , vol. 7, no. 4, pp. 1–33, 2024.
  * [4] L. He, H. Hou, S. Shi, X. Shuai, and Z. Yan, “Towards bone-conducted vibration speech enhancement on head-mounted wearables,” in _Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services_ , 2023, pp. 14–27.
  * [5] T. Hossain, M. S. Islam, M. A. R. Ahad, and S. Inoue, “Human activity recognition using earable device,” in _Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers_ , 2019, pp. 81–84.
  * [6] F. Kawsar, C. Min, A. Mathur, and A. Montanari, “Earables for personal-scale behavior analytics,” _IEEE Pervasive Computing_ , vol. 17, no. 3, pp. 83–89, 2018.
  * [7] C. P. Burgos, L. Gärtner, M. A. G. Ballester, J. Noailly, F. Stöcker, M. Schönfelder, T. Adams, and S. Tassani, “In-ear accelerometer-based sensor for gait classification,” _IEEE Sensors Journal_ , vol. 20, no. 21, pp. 12 895–12 902, 2020.
  * [8] F. Kawsar, C. Min, A. Mathur, A. Montanari, U. G. Acer, and M. Van den Broeck, “esense: Open earable platform for human sensing,” in _Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems_ , 2018, pp. 371–372.
  * [9] T. Röddiger, T. King, D. R. Roodt, C. Clarke, and M. Beigl, “Openearable: Open hardware earable sensing platform,” in _Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers_ , 2022, pp. 246–251.
  * [10] G. Li, Z. Cao, and T. Li, “Echoattack: Practical inaudible attacks to smart earbuds,” in _Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services_ , 2023, pp. 383–396.
  * [11] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International conference on machine learning_. PMLR, 2015, pp. 1180–1189.
  * [12] E. K. Antonsson and R. W. Mann, “The frequency content of gait,” _Journal of biomechanics_ , vol. 18, no. 1, pp. 39–47, 1985.
  * [13] C. N. Rinaudo, M. C. Schubert, W. V. Figtree, C. J. Todd, and A. A. Migliaccio, “Human vestibulo-ocular reflex adaptation is frequency selective,” _Journal of neurophysiology_ , 2019.
  * [14] A. Saeed, T. Ozcelebi, and J. Lukkien, “Multi-task self-supervised learning for human activity detection,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 2019.
  * [15] H. Xu, P. Zhou, R. Tan, M. Li, and G. Shen, “Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications,” in _Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems_ , 2021, pp. 220–233.
  * [16] M. Malekzadeh, R. G. Clegg, A. Cavallaro, and H. Haddadi, “Mobile sensor data anonymization,” in _Proceedings of the international conference on internet of things design and implementation_ , 2019, pp. 49–58.
  * [17] A. Stisen, H. Blunck, S. Bhattacharya, T. S. Prentow, M. B. Kjærgaard, A. Dey, T. Sonne, and M. M. Jensen, “Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition,” in _Proceedings of the 13th ACM conference on embedded networked sensor systems_ , 2015, pp. 127–140.
  * [18] J.-L. Reyes-Ortiz, L. Oneto, A. Samà, X. Parra, and D. Anguita, “Transition-aware human activity recognition using smartphones,” _Neurocomputing_ , vol. 171, pp. 754–767, 2016.
  * [19] M. Shoaib, S. Bosch, O. D. Incel, H. Scholten, and P. J. Havinga, “Fusion of smartphone motion sensors for physical activity recognition,” _Sensors_ , vol. 14, no. 6, pp. 10 146–10 176, 2014.
  * [20] W. Jiang and Z. Yin, “Human activity recognition using wearable sensors by deep convolutional neural networks,” in _Proceedings of the 23rd ACM international conference on Multimedia_ , 2015, pp. 1307–1310.
  * [21] S. Yao, S. Hu, Y. Zhao, A. Zhang, and T. Abdelzaher, “Deepsense: A unified deep learning framework for time-series mobile sensing data processing,” in _Proceedings of the 26th international conference on world wide web_ , 2017, pp. 351–360.
  * [22] S. Liu, S. Yao, J. Li, D. Liu, T. Wang, H. Shao, and T. Abdelzaher, “Giobalfusion: A global attentional deep learning framework for multisensor information fusion,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 2020.
  * [23] T. Ni, X. Zhang, C. Zuo, J. Li, Z. Yan, W. Wang, W. Xu, X. Luo, and Q. Zhao, “Uncovering user interactions on smartphones via contactless wireless charging side channels,” in _2023 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2023, pp. 3399–3415.
  * [24] Y. Chen, T. Ni, W. Xu, and T. Gu, “Swipepass: Acoustic-based second-factor user authentication for smartphones,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 6, no. 3, pp. 1–25, 2022.
  * [25] T. Ni, X. Zhang, and Q. Zhao, “Recovering fingerprints from in-display fingerprint sensors via electromagnetic side channel,” in _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_ , 2023, pp. 253–267.
  * [26] T. Ni, Y. Chen, W. Xu, L. Xue, and Q. Zhao, “Xporter: A study of the multi-port charger security on privacy leakage and voice injection,” in _Proceedings of the 29th Annual International Conference on Mobile Computing and Networking_ , 2023, pp. 1–15.
  * [27] C. Min, A. Mathur, and F. Kawsar, “Exploring audio and kinetic sensing on earable devices,” in _Proceedings of the 4th ACM Workshop on Wearable Systems and Applications_ , 2018, pp. 5–10.
  * [28] M. Laporte, P. Baglat, S. Gashi, M. Gjoreski, S. Santini, and M. Langheinrich, “Detecting verbal and non-verbal gestures using earables,” in _Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers_ , 2021.
  * [29] P. Zhu, Y. Zou, W. Li, and K. Wu, “Char: Composite head-body activities recognition with a single earable device,” in _2023 IEEE International Conference on Pervasive Computing and Communications (PerCom)_. IEEE, 2023, pp. 212–221.
  * [30] K. Li, R. Zhang, B. Liang, F. Guimbretière, and C. Zhang, “Eario: A low-power acoustic sensing earable for continuously tracking detailed facial movements,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 6, no. 2, pp. 1–24, 2022.
  * [31] S. Gashi, A. Saeed, A. Vicini, E. Di Lascio, and S. Santini, “Hierarchical classification and transfer learning to recognize head gestures and facial expressions using earbuds,” in _Proceedings of the 2021 International Conference on Multimodal Interaction_ , 2021, pp. 168–176.
  * [32] Y. Zheng, Y. Zhang, K. Qian, G. Zhang, Y. Liu, C. Wu, and Z. Yang, “Zero-effort cross-domain gesture recognition with wi-fi,” in _Proceedings of the 17th annual international conference on mobile systems, applications, and services_ , 2019, pp. 313–325.
  * [33] W. Jiang, C. Miao, F. Ma, S. Yao, Y. Wang, Y. Yuan, H. Xue, C. Song, X. Ma, D. Koutsonikolas _et al._ , “Towards environment independent device free human activity recognition,” in _Proceedings of the 24th annual international conference on mobile computing and networking_ , 2018, pp. 289–304.
  * [34] T. Gong, Y. Kim, J. Shin, and S.-J. Lee, “Metasense: few-shot adaptation to untrained conditions in deep mobile sensing,” in _Proceedings of the 17th Conference on Embedded Networked Sensor Systems_ , 2019.
  * [35] T. Ni, J. Li, X. Zhang, C. Zuo, W. Wang, W. Xu, X. Luo, and Q. Zhao, “Exploiting contactless side channels in wireless charging power banks for user privacy inference via few-shot learning,” in _Proceedings of the 29th Annual International Conference on Mobile Computing and Networking_ , 2023.
  * [36] T. Ni, Y. Chen, K. Song, and W. Xu, “A simple and fast human activity recognition system using radio frequency energy harvesting,” in _Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers_ , 2021, pp. 666–671.
  * [37] Z. Sun, T. Ni, H. Yang, K. Liu, Y. Zhang, T. Gu, and W. Xu, “Flora: Energy-efficient, reliable, and beamforming-assisted over-the-air firmware update in lora networks,” in _Proceedings of the 22nd International Conference on Information Processing in Sensor Networks_ , 2023, pp. 14–26.
  * [38] ——, “Flora+: Energy-efficient, reliable, beamforming-assisted, and secure over-the-air firmware update in lora networks,” _ACM Transactions on Sensor Networks_ , 2024.
  * [39] M. Han, H. Yang, T. Ni, D. Duan, M. Ruan, Y. Chen, J. Zhang, and W. Xu, “mmsign: mmwave-based few-shot online handwritten signature verification,” _ACM Transactions on Sensor Networks_ , 2023.
  * [40] T. Ni, G. Lan, J. Wang, Q. Zhao, and W. Xu, “Eavesdropping mobile app activity via $\\{$Radio-Frequency$\\}$ energy harvesting,” in _32nd USENIX Security Symposium (USENIX Security 23)_ , 2023, pp. 3511–3528.
  * [41] Z. Sun, T. Ni, Y. Chen, D. Duan, K. Liu, and W. Xu, “Rf-egg: An rf solution for fine-grained multi-target and multi-task egg incubation sensing,” in _Proceedings of the 30th Annual International Conference on Mobile Computing and Networking_ , 2024.
  * [42] A. Natarajan, G. Angarita, E. Gaiser, R. Malison, D. Ganesan, and B. M. Marlin, “Domain adaptation methods for improving lab-to-field generalization of cocaine detection using wearable ecg,” in _Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing_ , 2016, pp. 875–885.
  * [43] A. Akbari and R. Jafari, “Transferring activity recognition models for new wearable sensors with deep generative domain adaptation,” in _Proceedings of the 18th International Conference on Information Processing in Sensor Networks_ , 2019, pp. 85–96.
  * [44] D. Duan, H. Yang, G. Lan, T. Li, X. Jia, and W. Xu, “Emgsense: A low-effort self-supervised domain adaptation framework for emg sensing,” in _2023 IEEE International Conference on Pervasive Computing and Communications (PerCom)_. IEEE, 2023, pp. 160–170.
  * [45] N. Nguyen, A. Chakma, and N. Roy, “A scalable and domain adaptive respiratory symptoms detection framework using earables,” in _2021 IEEE International Conference on Big Data (Big Data)_ , 2021.
  * [46] H. Li, Y. Zhang, J. Han, Y. Yan, Y. Liu, and H. Yang, “Adapsqa: Adaptive ecg signal quality assessment model for inter-patient paradigm using unsupervised domain adaptation,” in _2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , 2022.
  * [47] S. Zhang, T. Lu, H. Zhou, Y. Liu, R. Liu, and M. Gowda, “I am an earphone and i can hear my user’s face: Facial landmark tracking using smart earphones,” _ACM Transactions on Internet of Things_ , vol. 5, no. 1, pp. 1–29, 2023.