# Synthetic Multimodal Dataset for Empowering Safety and Well-being in Home
Environments ††thanks: The first and second authors contributed equally to
this research.

Takanori Ugai
Fujitsu Limited
Kanagawa, Japan
National Institute of Advanced Industrial
Science and Technology (AIST)
Tokyo, Japan
<EMAIL_ADDRESS>
Shusaku Egami
National Institute of Advanced Industrial
Science and Technology (AIST)
Tokyo, Japan
<EMAIL_ADDRESS>
Swe Nwe Nwe Htun
National Institute of Advanced Industrial
Science and Technology (AIST)
Tokyo, Japan
<EMAIL_ADDRESS>
Kouji Kozaki
Osaka Electro-Communication University
Osaka, Japan
<EMAIL_ADDRESS>
Takahiro Kawamura
National Agriculture and Food Research Organization
Ibaraki, Japan
<EMAIL_ADDRESS>
Ken Fukuda
National Institute of Advanced Industrial
Science and Technology (AIST)
Tokyo, Japan
<EMAIL_ADDRESS>

###### Abstract

This paper presents a synthetic multimodal dataset of daily activities that
fuses video data from a 3D virtual space simulator with knowledge graphs
depicting the spatiotemporal context of the activities. The dataset is
developed for the Knowledge Graph Reasoning Challenge for Social Issues
(KGRC4SI), which focuses on identifying and addressing hazardous situations in
the home environment. The dataset is available to the public as a valuable
resource for researchers and practitioners developing innovative solutions
recognizing human behaviors to enhance safety and well-being in home
environments.

_K_ eywords Multimodal Dataset $\cdot$ Daily Activities $\cdot$ Knowledge
Graph $\cdot$ Videos $\cdot$ Dataset and Technical Competition

##  1 Introduction

This paper introduces a multimodal dataset developed for the Knowledge Graph
Reasoning Challenge for Social Issues (KGRC4SI)111https://challenge.knowledge-
graph.jp/2022/index_en.html. The challenge addresses the critical task of
identifying hazardous situations in home environments using video data and/or
knowledge graph reasoning techniques.

The dataset combines simulated video data of daily activities with knowledge
graphs (KGs) representing the spatiotemporal properties of the activities. It
includes 203 scenarios with 1,218 videos from different perspectives and room
layouts, RDF-format [1] KGs of 2,902,676 triples, comprehensive schema
details, and a SPARQL endpoint.

Four support tools were developed and are also made open to the public. Among
them, two are discussed in this paper: VirtualHome-AIST, which triples the
number of executable actions of the previous work, and VirtualHome2KG, a
system that outputs simulation results as KGs. VirtualHome-AIST also generates
2D bounding box information for objects. VirtualHome2KG tracks object state
changes and spatial situations, such as 3D coordinates.

The paper also discusses related works, highlighting scene graph generation
and event-centered KGs.

##  2 Related Works

In recent years, much attention has been paid to the study of scene graph
generation (SGG), which represent video contents (scenes) in a graph structure
[2]. SGG data and our synthetic KG capture the video context, frame-by-frame,
as a graph structure for the downstream tasks. Qiu et al. [3], discussing the
label inconsistencies in existing datasets and ignorance of temporal
relationships in SGG models, propose a synthetic dataset using VirtualHome [4]
for SGGs. The novelty of our dataset is we incorporate Semantic Web
technologies into the graph data management to strengthen the downstream
reasoning tasks. It shall also be noted that our KGs are automatically
generated from the information in the simulator and do not require manual
annotations.

While KGs typically represent factual knowledge, KGs representing events such
as historical events are known as Event Knowledge Graphs (EKGs) [5]. Several
models have been proposed for EKGs, such as Simple Event Model (SEM) [6] and
Event Ontology222http://motools.sourceforge.net/event/event.html. Rospocher et
al. [7] constructs EKG from news articles, and Gottschalk et al. [8] and
Kawamura et al. [9] construct EKGs from mystery novels. These EKGs are called
Event-Centric KGs. This paper also adopts the Event-Centric KG model.

##  3 Dataset Contents

The dataset comprises Scenario Data, Video Data simulated from the scenario
data using virtual space, and KGs representing the “who,” “what action,”
“object,” “when and where,” and the resulting “state” and “position” of the
objects. Knowledge Graph Embedding Data is provided to facilitate reasoning
based on machine learning techniques. This dataset is available to the public
as open data [10].

###  3.1 Scenario Data

Table 1: Number of Actions in Script Data Action | # of Usage | Action | # of Usage
---|---|---|---
SHAKE | 3 | SIT | 55
CLOSE | 5 | WIPE | 97
OPEN | 5 | PUTBACK | 105
LOOKAT | 7 | SWITCHOFF | 106
SCRUB | 22 | SWITCHON | 106
EAT | 26 | TURNTO | 109
PUTOBJBACK | 26 | GRAB | 298
DRINK | 52 | WALK | 727
Table 2: Number of Objects in Script Data Object | # of Usage | Object | # of Usage
---|---|---|---
kitchen | 5 | pillow | 30
television | 6 | desk | 32
livingroom | 7 | alcohol | 39
facecream | 12 | bread | 39
toothbrush | 12 | cupcake | 39
toothpaste | 12 | juice | 39
groceries | 15 | milk | 39
kitchencounter | 15 | wine | 39
kitchentable | 15 | sponge | 50
stove | 15 | sink | 51
waterglass | 15 | bed | 54
wallpictureframe | 21 | sofa | 66
keyboard | 24 | washingsponge | 67
mouse | 24 | towel | 114
cat | 28 | remotecontrol | 290
fridge | 28 | tv | 507

The scenario comprises text files known as Script Data. VirtualHome2KG
generates videos and knowledge graphs from this data as input. The Script Data
include action titles and brief descriptions, all in text format. The 203
scripts use 16 different actions multiple times. The longest action sequence
in the script data is 31, with a mean length of 8.6, a median length of 7, and
a standard deviation of 5.6. Table 1 shows the number of each action in the
data set. The total number of actions appearing in all scripts is 1749. The
mean number of action occurrences is 109, the median is 52, and the standard
deviation is 180. Here, if an action, e.g., GRAB is included twice in one
scenario, we consider the number usage of GRAB is 2. GRAB and WALK are the
most commonly used actions, especially WALK is included in every scenario. 59%
of the actions are accounted for by the two actions, Grab and Walk. Table 2
shows the number of each object in the data set. The total number of objects
appearing in all scripts is 1749. The mean number of object occurrences is 55,
the median is 28, and the standard deviation is 97. Here, if an object, e.g.,
tv is included thrice in one scenario, we consider the number usage of tv is
3. Remotecontrol and tv are the most commonly used objects. 46% of the actions
are accounted for by the two objects, remotecontrol, and tv. The scripts are
grouped into six categories, 42 in EatingDrining, 7 in HouseArrangement, 70 in
HouseCleaning, 70 in Leisure, 7 in SocialInteraction and 7 in Other. These
categories follow the data set which VirtualHome provides.

###  3.2 Video Data

Table 3: Number of Video in Categories | EatingDrinking | HouseArrangement | HouseCleaning | Leisure | SocialInteraction | Other | All Categories
---|---|---|---|---|---|---|---
S1 | 36 | 6 | 66 | 60 | 6 | 6 | 180
S2 | 36 | 6 | 60 | 60 | 6 | 6 | 174
S3 | 36 | 6 | 54 | 60 | 6 | 6 | 168
S4 | 36 | 6 | 54 | 60 | 6 | 6 | 168
S5 | 36 | 6 | 60 | 60 | 6 | 6 | 174
S6 | 36 | 6 | 66 | 60 | 6 | 6 | 180
S7 | 36 | 6 | 60 | 60 | 6 | 6 | 174
All Scenes | 252 | 42 | 420 | 420 | 42 | 42 | 1218

The dataset contains videos in mp4 files derived from the 203 distinct action
scenarios. Each scenario has various camera perspectives, including a rear
view of the avatar (identified by file names ending in 0), an indoor camera
with dynamic views (ending in 1), and fixed camera views positioned in each
corner of the room (ending in 2-5). Furthermore, for each action scenario, we
generated data representing a minimum of 1 and a maximum of 7 patterns, each
depicting different room layouts(scenes). The dataset comprises a total of
1,218 videos. Table 3 shows the number of videos by scene and category. The
shortest video has 16 seconds, and the longest has 27 seconds. Notably,
specific videos intentionally simulate the movements of older adults with
avatars moving slowly.

###  3.3 Knowledge Graph

Figure 1: Class Diagram based on Event-Centric KG model [11]

The dataset includes KGs in RDF format. There are 203 knowledge graphs, each
corresponding to the videos in the dataset, with 2,902,676 triples, 62
properties, and 772,358 entities excluding literals. One KG has an average of
17,195 triples. Additionally, the dataset offers SPARQL
endpoint333https://kgrc4si.ml:7200/sparql and query examples, enabling users
to interact with the KGs. Figure 1 shows how the Event-centric KG model
represents the spatiotemporal information of the avatar and the objects. An
activity comprises multiple events where avatars, actions, objects, places,
time, and situations are associated with each event.

###  3.4 Embeddings

The dataset incorporates embedding vectors generated using three models:
TransE [12], ComplEx [13], and RotatE [14]. We created these embedding vectors
using the DGL-KE [15] with default parameters. Furthermore, we employed
jRDF2Vec [16] to generate additional embedding vectors. The embedding data
resulting from these approaches serve as a valuable resource for conducting
machine learning-based reasoning and analysis.

##  4 Supporting Tools

###  4.1 VirtualHome-AIST

VirtualHome [4] is a Unity-based444https://unity.com/ simulator with Python
APIs for the simulator operation. It offers various 3D environments,
representing different types of dwellings with diverse furnishings. The
previous study implements 21 fundamental actions with motion data, as shown in
Table 4555The table is based on VirtualHome ver.2.2. This study has
implemented 37 new actions as VirtualHome-AIST to replicate a more
comprehensive range of everyday scenarios, and 16 of them were used to
synthesize the Scenario Data. Since the cost of motion data implementation is
high, the added actions were strategically selected from previous work that
provides action classes as Primitive Action Ontology [17].

Table 4: Actions already implemented in VirtualHome and actions we have added Motion implemented actions in VirtualHome |  Find, Walk, Run, WalkTowards, WalkForward, TurnLeft, TrunRight, Sit, StandUp, Grab, Open, Close, PutBack, Put, PutIn, SwitchOn, SwitchOff, Drink, TurnTo, LookAt, Touch
---|---
Actions with added motion |  Wipe, Read, Pour, Type, Squeeze, Cut, Eat, Brush, Fold, Jump, JumpUp, JumpDown, Kneel, Lift, Rinse, Squat, Streach, Sweep, Stir, Throw, Unfold, Vacuum, Wrap, Write, Fall, FallSit, FallTable, FallBack, Climb, GoDown, Stand, Straddle, LegOpp, Scrub, Shake, Smell, Soak

###  4.2 VirtualHome2KG

VirtualHome2KG [18, 11] extends VirtualHome to store the detailed context
information as KGs. In VirtualHome2KG, the activity is a “live” activity. For
example, in Figure 2, “[GRAB] `<`groceries`>` (169)”, “[GRAB] `<`fridge`>`
(152)”, “[WATCH] `<`fridge`>` (152)”, represent a part of a sequence of
actions, object names, and object IDs contained in the activity named “Put
groceries in fridge”. And when this is executed, VirtualHome and
VirtualHome2KG update and record the environment information of the virtual
space. After “[WALK] `<`fridge`>` (152)” is executed, the coordinates of the
avatar and the groceries are updated, and after “[OPEN] `<`fridge`>` (152)” is
executed, the fridge state is updated to “OPEN”. Execution of each line of the
script data results in an Event node with all the auxiliary information stored
around the Event node (Figure 2). This recording mechanism captures the indoor
environment at every step of the activity execution.

Figure 2: Event-centric KG and corresponding video frames

###  4.3 Visualzation Tool and Scenario Editor

We developed a visualization
tool666https://github.com/aistairc/virtualhome2kg_visualization to synchronize
and display videos and KGs. The tool plays the chosen video and enumerates the
events within the video while concurrently visualizing and zooming in to the
corresponding KG to support the comprehension of the video’s event structure.
We also developed a scenario
editor777https://github.com/aistairc/virtualhome2kg_generation to assist in
writing script data. Creating an executable script is tedious as one must
choose combinations of feasible movements and objects. Even if the script runs
successfully, one has to assess if the resulting motion of the avatar is not
unnatural. This tool streamlines the data creation process by showcasing
previous successful scripts.

##  5 KGRC4SI

The Knowledge Graph Reasoning Challenge for Social Issues (KGRC4SI) is a
dedicated challenge for systems to identify and explain hazardous situations
in home environments. The first challenge focused on older adults. We envision
detecting accident risks, such as falls, caused by human behavior or human-
environment interaction. By providing a multimodal dataset of visual
information and structured symbolic information, we expect the participation
of experts from diverse fields and the dataset to be a platform for applying
technologies that aim to understand humans’ daily behaviors.

###  5.1 Challenge Tasks

We defined one main task and two subtasks for tools and metrics. Here, we
focus on the main task. The main task comprises three smaller tasks:

  1. 1.

Detect dangerous situations by analyzing the provided dataset. Identify
environments, actions, and combinations that pose a high risk of accidents for
older adults. The output should include frame IDs of the videos and partial
graphs of the KGs.

  2. 2.

Explain the reasons behind the identified risks. Describe the actions of the
avatars, the involved objects, and their relationships.

  3. 3.

Provide alternatives to decrease the detected risks. Suggest an alternative
activity that does not sacrifice the goal of the original risky actions.

###  5.2 Evaluation Criteria

Main task: The program committee (PC) verified the reproducibility of the
submitted program code. The PC then rated the following items on a 5-point
scale: (1) Adequacy assessment of defined risks, (2) Detail degree of defined
risks, (3) Accuracy evaluation of detected risks, (4) Explanation’s quality of
detected risks, (5) Goal alignment of alternative solutions, (6) technical
innovations and knowledge resource contribution.

###  5.3 Accepted Entries

We hosted KGRC4SI 2022 in Japan as a pilot project with a six-month
application period with 6 accepted submissions.

Five of the six entries had publicly available sources and data, and the PC
member could also apply the private data prepared by the contenders. Diverse
technologies were adopted in the entries, including knowledge processing,
knowledge graph embedding, and generative AI. An example of the identified
risk was “Trying to reach for something in a place higher than oneself.” Three
entries applied rule-based approaches to delineate risks and applied knowledge
processing techniques to identify potential risks, while two entries applied
link prediction with knowledge graph embedding. Three entries utilized Large
Language Models (LLMs) to facilitate description generation, thereby enabling
the detailed explanation of risks and the generation of alternative solutions.
One entry used generative AI to identify risks within videos. Two entries used
generative AI by LLMs to address the subtask of script creation.

##  6 Summary and Future Works

The paper introduces a synthetic multimodal open dataset of daily human
activities, merging visual information (video) and structured symbolic
information (KG). Four systems have been made available to the public to
generate the dataset. One system triples the number of avatars’ activities
compared to previous work, and the other enables automatic KG generation as
the gold standard annotation of the video content. Compared to the previous
SGG datasets, the novelty of the dataset is that we incorporate Semantic Web
technology to manage the intrinsically complex relationships among objects and
make them reusable for both machine learning and symbolic reasoning
approaches, which is impossible for the current SGG datasets. Through the
pilot challenge, we discovered that our dataset challenges LLMs. While LLMs do
provide general know-how answers based on common sense, the dataset is unseen
to the LLM model and requires a grounding of the physical world to some extent
to provide case-based concrete solutions.

The current dataset only has RDF-based KGs, and we plan to provide the graph
data in de facto SGG format for easy handling in machine learning
applications. We plan to add more scenarios, videos, and graph data
continuously. By doing so, we expect the dataset platform becomes applicable
to other domains, including children, factory workers, and traffic safety.

## Acknowledgement

This paper is based on results obtained from a project, JPNP20006,
commissioned by the New Energy and Industrial Technology Development
Organization (NEDO), and Japan Society Promotion of Science (JSPS) KAKENHI
Grant Numbers JP19H04168, JP22K18008, and JP23H03688.

## References

  * [1] Richard Cyganiak, David Wood, and Markus Lanthaler.  RDF 1.1 Concepts and Abstract Syntax, February 2014.
  * [2] Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann.  A comprehensive survey of scene graphs: Generation and application.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1–26, 2023.
  * [3] Yue Qiu, Yoshiki Nagasaki, Kensho Hara, Hirokatsu Kataoka, Ryota Suzuki, Kenji Iwata, and Yutaka Satoh.  Virtualhome action genome: A simulated spatio-temporal scene graph dataset with consistent relationship labels.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3351–3360, January 2023.
  * [4] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.  Virtualhome: Simulating household activities via programs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018.
  * [5] Saiping Guan et al.  What is Event Knowledge Graph: A Survey.  IEEE Transactions on Knowledge and Data Engineering, pages 1–20, 2022.
  * [6] Willem Robert Van Hage, Véronique Malaisé, Roxane Segers, Laura Hollink, and Guus Schreiber.  Design and use of the simple event model (sem).  Journal of Web Semantics, 9(2):128–136, 2011.
  * [7] Marco Rospocher, Marieke van Erp, Piek Vossen, Antske Fokkens, Itziar Aldabe, German Rigau, Aitor Soroa, Thomas Ploeger, and Tessel Bogaard.  Building event-centric knowledge graphs from news.  Journal of Web Semantics, 37:132–151, 2016.
  * [8] Simon Gottschalk and Elena Demidova.  Eventkg: a multilingual event-centric temporal knowledge graph.  In European Semantic Web Conference, pages 272–287, 2018.
  * [9] Takahiro Kawamura, Shusaku Egami, Koutarou Tamura, Yasunori Hokazono, Takanori Ugai, Yusuke Koyanagi, Fumihito Nishino, Seiji Okajima, Katsuhiko Murakami, Kunihiko Takamatsu, Aoi Sugiura, Shun Shiramatsu, Xiangyu Zhang, and Kouji Kozaki.  Report on the First Knowledge Graph Reasoning Challenge 2018 - Toward the eXplainable AI System -.  Proceedings of the 9th Joint International Semantic Technology Conference (JIST2019), pages 18–34, 11 2019.
  * [10] Takanori Ugai, Shusaku Egami, Swe Nwe Nwe Htun, Kouji Kozaki, Takahiro Kawamura, and Ken Fukuda.  Synthetic Multimodal Dataset for Daily Life Activities, June 2023.
  * [11] Shusaku Egam, Takanori Ugai, Mikiko Oono, Koji Kitamura, and Ken Fukuda.  Synthesizing event-centric knowledge graphs of daily activities using virtual space.  IEEE Access, pages 1–1, 2023.
  * [12] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multi-relational data.  In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  * [13] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard.  Complex embeddings for simple link prediction.  In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 2071â2080. JMLR.org, 2016.
  * [14] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang.  Rotate: Knowledge graph embedding by relational rotation in complex space.  In International Conference on Learning Representations, 2019.
  * [15] Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, and George Karypis.  Dgl-ke: Training knowledge graph embeddings at scale.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 739â748, New York, NY, USA, 2020. Association for Computing Machinery.
  * [16] Jan Portisch, Michael Hladik, and Heiko Paulheim.  Rdf2vec light - A lightweight approach for knowledge graph embeddings.  CoRR, abs/2009.07659, 2020.
  * [17] Satoshi Nishimura, Shusaku Egami, Takanori Ugai, Mikiko Oono, Koji Kitamura, and Ken Fukuda.  Ontologies of action and object in home environment towards injury prevention.  In The 10th International Joint Conference on Knowledge Graphs, pages 126–130, 2021.
  * [18] Shusaku Egami, Satoshi Nishimura, and Ken Fukuda.  A framework for constructing and augmenting knowledge graphs using virtual space: Towards analysis of daily activities.  In Proceedings of the 33rd IEEE International Conference on Tools with Artificial Intelligence, pages 1226–1230, 2021.


# Hybrid Soft Electrostatic Metamaterial Gripper for Multi-surface, Multi-
object Adaptation

Ryo Kanno1,2, Student Member, IEEE, Pham H. Nguyen1,3,∗, Member, IEEE, Joshua
Pinskier4, Member, IEEE,
David Howard4, Member, IEEE, Sukho Song1,3, Member, IEEE, Mirko Kovac1,2,3,∗,
Member, IEEE R. Kanno, P. H. Nguyen, S. Song, and M. Kovac are with the
Laboratory of Sustainability Robotics, Empa - Swiss Federal Laboratories for
Materials Science and Technology, 8600 Dübendorf, Switzerland. $\\{$ryo.kanno,
huy.pham, sukho.song<EMAIL_ADDRESS>Kanno, and M. Kovac are with
the École Polytechnique Fédérale de Lausanne, 1005 Lausanne, Switzerland.P. H.
Nguyen, S. Song, and M. Kovac are with the Aerial Robotics Laboratory,
Imperial College London, South Kensington Campus, London, SW7 2AZ, United
Kingdom.J. Pinskier and D. Howard are with the Commonwealth Scientific and
Industrial Research Organisation (CSIRO). $\\{$josh.pinskier,
<EMAIL_ADDRESS>Address all correspondence to these
authors.

###### Abstract

One of the trendsetting themes in soft robotics has been the goal of
developing the ultimate universal soft robotic gripper. One that is capable of
manipulating items of various shapes, sizes, thicknesses, textures, and
weights. All the while still being lightweight and scalable in order to adapt
to use cases. In this work, we report a soft gripper that enables delicate and
precise grasps of fragile, deformable, and flexible objects but also excels in
lifting heavy objects of up to $1617x$ its own body weight. The principle
behind the soft gripper is based on extending the capabilities of
electroadhesion soft grippers through the enhancement principles found in
metamaterial adhesion cut and patterning. This design amplifies the adhesion
and grasping payload in one direction while reducing the adhesion capabilities
in the other direction. This counteracts the residual forces during peeling (a
common problem with electroadhesive grippers), thus increasing its speed of
release. In essence, we are able to tune the maximum strength and peeling
speed, beyond the capabilities of previous electroadhesive grippers. We study
the capabilities of the system through a wide range of experiments with single
and multiple-fingered peel tests. We also demonstrate its modular and adaptive
capabilities in the real-world with a two-finger gripper, by performing
grasping tests of up to $5$ different multi-surfaced objects.

##  I INTRODUCTION

Figure 1: (a) Soft Electrostatic Metamaterial (SEM) gripper. (b) The SEM
adhesive. The scale bar is 20 mm. Figure 2: Adhesion Simulation in COMSOL. a)
The metamaterial is first evaluated by lifting in the forward direction until
the material releases the substrate [Top], then repeated in the reverse
direction [Bottom]. Results of (b) 3 Point Polygon, (c) 5 Point Polygon, and
(d) 7 Point Polygon optimizations, showing high-performing shapes and
optimization chronology. The color bar shows the sequence of design
evaluations, with brighter colors occurring later. The clustering of bright
numbers at the top right is indicative of optimizer progress.

The race towards the ultimate universal soft gripper has been on display since
the dawn of soft robotics [1]. The goal has always been to develop a
lightweight, compliant, robust, and versatile system that is capable of
diverse grasps ranging from handling delicate, flat, and deformable objects
with varying weights, material properties, surface textures, shapes and sizes
[2]. To achieve this balance between various trade offs such as innate
softness, precision, and grasp strength, electroadhesive gripper technology
has emerged with varying successes and limitations [3, 4, 5, 6]. Electrostatic
soft grippers have highlighted their capabilities to gently grasp various
types of objects, whether that be thin sheets, deformable objects, or fragile
objects [7, 3]. One of their key advantages is their ability to adhere to
varying surface roughness. The electrostatic force generated also assists with
alignment and engagement to the contact surface of the object. They also
highlight the capabilities of self-sensing. One weakness of these
electrostatic soft grippers is that their maximum normal stress is quite low,
thus limiting their ability to carry heavy loads [7, 3]. Furthermore, during
disengaging from the target object, small residual forces remain, thus slowing
down the release phase [8]. This might create a challenge when trying to
rapidly pick and place delicate objects. To tackle some of the weakness seen
with electrostatic soft grippers, researchers have looked at hybrid systems,
whether that be combining electroadhesion with suction cups [6],
microfibrillar structures [9], jamming structures [5], and magnetic
augmentation [10, 7].

In this work, we aim to develop a novel soft gripper, called the Soft
Electrostatic Metamaterial (SEM) gripper, that tackles limitations of the
electrostatic soft grippers, as seen in Fig. 1a. In essence, we tackle the
problem of residual forces during disengaging the gripper, by controlling the
directionality and force of peeling. At the same time, we also amplify the
lifting capacity of the gripper.

In order to achieve this, we integrate metamaterial cuts with the localized
dielectric elastomer substrate, as seen in Fig. 1b. The metamaterial cuts are
inspired by the capabilities of metamaterial adhesives seen in Hwang et. al
[11]. To our knowledge, metamaterial adhesion has not been utilized in soft
robotic grippers, because they are still reliant on the substrate material for
their adhesive properties and require mechanical preloading onto the target
object to be effective.

With the combination of metamaterial cuts and electroadhesion, we are able to
tune the electroadhesion force to pull the SEM membrane into contact with the
object’s surface, and provide the preloading required to fully engage the
adhesive, without unpredictable deformation or damage on the target object’s
surface. The combination of these two technologies led to the introduction of
the SEM gripper that can handle flat, irregularly-shaped, and oversized
objects with different textures in one system.

The remainder of the paper is as follows. Section II introduces the working
principle, shape optimization, and fabrication of the SEM adhesives. Section
III presents characterizations of the adhesive, utilizing the cut patterns
determined through simulation. Section IV highlights the grasping force
characterization and pick-and-release demonstration of the adhesives and the
SEM gripper’s graspability with up to $5$ different surfaces of objects.
Section VI concludes this paper and discusses future directions.

##  II Design, Shape Optimization, and Fabrication

The SEM gripper was developed to improve three main limitations seen with
electrostatic soft grippers and metamaterial adhesives. First, was the
reduction of the residual forces created by electrostatic grippers during
disengagement. Second, was amplifying the electrostatic gripper’s grasping
force. And finally, was the providing controlled preloading required for
metamaterial adhesives.

Design |  Forward Energy |  Energy Ratio
---|---|---
Square | 154.21 | 1.58
Triangle | 278.21 | 4.06
Hybrid | 205.73 | 2.54
TABLE I: Reference Shape Simulation Results

###  II-A Working Principle

To design and develop this gripper, we utilized two heritage projects as both
reference and comparison [3, 11]. In Fig. 1b, we notice the combination of
interdigitated electrodes and metamaterial cut patterns. Through the
programmable metamaterial cut architecture, we are able to achieve amplified
adhesion in one direction, while releasable adhesion in the other. Thus
overcoming the residual force during disengagment in one direction and
improving the overall grasping force. Further, by activating electrodes, we
are able to create electrostatic force to engage the adhesive to the surface
of the grasping object to activate the metamaterial adhesive properly, as seen
in Fig. 4c.

Figure 3: Fabrication process. a-i) First dielectric mixture layer is blade-
casted by a film applicator. a-ii) Laser cut interdigitated electrodes are
attached to the pre-cured dielectric layer onto a PET sheet. a-iii) Second
dielectric mixture layer is blade-casted. a-iv) Metamaterial cut lines are
created using laser cutter. Conductive tapes are attached. b) Prototyped SEM
adhesive [Left]. An electrostatic adhesion is generated by applying high
voltage. [Right]

###  II-B Shape Optimization

Efficient computational design tools are often required to explore the large
space of possible designs, which exploit available data and rapidly iterate
over the design space [12]. Here, the shape of the metamaterial cut pattern
will significantly impact adhesion performance, hence a series of model-based
grasp optimizations were undertaken to identify high-performing cut
geometries.

A multi-objective optimization problem was established to find designs which
give both high-adhesion force and high adhesion directionality, which
corresponds to low detachment force, and solved using the well known NSGA-II
evolutionary algorithm [13]. As forces give very noise measurements, internal
energy is used to assess adhesion force. The cut geometries are parameterized
by a set of open-ended polygons with fixed endpoints and $n$ internal points
where $n\in\\{3,5,7\\}$. Hence, three sets of optimizations are performed,
each with $2n$ optimization variables, the $x$ and $y$ coordinate for each
point. In each optimization run, 600 designs are evaluated, comprising 12
generations each with a population of 50. Some are highlighted in Fig. 2.

Polygons are formed by joining points in the clockwise direction and swapping
intersecting lines as in Auer et. al [14]. Designs are evaluated using an
adhesion study in COMSOL multiphysics, with shear and tensile strength
$80\text{\,}\mathrm{MPa}$, energy release rate
$2.8\text{\,}\mathrm{kJ}\text{\,}{\mathrm{m}}^{-2}$, and Mode Mixity exponent
$2.284$. In each simulation, the metamaterial sheet is initially attached to
an acrylic substrate, and raised from one side in $0.1\text{\,}\mathrm{mm}$
increments until detachment. The same process is then repeated with the
displacement applied at the opposite end to find the reverse direction release
energy and calculate the ratio of forward to reverse energy. This process is
illustrated in Fig. 2a.

Three benchmark designs are used for comparison: square, triangular and hybrid
(house-shaped) designs. Data from these reference shapes are presented in
Table I for comparison with the 3 sets of optimizations.

The results from the 3 optimizations are presented in Fig.2b-d, illustrating
the high performing designs. The presented results exclude several outlying
points with nominal performance several orders of magnitude better than those
presented. These non-physical results occur in designs with very high aspect
ratio sections (i.e long, thin parts). From the simulated results, three
features are noticeable: firstly, it appears to be advantageous to occupy a
large area, and have a pointed upper edge. Secondly that there is a strong
correlation between forward grasp strength and grasp ratio, this occurs
because the reverse direction release energy is relatively insensitive to
geometry, and varies within a smaller range than the forward direction.

Figure 4: SEM gripper Mechanism. a) A parallel gripper controlled with a
single servo motor [Top]. The mechanism of the parallel gripper [Bottom]. b)
The bistable gripper mechanism consists of two arms connected by a hinge
joint. Two rubberbands are connected at the top and bottom of the arm in order
to create the bistable mechanism. The tendon assists in pulling it back to its
initial state. c) The gripper snaps onto the object as soon as on contact with
object, voltage is then applied to enhance adhesion between the object and the
adhesive [Top]. The two-arm symmetrical SEM adhesive. Note that the grasping
stress applied to SEM adhesive is the same direction for maximum peeling force
enduring large payload [Bottom].

###  II-C Adhesive Fabrication Steps

To fabricate the SEM adhesives we utilized a layer-by-layer process as seen in
Fig. 3. The elastomer mixtures in this work were mixed utilizing a planetary
centrifugal mixer (ARE-250, Thinky, Japan). First, the urethane mixture
(Vytaflex 20, Smooth-on, PA), was blade-casted to a thickness of 200 $\mu$m
using a universal film applicator (ZUA 2000 Zehntner, Screening Eagle
Technologies, TX) on a 42 $\mu$m thick PET sheet (L480.385, Rausch Packing,
Switzerland), highlighted in Fig. 3a-i). This layer was pre-cured at room
temperature for 20 minutes to obtain a dielectric layer. A 30 $\mu$m carbon
black-based double-sided adhesive sheet (ARclad 8006, Adhesive Research, PA),
was laser cut (Nova 24, Thunder Laser, Germany) to the interdigitated
geometry, as seen in Fig. 3a-ii). Note that the line and gap thickness were
set at 1 mm in this work. To attach the laser-cut electrode to the urethane
film in the previous step, the back end of the electrode is affixed onto an
inverted mask of double-sided VHB tape (VHB4905, 3M, MN). This combination is
then attached to the urethane film and peeled, to leave the exposed
interdigitated electrode, as shown in Fig. 3a.ii). The electrode sheet is then
cured at room temperature for three hours. A second layer of urethane or PDMS
mixture (LSR4130, Elkem Silicones, Norway) is then blade-casted at 100 $\mu$m
thickness, to cover the exposed electrode, as seen in Fig. 3a-iii), making the
overall thickness of the sheet 300 $\mu$m. The sheet is laser cut to generate
the metamaterial geometry, after another 3 hour room temperature cure, seen in
Fig. 3a-iv). Conductive tapes are attached to the exposed electrode lines, as
seen in Fig. 3a-iv).

For our characterization tests, we utilized a 4x4 grid, of 10 mm x 15 mm
rectangles, as in Fig. 3b, comparative to the samples in Hwang et. al [11].
For our gripper characterization tests, similar to Shintake et. al [3], 2x4
grids of the same size were made, as seen in Fig. 1b.

Figure 5: a) Normalized adhesion directionality of simulation vs experiment.
b) Experimental normalized forward adhesion stress of different shapes.

The obtained adhesive is shown in Fig. 3a-v), with a weight of $1.8$ g. Within
the square geometry, electrodes are polarized when high voltage is applied and
create a fringe electric field allowing to adhere objects shown in Fig. 3b.
The adhesives were able to withstand up to 6 kV$/$mm applied by a high-voltage
amplifier (CB101, XP Power, NJ).

###  II-D SEM Gripper Design

In this work, we developed two different types of SEM grippers as seen in Fig.
4. The parallel motion gripper shown in Fig. 4a, employs a motorized mechanism
that opens and closes along the same axis. This simple single motor mechanism
assists with precision for the testing procedures seen in the next section.
The motorized mechanism allows for precise control to set an ideal position,
ensuring that the SEM adhesive can be securely attached to the object without
causing damage or slippage.

Figure 6: Metamaterial vs SEM characterization. Metamaterial adhesion of a)
adhesion stress versus peeling distance with roller preloading, b-c) adhesion
directionality and enhancement ratio of urethane and PDMS contact layer.
Metamaterial and electrostatic co-adhesion of d) adhesion stress versus
peeling distance applying 4 kV$/$mm of electric field without roller
preloading, e-f) adhesion directionality and enhancement ratio of urethane
rubber applying 0, 2, 4, and 6 kV$/$mm of electric field.

On the other hand, the gripper, illustrated in Fig. 4b,c, snaps around a hinge
joint. This bistable gripper was inspired by Hawkes et. al [15] that utilized
cardboard. In this work, we instead utilized 3D-printed joints, the arms of
the gripper utilizing PETG (Bambu Labs, TX), and pre-strained elastic bands to
generate bistability. When the object comes into contact with the double-sided
SEM adhesive membrane, as seen in Fig. 4b, the gripper collapses, allowing the
SEM adhesive to wrap around the object. At the same time, the SEM gripper is
activated (electric field is turned on to approx. 0-4 kV$/$mm), to enhance the
gripper-to-object adhesion area. The direction of grasping stress and maximum
force is preprogrammed based on the metamaterial cut layout. Note that for
flat objects, the tabs are inverted compared to when grasping round objects.
When releasing the object, the electroadhesion is turned off and the tendon is
pulled. The directionality of the metamaterial adhesion allows quick peeling
overcoming the problem of residual electroadhesion holding onto the object.
Thus, allowing the gripper to quickly release the object.

##  III SEM Adhesive Characterization

In this section, we validated the simulated metamaterial cut models through a
series of experimental tests. Our objective was to characterize the adhesive
properties using predefined metamaterial cuts configured in a 4x4 grid,
applicable to both the metamaterial adhesive and the SEM adhesive, in
scenarios with and without roller preload. In addition, testing was done at
different flat surface conditions (glass and wood) for both adhesives in order
to investigate the difference in adhesion properties. Furthermore, we
characterized two different curved surface diameters ($\phi$ = 51 and 77 mm)
of PVC pipes. The adhesives were fabricated using either urethane or PDMS, and
both materials were characterized as part of this study.

Figure 7: Comparison with metamaterial adhesion, electroadhesion, and hybrid
adhesion. A) Electroadhesion with preloading [8]. B) Metamaterial adhesion
without preloading. C) Hybrid adhesion without preloading. D) Metamaterial
adhesion with preloading.

###  III-A Metamaterial Simulation vs Experimental Results

To validate the precision of the simulations, we conducted experiments on
selected representative results from each set of internal points and
subsequently compared these results with the simulations. The outcomes of this
comparison are depicted in Fig. 5a. Our analysis reveals a good alignment
between the experimental data and the simulation models, highlighting our
capability to design and model metamaterial cuts for specific adhesion
directionality requirements. The data also showcased maximum adhesive
directionality with the 7-pt design, and good adhesion directionality with
various 5-pt designs. However, for this work, we utilized the square-shaped
design for the SEM adhesive. The first reason was to simplify the fitting of
the electrode design within the shape geometry. The second reason was that we
noticed that its forward adhesion stress was still the highest compared to the
different designs, as seen in Fig. 2b. Thus, we decided to utilize this shape
as a benchmark shape in this work and will pursue more intricate designs in
the future, depending on the forward and reverse adhesion stress requirements.

###  III-B Adhesion stress versus peeling distance

Both types of samples were tested on a flat glass plate. For Fig. 6a, the
metamaterial adhesive was preloaded with a roller, while for Fig. 6d, the SEM
adhesive was not preloaded but activated with 4 kV$/$mm. From the results, we
notice 4 peaks that resemble the peeling at each row of the adhesives when
crack propagation occurs. The preloaded metamaterial adhesive had slightly
higher adhesion stress in comparison to the SEM adhesive, 1.8 vs 1.6 kPa. For
the metamaterial without roller preloading, the adhesion stress was accounted
at 0.52 kPa, approximately 3.45x and 3.07x lower than either version,
respectively. Thus, it highlights how well elecrostatic force attaches the
adhesive to the object substrate.

Figure 8: Adhesion stress characterization of gripper on curved surfaces
($\phi$ 51 mm and 77 mm). Tips of adhesives are (A) not surface attached to
the pipe, (B) surface attached. (C,D) Attached and with electric field of 2
kV$/$mm and 4 kV$/$mm is applied.

###  III-C Adhesion directionality and enhancement

The first directionality test was the comparison between the urethane versus
the PDMS contact layer. As seen in Fig. 6b and c, the enhancement ratio
($F_{\mathrm{max}/\mathrm{min}}$) on flat glass and wood surfaces are better
for urethane, while the performance for curved surfaces is quite similar. For
adhesion directionality, which means the ratio between forward and reverse
peeling tests, are quite similar to each other, except for the flat glass and
wood surfaces. Further, for manufacturing the adhesives, urethane layer had a
faster curing time. Thus we proceeded to make the SEM adhesives with only
Vytaflex 30, urethane contact layer.

As we look at the different results for varying the voltage applied to the
adhesive, in Fig. 6e and f, we notice the improvement in the forward peeling
direction as the voltage increases. Even though 6 kV$/$mm was applicable for
testing, the adhesives would break with prolonged and consistent testing.
Thus, we focused on utilizing only 4 kV$/$mm.

The enhancement ratio is also substantial as the voltage increases from 0-4
kV$/$mm but is more stable around 4-6 kV$/$mm, shown in Fig. 6f. The
enhancement ratio was approximately 60x at 6 kV$/$mm for wood and PVC pipes,
highlighting the multi-object adhesion capabilities of this system. The
largest adhesion directionality value was $65.5x$ with the PVC pipe ($\phi$ 51
mm). In the case of enhancement ratio was $65.6x$. Overall, the adhesion
directionality and enhancement ratio of the SEM adhesive have better
properties than the metamaterial adhesive.

Figure 9: SEM gripper experiment with a) an apple, sphere surface, that weighs
191 g (i: approaching the object, ii: bistable grasping with high voltage
being applied, iii: lifting the object, iv: releasing the object). b) Tape
(cylindrical shape, 284 g). c) Coffee bag (irregular shape, 350 g). d) Tape
(circular surface, 499 g). e) Paper (flat surface, 10 g).

###  III-D Adhesion comparison

We compared the peeling force between the metamaterial [11], electrostatic
adhesion [3], and SEM adhesives, as seen in Fig. 7. For preloading, a standard
rubber linoleum roller was utilized. We noticed that the electrostatic
adhesive displayed quite low values even when activated at 6 kV$/$mm and that
the preloaded metamaterial adhesion showed the highest adhesion stress. When
not preloaded the metamaterial adhesion is weaker in comparison. To compare,
the adhesion stress difference between the preloaded metamaterial and hybrid
SEM adhesive was approximately 20% less. The adhesive stress difference
between SEM and metamaterial, without preloading, was approximately 76% more.

##  IV SEM Gripper Characterization

In order to characterize the gripper adhesion shear force, a two-fingered
grasping gripper (2.1g in total) was developed, as seen in Fig. 8. The two PVC
pipes of $\phi$ 51 and $\phi$ 77 mm diameter were used to characterize this
gripper, and the voltage was varied from 0-4 kV$/$mm. There were four testing
conditions, A: Was without manually preloading and no voltage. B: Was
preloading and no voltage. C and D were similar the difference was just the
activation voltage, 2 and 4 kV$/$mm. Notice that the adhesion stress increased
when applying higher voltages. The maximum stress was found at 55.6 kPa, which
was approximately 33.3N pulling force on the $\phi$ 77 mm pipe. Thus, the
gripper was able to lift up to $1617x$ of its own weight.

##  V Grasping Demonstration

In order to demonstrate the developed SEM gripper for real-world applications,
we performed pick-and-release demonstrations with a total of 5 different
objects including deformable and delicate ones, with 4 kV$/$mm applied. The
result can be seen in Fig. 9 and supplementary video. The objects ranged from
an apple (191 g), a cylindrical tape roll (284 g), a coffee bag (350 g), a
duct tape roll (500 g), and a flat surface pamphlet (10 g). From these tests,
we highlighted both flat, curved, and spherical surface objects grasping and
releasing at up to $237x$ the weight of the adhesives.

##  VI Conclusion and Future Work

In this paper, we developed a Soft Electrostatic Metamaterial gripper. By
combining the unique properties of metamaterials and electroadhesion, we
successfully addressed the inherent limitations requiring the external
preloading of metamaterial grippers. The SEM gripper demonstrated its ability
for self-preloading and high adhesion force, allowing for pick-and-release
actions of a variety of objects. Through our extensive characterization and
testing, we unveiled the SEM gripper’s proficiency in adhering to a multitude
of surfaces, ranging from a flat glass pad to an irregularly shaped coffee
bag. By modulating the applied voltage, we achieved a controlled adhesion
force, enabling the gripper to lift objects up to $1617x$ of its weight. The
pick-and-release demonstrations with objects of varied shapes and weights
further exemplified the real-world applicability and versatility of the SEM
gripper. Future research could delve into optimizing the metamaterial shape
and electrode designs, further refining the gripper’s sensitivity to handle
even more delicate or intricately shaped objects. Further, developing
automated grasping mechanisms would better classify the systems’ capabilities
for picking-and-placing. There also exists a future avenue to explore the
integration of sensing capabilities for feedback control of the system, in
order to verify safe attachment or not.

## ACKNOWLEDGMENTS

We sincerely thank Prof. Dorina Opris and Ms. Jana Wolf for providing
suggestions for dielectric fabrication methods. We also thank the members of
the Laboratory of Sustainability Robotics and Aerial Robotics Lab for their
support and stimulating discussions on this topic.

This work was supported in part by EPSRC Awards (grant no, EP/R009953/1), in
part by the Swiss National Science Foundation (SNSF) International
Cooperation, EU ERANet grant (grant no, 194986), in part by the Royal Society
Wolfson fellowship (grant no. RSWF/R1/18003), in part of the ERC Consolidator
Grant as funded by (State Secretariat for Education, Research, and Innovation)
SERI (grant no. MB22.00066), and in part by the Empa-EPFL-Imperial research
partnership.

## References

  * [1] J. Shintake, V. Cacucciolo, D. Floreano and H. Shea  “Soft Robotic Grippers”  In _Advanced Materials_ 30.29, pp. 1707035  DOI: 10.1002/adma.201707035
  * [2] J. Hughes et al.  “Soft Manipulators and Grippers: A Review”  In _Frontiers in Robotics and AI_ 3, 2016, pp. 69  DOI: 10.3389/frobt.2016.00069
  * [3] Jun Shintake et al.  “Versatile soft grippers with intrinsic electroadhesion based on multifunctional polymer actuators”  In _Advanced materials_ 28.2 Wiley Online Library, 2016, pp. 231–238
  * [4] Jianglong Guo, Jinsong Leng and Jonathan Rossiter  “Electroadhesion technologies for robotics: A comprehensive review”  In _IEEE Transactions on Robotics_ 36.2 IEEE, 2019, pp. 313–327
  * [5] Yegor Piskarev et al.  “A Soft Gripper with Granular Jamming and Electroadhesive Properties”  In _Advanced Intelligent Systems_ Wiley Online Library, 2023, pp. 2200409
  * [6] Yuto Okuno, Hiroki Shigemune, Yu Kuwajima and Shingo Maeda  “Stretchable suction cup with electroadhesion”  In _Advanced Materials Technologies_ 4.1 Wiley Online Library, 2019, pp. 1800304
  * [7] Amy Kyungwon Han, Amar Hajj-Ahmad and Mark R Cutkosky  “Hybrid electrostatic and gecko-inspired gripping pads for manipulating bulky, non-smooth items”  In _Smart Materials and Structures_ 30.2 IOP Publishing, 2020, pp. 025010
  * [8] Vito Cacucciolo, Herbert Shea and Giuseppe Carbone  “Peeling in electroadhesion soft grippers”  In _Extreme Mechanics Letters_ 50, 2022, pp. 101529  DOI: https://doi.org/10.1016/j.eml.2021.101529
  * [9] Vahid Alizadehyazdi, Michael Bonthron and Matthew Spenko  “An electrostatic/gecko-inspired adhesives soft robotic gripper”  In _IEEE Robotics and Automation Letters_ 5.3 IEEE, 2020, pp. 4679–4686
  * [10] Jianglong Guo, Chaoqun Xiang, Plinio Zanini and Jonathan Rossiter  “Magnetic Augmented Self-sensing Flexible Electroadhesive Grippers”  In _IEEE Robotics and Automation Letters_ 4.3, 2019, pp. 2364–2369  DOI: 10.1109/LRA.2019.2903570
  * [11] Dohgyu Hwang et al.  “Metamaterial adhesives for programmable adhesion through reverse crack propagation”  In _Nature Materials_ 22.8 Springer ScienceBusiness Media LLC, 2023, pp. 1030–1038  DOI: 10.1038/s41563-023-01577-2
  * [12] Joshua Pinskier and David Howard  “From Bioinspiration to Computer Generation: Developments in Autonomous Soft Robot Design”  In _Advanced Intelligent Systems_ 4.1, 2022, pp. 2100086  DOI: https://doi.org/10.1002/aisy.202100086
  * [13] K. Deb, A. Pratap, S. Agarwal and T. Meyarivan  “A fast and elitist multiobjective genetic algorithm: NSGA-II”  In _IEEE Transactions on Evolutionary Computation_ 6.2, 2002, pp. 182–197  DOI: 10.1109/4235.996017
  * [14] Thomas Auer and Martin Held  “Heuristics for the Generation of Random Polygons”  In _Canadian Conference on Computational Geometry_ , 1996  URL: https://api.semanticscholar.org/CorpusID:13524750
  * [15] Elliot Wright Hawkes et al.  “Grasping without squeezing: Design and modeling of shear-activated grippers”  In _IEEE Transactions on Robotics_ 34.2 IEEE, 2017, pp. 303–316