# Kathakali Hand Gesture Recognition With Minimal Data

Kavitha Raju
IIIT-Kottayam
Bridge Connectivity Solutions
<EMAIL_ADDRESS>
Nandini J. Warrier
IIIT-Kottayam
<EMAIL_ADDRESS>

###### Abstract

The Indian classical dance-drama Kathakali has a set of hand gestures called
Mudras, which form the fundamental units of all its dance moves and postures.
Recognizing the depicted mudra becomes one of the first steps in its digital
processing. The work treats the problem as a 24-class classification task and
proposes a vector-similarity-based approach using pose estimation, eliminating
the need for further training or fine-tuning. This approach overcomes the
challenge of data scarcity that limits the application of AI in similar
domains. The method attains 92% accuracy which is a similar or better
performance as other model-training-based works existing in the domain, with
the added advantage that the method can still work with data sizes as small as
1 or 5 samples with a slightly reduced performance. Working with images,
videos, and even real-time streams is possible. The system can work with hand-
cropped or full-body images alike. We have developed and made public a dataset
for the Kathakali Mudra Recognition as part of this work.

Kathakali Hand Gesture Recognition With Minimal Data

Kavitha Raju IIIT-Kottayam Bridge Connectivity Solutions
<EMAIL_ADDRESS>Nandini J. Warrier IIIT-Kottayam
<EMAIL_ADDRESS>

##  1 Introduction

This is a work attempting to bring the advancements we have in artificial
intelligence into the world of cultural heritage and knowledge, where it can
be put to use and benefit by solving real-world tasks especially in preserving
and handing down the ancient knowledge system we have been blessed with in
this modern era of digitization. Within the wide spectrum of AI, we now have
exceptional capabilities in processing images and videos using computer vision
techniques, handling vectorized data using dedicated database management
systems, etc, which we utilize here. The focus area we have chosen is the
Indian classical art form Kathakali, but the approaches we put forth can be
easily adapted in other similar use cases like different dance forms or even
be extended to processing sign languages. Our approach aims for the most cost
and resource-effective development, by building on top of existing
technologies, when the general trend in AI is the opposite, requiring large
amounts of data and computational capabilities.

Kathakali is a dance drama, where a story is acted out using an exquisite and
well-defined set of gestures and facial expressions, which originated in the
17th century. These gestures have evolved out of other ancient art forms that
prevailed even before Kathakali and the authoritative treatises on the Indian
dance such as Natyasatra, Abhinaya darpana, etc., which now forms the language
of Kathakali(VENU, 2000). On one hand, it has a language system that is
capable of expressing even the minute details of the story in a well-defined
and structured manner, whereas, on the other hand, we can observe the common
challenges of natural language processing like ambiguity, diverse dialects,
etc. here also when attempting computational modeling or processing. Even
outside of the computational aspects, the language of Kathakali is so
intricate and elaborate that it requires expert knowledge for a person to
understand the performance.

Mudras, or hand gestures, of Kathakali are like the alphabets of the language,
forming its fundamental building blocks. There are 24 such mudras, when used
in specific manners(body posture, combination and movements), which can be
interpreted as meaningful words of the language(Thampuran, 1892). Mudras are
formed by holding the fingers of a hand in a specific shape. These hand shapes
can be held in different angles and orientations or moved in different
directions and still be identified as the same Mudra. The concept of such
mudras is commonly seen in other Indian classical dance forms and they share
the majority of these features even though there might be slight differences
in the number of Mudras and their shapes.

The challenges posed by such a use case when attempting to process digitally
are many. It is a stage-performed art form and the raw data can be captured as
images and videos. From there to get to a state to automatically process and
understand the enacted story, there are the hurdles of detecting the artist,
tracking his finger, face, and body movements, recognizing the mudras and
words shown, interpreting the meaning of the combination and sequence of such
gestures to be able to translate it to a written language like English. To
list out the technology stack for such a pipeline, there will be computer
vision techniques like gesture recognition to natural language processing
techniques like language understanding and translation. All these are
essentially achieved today with the help of deep learning, which in turn
depends on a large amount of training data and infrastructures. Unfortunately,
such large training corpora will be hard to obtain for use cases like these.
Even if we manage to build some data in one such task, achieving something
similar in a different art form would again bring us to square one, which is
not helpful when we consider the ease of adaptation. An easy adaptation is
critical in this domain as here we aim to preserve knowledge systems and
cultures that are at the risk of neglect. So we need to be able to work with
as minimum data as possible.

Figure 1: Mudras in Kathakali: Our 24 classes(VENU, 2000)

Breaking down the bigger task of automatic Kathakali interpretation into
component tasks, a good place to start is by recognizing the Mudras as they
are the atomic units of the system. In this work we develop an approach to
detect mudras used in Kathakali, using only a very few samples to learn from.
This gives us the possibility of applying the same methods easily in different
dance forms or related areas like sign recognition by just building a
dictionary of one or more sample images to learn how the gesture is. This
approach not only would serve as the first step towards building an
interpretation system but also can be applied to; building more training data
from unannotated data like videos of performances, as a pedagogical tool
aiding digital training, etc.

Our methodology makes use of existing general-purpose pose estimation
technologies and uses that information to build vector representations of the
gesture/mudra classes we have. These vectors are formed using Euclidean
coordinates of different landmarks of the hand, after applying necessary
normalizations. Now the test data is also subjected to similar vectorization
and then compared against our database of known Mudra vectors to find the one
it is most similar to. The pose estimation model used is Mediapipe from Google
which can estimate 3D coordinates of hand landmarks from images, videos, and
even real-time feeds(Lugaresi et al., 2019).

##  2 Related Works

Kathakali, and traditional art forms in general, are not areas of wide
popularity, and hence technological research is scarce. That is indicative of
the relevance of such research explorations and the existing gap. Among the
existing works in Kathakali and other related fields, one common trend seen is
that all such attempts rely on building deep learning models from the ground
up by building datasets first which is the major challenge of this domain. Our
work on the other hand aims to use minimum data, leverage the powers of
existing models, and apply it to our task. Such an approach is novel in the
explored literature.

In the Kathakali Hand Gesture Recognition works done by Bhavanam et.
al.(Bhavanam and Iyer, 2020) in 2020, which is the same task as we are
attempting, they achieved an accuracy of 74% by applying Convolutional Neural
Networks on a dataset of 654 images cropped to show only one hand. They have
developed and published this dataset(Neelakanta Iyer, Ganesh; Bhavanam,
Lakshmi Tulasi, 2019) which comprises color images of 56x56 pixels. Our work
has also been tested on this dataset and observations are given in the results
section. As part of this work, authors have compared the performance of the
SVM classifier also on the same dataset and found CNN to be outperforming it.

In a similar work done by Malavath et. al. (Malavath and Devarakonda, 2023) in
2023 upon the same dataset as above they reported an accuracy of 79% accuracy
again using CNN models. In the work, they also provide a comparison with a
Naive Bayes model.

Bulani et. al. has a work that builds a Framework to Computationally Analyze
Kathakali Videos(Bulani et al., 2022) (2022). As part of this framework, they
build a pose estimation model that can detect body landmarks on a Kathakali
performance video. This is essential as the general purpose pose estimation
systems will not work on Kathakali images because of its vibrant costumes and
face painting. Our approach which relies on pose estimation to build vectors,
can build on top of such works when adapting to videos in costumes and facial
expression recognition. In their work they further explore the capabilities of
such a pose estimation system for style transfer and avatar puppeteering of
Kathakali model, synthesizing cartoon videos of Kathakali performance. This
work showcases the effectiveness of pose-estimation-based processing and its
scope. In our work, the pose estimation module we use is easily replaceable by
other similar technologies like these.

Iyer et. al. (Iyer and Bhardwaj, 2024) have conducted experimentation on
Kathakali character(vesham) recognition, which is a related problem in the
same area, but concerned with identifying the role of the performer based on
his attire. This is also treated as an image classification task using deep
learning techniques such as CNN and reports a top accuracy of 97.5% through
rigorous parameter tuning and experiments. Selvi et. al.’s work (Selvi et al.,
2021) in 2021, also deals with a different problem of expression detection in
the same domain of Kathakali Analysis.

Bharatanatyam is another classical dance form in India that has wider
popularity compared to Kathakali. It also has the concept of mudras. In a
recent work by Niveditha Parthasarathy et. al.(Parthasarathy and Palanichamy,
2023), they have published a benchmark dataset of 5 mudra classes and also
proposed a method for classification mudras using MobileNetV2 models after
finetuning. The work had real-time and offline performance tests. In the
offline tests where they used 6000 images for 5 classes, the reported accuracy
is 86.45. We have included experiments on this dataset as part of our work.

Another Bharathanatyam work by Vadakkot et. al. in 2023(Vadakkot et al.,
2023), reports an accuracy of 92% upon using CNN for Bharatanatyam mudra
recognition considering 29 classes of Mudras. Further, they also propose an
approach of using Eigen Mudra projections. As part of their work, they have
built a dataset of 68,073 high-resolution color images, from 6 subjects. The
higher accuracy of this work compared to the Kathakali work(Bhavanam and Iyer,
2020) could be attributed to the bigger dataset and better quality images,
which might have also required bigger computational resources. Finetuning of
pre-trained models like VGG19 is reported to have improved the performance to
94%, which shows the effectiveness of making use of general-purpose models.

At a similar task of Indian Classical dance mudra recognition, Kumar et. al.
in 2017(Kumar and Kishore, 2018) explored a different approach of using a
Histogram and other such features and employing an SVM classifier. It is also
a 24-class classification task for the dance form Kuchipudi. This work
utilizes the pre-deep-learning approaches of more mathematically aware feature
extraction but shows the effectiveness of working with interpretable features,
a smaller size dataset of 25 samples per class, and a simpler ML algorithm.
Their reported mudra recognition frequency is 89%.

Other works in Indian classical dance mudra recognition, like Pradeep et.
al.’s work in 2023(Pradeep et al., 2023) and Haridas et. al.’s work in
2022(Haridas and Bai V, 2022) also employ CNN. The latter used YOLO for
Bharathanatyam mudras and reported an accuracy of 73%. The former evaluates
multiple CNN models. This work also tries to locate hands in bigger images and
then classify them, which causes a lower accuracy of 50%. Other works like
Nandeppanavar et. al’s(Nandeppanavar et al., 2023) attempts using VGG-19 and
ResNet50V2 which reports 96.44% accuracy, also exhibit a similar pattern of
depending on deep learning architectures for either finetuning or training
from scratch.

The dance pose recognition works on K-Pop done by Dohyung Kim et. al. in
2017(Kim et al., 2017) utilize pose detection as their feature. But for pose
detection, they were using Kinect which requires a studio environment to
capture that data. But present-day advancements in pose estimation models that
can work on images and videos eliminate that limitation for us. They used a
dataset of 800 data points for 200 classes, which is 4 samples per class, and
indicates the effectiveness of the pose-estimation-based approach on a smaller
dataset. Multiple machine learning algorithms were tried on these features,
but they were all lighter techniques compared to deep learning, like SVM, KNN,
ELM, etc.

In a slightly more diverse application domain of pose recognition for human-
robot interaction, Qing Gao et. al. in their 2022 work(Gao et al., 2021),
adapts the use of hand pose estimation. Previously hand-glove-based approaches
have shown good performance in such tasks. Now those principles are applied
using pose estimates from images/videos via openpose(Cao et al., 2019). They
are using a 3D-CNN and an LSTM to further process these pose features for
their task and report the highest accuracy of 92.4%.

Other than the dependence on large training corpora, one common issue with
most of the CNN-based works explored above is that they are working with
cropped images of hands. When applying to a use case, we would expect the
model to classify mudras on video data, and scaling these approaches to such a
requirement will need considerable additional work in locating hands in the
bigger frames and also in processing continuous video data efficiently.

##  3 Methodology

###  3.1 Data

Good quality images or videos where hands showing mudra are clearly visible
are the data we can work with. The main advantage of our approach is its
ability to work with just a few samples of data. We tested the system using
just one sample per class and found that it can deliver reasonable results
even in such a constrained setting. With just one sample per class, the
accuracy would be lower but for applications where a small compromise on the
accuracy of each prediction doesn’t hurt too much, this gives us something to
work with. For instance, if we can consider a window of frames in a video or
depend on language models for getting a more probable sequence of mudra, a
small compromise on the individual prediction accuracy could be accommodated.

Upon increasing the number of samples to around 5 samples per class and
covering different orientations of the hand making sure different finger
positions are well covered, especially for mudras where some fingers would not
be visible in certain angles, we have observed a considerable increase in the
accuracy.

Another key highlight of the system is that even though training samples only
contain portions of images with just a hand, it can be applied to videos or
images where the full upper body or more background scenery is visible. As
long as the input data has enough clarity for the pose detection system to
correctly detect the hand landmarks, the training setup and testing setup are
not too dependent on each other.

###  3.2 Features

The raw images obtained from video frames, real-time feeds, or image datasets
are passed on to a pose estimation model and we obtain the hand landmarks
output of that model. In this implementation, we have relied on the media pipe
framework(Lugaresi et al., 2019) by Google and the hands model(med, ) they
provide. This enables us to get a jump start with respect to the computer
vision aspects of the task. Also helps in eliminating a lot of irrelevant
features like backgrounds, person-dependent features, etc, and focuses only on
what is relevant for us in our task, giving a more robust system deployable in
diverse settings than what it has been trained in. Mediapipe gives 3D
coordinates values of 21 landmarks of hands thus giving us a 63 dimension
feature vector for an image.

Figure 2: Handlandmarks from Mediapipe(med, )

The more common approach of using image pixel values as features for training
CNNs from scratch or fine-tuning pertained image classification neural
networks would all require large amounts of labeled training data to be able
to automatically deduce the relevant features and converge with reasonable
accuracy. But even then, those models will be highly sensitive to changes in
training and test data.

Another option commonly adopted while making use of existing pertained models
is to use the embeddings from those models as our features. But that, compared
to the coordinate values we use, lacks the advantage of being interpretable.
The coordinate values can be understood by us and subjected to post-processing
tasks like normalization as per our requirements, which we utilize in our
system.

###  3.3 Normalization

Once the x, y, and z coordinates are obtained with respect to the image frame,
we subject it to normalization. As we want the system to give a consistent
performance on images of different types where factors like zoom, location
within the frame, person-built, angle, etc could vary drastically, this
normalization step is crucial for us. This becomes even more critical as we
aim to give consistent performance but provide only very little data for it to
generalize over these diversities.

The normalization applied is the projection of the obtained coordinates to a
new coordinate system such that it rotates, scales, and translates the points
with respect to each other to a new location. For a 3-dimensional point this
can be achieved by multiplying it with a 4x4 transformation matrix(Strang,
2016).

The method adopted for obtaining this transformation matrix is to make use of
4 reference points, namely the wrist, the base points of the index finger, the
pinky finger, and the thumb. The first set of points are the four points we
have fixed at the center of the frame, facing frontward and of fixed body
size. The second set is the corresponding coordinate values obtained from pose
estimation for these body points. The first and second are compared to
determine the transformation required to project the second to the first
coordinate system.

Algorithm for Normalization
_Input_ :Original hand landmark coordinates.
_Output_ :Normalized coordinate values.

  1. 1.

Fix the 4 primary points $P$

    1. (a)

Wrist at $(px1,py1,pz1)$

    2. (b)

Index finger base at $(px2,py2,pz2)$

    3. (c)

Pinky finger base at $(px3,py3,pz3)$

    4. (d)

Thumb base at $(px4,py4,pz4)$

  2. 2.

Get the 4 secondary points from input, $S$

    1. (a)

$(sx1,sy1,sz1)\leftarrow$ Wrist (point 0 2)

    2. (b)

$(sx2,sy2,sz2)\leftarrow$ Index base (point 17)

    3. (c)

$(sx3,sy3,sz3)\leftarrow$ Pinky base (point 5)

    4. (d)

$(sx4,sy4,sz4)\leftarrow$ Thumb base (point 1)

  3. 3.

Compute transformation matrix $X=S^{-1}P$

  4. 4.

For each point in input hand landmarks

    1. (a)

$normalized\\_point=X*point$

  5. 5.

Return $normalized\\_landmarks$

Figure 3: Normalized coordinates of hand landmarks

The figure3 depicts the input images and their normalized coordinate values
for two different mudras in each column. It can be observed that the
difference in the size of the hand, the angle of the hand, the zoom
difference, the location of the hand in the image frame, etc get normalized
effectively to be able to find a close match correctly.

Once the transformation matrix is thus obtained, it is applied to all points
obtained from the model to project them into the normalized position, scale,
and orientation. This normalization of coordinate features is performed on
both training data(reference samples in the database) and incoming test data.
As these are simple mathematical computations they can be incorporated without
hampering performance in a real-time setting too.

Figure 4: Processes involved in Mudra Recognition

###  3.4 Database

Vector-based data processing is a field that has been studied for decades,
even though we have been seeing its increased popularity and the rise of
dedicated database management systems over the last few years. Now there are
multiple production-ready vector database management systems available,
capable of doing efficient similarity searches over large data collections(Pan
et al., 2023). Our use case requires us to store only about 24 (or a few times
24) data points and do a similarity search over it as we go for a few samples
per class approach. However, we have relied upon robust database systems that
are capable of indexing these vectors and scaling up easily to store more
samples as we would want to extend our data from mudras to more sophisticated
and varied content like words of Kathakali or other dance forms. For the
current implementation, we have used the pgvector(pgv, 2024) database which is
an extension of Postgres DB for vector-based computing. Each normalized
coordinate vector in the training data is stored in the database along with
its mudra label.

The use of a database system is not mandatory for the implementation of the
method. Alternatively using a K-Nearest Neighbour(KNN) implementation with n=1
(or appropriate values) or an Euclidean distance search implemented from
scratch will also provide a similar result as long as the training dataset is
small enough to be contained in memory.

###  3.5 Recognition

When new test samples are obtained, either from videos or images, these are
also normalized and checked against the data samples in the Database to find
those with the closest similarity. The similarity score used is Euclidean as
we are directly using the x,y, and z coordinates in Euclidean space. A
threshold is set to the similarity score to avoid very low matches.

The system can be modified to include top N (N=3, 5, etc) matches rather than
just one match if the use case permits it. For instance, if the mudra
prediction can be used as logits for a Language model to predict the correct
sequence used, considering more than one of the top predictions would be
helpful to deduce the more probable sequence. When running on videos, A window
of the last 10 frames can be taken and the most frequent match among these
N*10 values is taken as a valid prediction. This window-based approach allows
fewer fluctuations in prediction results.

##  4 Experiments and Results

We have conducted extensive experiments to compare the performance of our
proposed method with that of existing works as well as to test its robustness
upon throttling the data set size. The first set of experiments is conducted
upon publicly available datasets and we compare how our method performed
against the reported accuracies from other works which use deep learning model
training. The next two sets of experiments are done on our dataset by
considering different data splits like only 1 training sample, 5 training
samples, 10 training samples, and 80% training set.

The procedures followed in all the below-mentioned experiments are as per the
steps discussed in the methodology. Pose estimation is done for all the
training image samples, after which the coordinates values are normalized and
stored in the database with corresponding class labels. Similarly, test set
images are also subjected to pose estimation and normalization. After this, a
vector similarity search based on Euclidean distance is run on the database
and the topmost match is considered as the prediction. Even though there is no
machine learning or neural network training involved in our approach we still
refer to the reference data set as the training set in this article following
the common practice in related literature.

Exp | Dataset | Their | Our
---|---|---|---
No. | Description | Score | Score
1.1 | Prior Kathakali(Bhavanam and Iyer, 2020) | 74% | 0.83
1.2 | Partial Kathakali(Saurav and Afthab, 2021) | - | 0.92
1.3 | Bharathanatyam(Parthasarathy and Palanichamy, 2023) | 86.45 | 0.96
1.4 | Our Kathakali dataset | - | 0.92
Table 1: Performance on different public datasets Training Dataset | All 24 classes | Only 10 classes
---|---|---
Exp No. | Accuracy | Exp No. | Accuracy
1 sample | 2.1 | 0.63 | 3.1 | 0.81
5 samples | 2.2 | 0.75 | 3.2 | 0.91
10 samples | 2.3 | 0.83 | 3.3 | 0.91
80% of dataset | 2.4 | 0.92 | 3.4 | 0.95
Table 2: Performance on different training set sizes

### Experiment 1.1: Existing Kathakali Mudra Dataset

The dataset published in the work of Lakshmi Tulasi Bhavanam et.
al.(Neelakanta Iyer, Ganesh; Bhavanam, Lakshmi Tulasi, 2019) contains images
for 24 classes of Kathakali mudras, which they used in their CNN-based work.
We have used the dataset and with a random 80:20 split, performed
classification using our method. Upon examining the dataset it was noticed
that the images are of 56x56 pixel size and hence of very low quality. The
fingers are often indistinguishable and correct mudra recognition is difficult
even manually for some samples. As a result of low-quality images pose
estimation output is often incorrect in this dataset.

### Experiment 1.2: Partial Kathakali Mudra Dataset

A 5-class dataset is available on Kaggle(Saurav and Afthab, 2021) for
Kathakali mudras. It contains, ’mudrakhya’, ’pataka’, ’kataka’, ’kartari
mukha’, and ’musti’, which is a small subset of our 24 classes. Different hand
orientations including some ambiguous poses where all fingers are not visible
and hence mudra is not clear are included in the dataset. This dataset was
also used and with an 80:20 split on the data for the training and the test
sets, we applied our method for mudra classification.

### Experiment 1.3: Bharathanatyam Dataset

In the works of Niveditha Parthasarathy et. al, they have published a
benchmark dataset(Parthasarathy and Palanichamy, 2023) for Bharathanatyam
mudras, which includes short video clips for 5 mudra classes. Image frames are
obtained from these video clips at the rate of FPS=5. From the available
videos in the published dataset, we obtained 916 images for the experiment,
even though the original work has reported using 6000 images.

### Experiment 1.4: Our dataset

As part of this work, we have developed a 24-class dataset for Kathakali
Mudras involving 8 participants, different angles of hand, left and right
hands, different zoom, lighting, and such capturing conditions. In this
dataset also, a random 80:20 split was done for training and test sets and
subjected to performance testing using the proposed method.

### Experiment 2.x: On small training sets

To observe the performance of the approach in a data-constrained situation we
have throttled our training set size to extremely small sizes and tested it
against the remaining samples. The experiment procedures followed in this set
of experiments are also the same as the previous ones, which involved pose
estimation, normalization, and vector similarity-based prediction. The dataset
used is our dataset.
Experiment 2.1: One training sample per class
Experiment 2.2: 5 training samples per class
Experiment 2.3: 10 samples per class
Experiment 2.4: 80:20 split of the dataset (same experiment as 1.4)

### Experiment 3.x: Including only unambiguous classes

The hand shapes of certain pairs of mudras can be very similar to each other
like in the case of Tripataka and Anjlai or Mushti and Katakamukha. This can
cause a certain amount of ambiguity in the task of mudra recognition and can
have a major impact on the samples where hand orientation makes their
difference minimal. Several works (Parthasarathy and Palanichamy, 2023)
(Saurav and Afthab, 2021) explore the performance of their methods on a
simplified problem including only a subset of mudras that have substantial
differences in the hand shape. To explore this performance difference we have
reduced our dataset and task to a 10-class problem including only the
following 10 mudras: "pataka", "mudrakhya", "sukatunda", "hamsapaksha",
"sikhara", "ardhachandra", "mukura", "arala", "mukula", "katakamukha".
Experiment 3.1: One training sample per class
Experiment 3.2: 5 training samples per class
Experiment 3.3: 10 samples per class
Experiment 3.4: 80:20 split of the dataset

This dataset of 24 class Kathakali mudras which we developed as part of this
work is publicly available at [github or huggingface link]. All the data
splits mentioned in experiments 1.4, 2.x and 3.x on our dataset are also
published along with our data. The data split for some of the experiments has
a test set size which is several folds of the training set size which is not
common for Machine learning or deep learning evaluations. The evaluation we
have done so is because of the tightened data size constraints. The table:3
lists the sizes of training sets and test sets in each of the experiments.

##  5 Discussion

The experiment results clearly show that the proposed approach can attain the
performance of deep learning models trained or finetuned for specific tasks.
Furthermore, it can give satisfactory results even with very small data
available for training a baseline system in a new task. Though the work is
focused on Kathakali the proposed method can be applied to other similar tasks
easily as demonstrated by the Bharathanatyam mudra recognition experiments.

Exp | Experiment | Train set | Test set
---|---|---|---
No. | Description | Size | Size
1.1 | Prior Kathakali | 485 | 169
1.2 | Partial Kathakali | 991 | 245
1.3 | Bharathanatyam | 732 | 184
1.4 | 24-class 80:20 split | 742 | 186
2.1 | 24-class 1 sample | 24 | 904
2.2 | 24-class 5 samples | 120 | 808
2.3 | 24-class 10 samples | 240 | 688
2.4 | 24-class 80:20 split | 742 | 186
3.1 | 10-class 1 sample | 10 | 381
3.2 | 10-class 5 samples | 50 | 341
3.3 | 10-class 10 samples | 100 | 291
3.4 | 10-class 80:20 split | 315 | 76
Table 3: Sizes of training and test splits in the experiments

The experiments were done on isolated, cropped images to be able to compare
effectively with other works. The pre-trained pose detection model we use is
capable of performing good quality pose estimation on images with full body
and background details, continuous videos, real-time streams, and even on
mobile devices[cite]. As the additional computation of normalization and
vector similarity done on top of it are of negligible computational costs, the
method easily extends to a continuous, real-time, and cross-platform solution.

The application of Kathakali mudra detection comes as a first step towards
Kathakali interpretation. Kathakali’s interpretation has its relevance as the
performance includes signed dialogues that are not intelligible for a novice
in the audience. But applications of the mudra recognition system come in
other forms also. It can serve as a pedagogical tool applying the system to
detect select mudras and recommend ideal orientations. Another application is
in tagging unannotated data for creating larger corpora as our target domain
has the problem of data scarcity.

##  6 Conclusion

The main contribution of this work is the novel approach for image
classification tasks involving hand gestures such as dance mudra recognition
in Indian classical dance forms which can give satisfactory results even with
a very constrained dataset size. This is critical as there are several similar
applications for this in art forms and Sign languages where the foremost
constraint for deep learning approaches is their huge data requirements. Our
method could help with a jump start in such tasks which can later be employed
for creating more data from unannotated corpora.

One of the future directions we plan to explore is extending to Kathakali Word
recognition. A Kathakali word is formed using one or more of the mudras held
or moved in specified postures. These span over multiple frames. Our current
mudra recognition is limited to single frame level classification, but the
bigger task requires us to work with videos involving more context. This calls
for more challenges and exploration of solutions. We plan to employ full-body
pose estimation in place of only hands and figure out effective ways to
segment videos based on pose changes. Again we will aim to work with as
minimum data as possible.

Other future directions we have in mind are applying the same solutions to
different problems like Koodiyattam and other dance form mudra recognitions.
Apply similar techniques in word-level sign language recognition as in
Kathakali, to be able to adapt the solutions easily to different sign
languages and their variants with minimum data dependency.

##  7 Acknowledgments

We express our sincere thanks to volunteers from artists, students, and family
members who extended guidance and contributed to the building of the Kathakali
Mudra dataset which is being published as part of this work.

## References

  * [1] Hand landmarks detection guide | MediaPipe | Google for Developers.
  * pgv [2024] 2024\.  pgvector/pgvector.  Original-date: 2021-04-20T21:13:52Z.
  * Bhavanam and Iyer [2020] Lakshmi Tulasi Bhavanam and Ganesh Neelakanta Iyer. 2020.  On the classification of kathakali hand gestures using support vector machines and convolutional neural networks.  In _2020 International Conference on Artificial Intelligence and Signal Processing (AISP)_ , pages 1–6. IEEE.
  * Bulani et al. [2022] Pratikkumar Bulani, Sarath Sivaprasad, Vineet Gandhi, et al. 2022.  Framework to computationally analyze kathakali videos.
  * Cao et al. [2019] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. 2019.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Gao et al. [2021] Qing Gao, Yongquan Chen, Zhaojie Ju, and Yi Liang. 2021.  Dynamic hand gesture recognition based on 3d hand pose estimation for human–robot interaction.  _IEEE Sensors Journal_ , 22(18):17421–17430.
  * Haridas and Bai V [2022] Sneha Haridas and Ramani Bai V. 2022.  Detection and classification of indian classical bharathanatyam mudras using enhanced deep learning technique.  In _2022 International Conference on Innovations in Science and Technology for Sustainable Development (ICISTSD)_ , pages 18–23.
  * Iyer and Bhardwaj [2024] Ganesh Neelakanta Iyer and Sachit Bhardwaj. 2024.  Kathakali character identification–using deep learning techniques and web technologies for indian cultural heritage.  _Digital Applications in Archaeology and Cultural Heritage_ , 32:e00300.
  * Kim et al. [2017] Dohyung Kim, Dong-Hyeon Kim, and Keun-Chang Kwak. 2017.  Classification of k-pop dance movements based on skeleton information obtained by a kinect sensor.  _Sensors_ , 17(6):1261.
  * Kumar and Kishore [2018] KVV Kumar and PVV Kishore. 2018.  Indian classical dance mudra classification using hog features and svm classifier.  In _Smart Computing and Informatics: Proceedings of the First International Conference on SCI 2016, Volume 1_ , pages 659–668. Springer.
  * Lugaresi et al. [2019] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. 2019.  Mediapipe: A framework for building perception pipelines.
  * Malavath and Devarakonda [2023] Pallavi Malavath and Nagaraju Devarakonda. 2023.  Classification of kathakali asamyuktha hasta mudras using naive bayes classifier and convolutional neural networks.  In _Applied Computing for Software and Smart Systems: Proceedings of ACSS 2022_ , pages 131–144. Springer.
  * Nandeppanavar et al. [2023] Anupama S. Nandeppanavar, Shanta S. Kallur, Puneeth Thotad, and Vaishnavi A. Sankannavar. 2023.  Bharatanatyam hasta mudra categorization using deep learning approaches.  In _2023 IEEE North Karnataka Subsection Flagship International Conference (NKCon)_ , pages 1–6.
  * Neelakanta Iyer, Ganesh; Bhavanam, Lakshmi Tulasi [2019] Neelakanta Iyer, Ganesh; Bhavanam, Lakshmi Tulasi. 2019.  Kathakali Mudras.
  * Pan et al. [2023] James Jie Pan, Jianguo Wang, and Guoliang Li. 2023.  Survey of vector database management systems.  _arXiv preprint arXiv:2310.14021_.
  * Parthasarathy and Palanichamy [2023] Niveditha Parthasarathy and Yogesh Palanichamy. 2023.  Novel video benchmark dataset generation and real-time recognition of symbolic hand gestures in indian dance applying deep learning techniques.  _ACM Journal on Computing and Cultural Heritage_ , 16(3):1–19.
  * Pradeep et al. [2023] Radhika Pradeep, R Rajeshwari, VR Ruchita, Radhika Bubna, and HR Mamatha. 2023.  Recognition of indian classical dance hand gestures.  In _2023 International Conference on Inventive Computation Technologies (ICICT)_ , pages 814–820. IEEE.
  * Saurav and Afthab [2021] Sandra David Saurav, Sruthi and Mohammed Afthab. 2021.  Kathakali Mudras(5).
  * Selvi et al. [2021] C Selvi, Y Anvitha, CH Asritha, and PB Sayannah. 2021.  Kathakali face expression detection using deep learning techniques.  In _Journal of Physics: Conference Series_ , volume 2062, page 012018. IOP Publishing.
  * Strang [2016] Gilbert Strang. 2016.  _Introduction to Linear Algebra_ , 5 edition.  Wellesley-Cambridge Press.
  * Thampuran [1892] Kadathanattu Udyavarma Thampuran. 1892.  _Hasta Lakshana Deepika_.  Janran-jinee Achukoodam (Printers).
  * Vadakkot et al. [2023] Gayathri Vadakkot, Karthi Ramesh, and Govind Divakaran. 2023.  Automatic one-hand gesture (mudra) identification in bharatanatyam using eigenmudra projections and convolutional neural networks.  _Journal of Electronic Imaging_ , 32(2):023046–023046.
  * VENU [2000] G. VENU. 2000.  _The Language of Kathakali: Notations of 874 Hand Gestures_.  NATANA KAIRALI, KERALA.


# Generalized second law of thermodynamics in Gauss-Bonnet braneworld

Ahmad Sheykhi<EMAIL_ADDRESS>and Bin Wang<EMAIL_ADDRESS>1Department of Physics, Shahid Bahonar University, P.O. Box 76175, kerman,
Iran
2Research Institute for Astronomy and Astrophysics of Maragha (RIAAM),
Maragha, Iran
3 Department of Physics, Fudan University, Shanghai 200433, China

###### Abstract

We investigate the validity the generalized second law of thermodynamics in a
general braneworld model with curvature correction terms on the brane and in
the bulk, respectively. Employing the derived entropy expression associated
with the apparent horizon, we examine the time evolution of the total entropy,
including the derived entropy of the apparent horizon and the entropy of the
matter fields inside the apparent horizon. We show that the generalized second
law of thermodynamics is fulfilled on the $3$-brane embedded in the $5$D
spacetime with curvature corrections.

It was first pointed out in Jac  that the hyperbolic second order partial
differential Einstein equation has a predisposition to the first law of
thermodynamics. This profound connection between the first law of
thermodynamics and the gravitational field equations has been extensively
observed in various gravity theories Elin ; Cai1 ; Pad . Recently the study on
the connection between thermodynamics and gravity has been generalized to the
cosmological situations Cai2 ; Cai3 ; CaiKim ; Fro ; Wang , where it was shown
that the differential form of the Friedmann equation on the apparent horizon
in the FRW universe can be rewritten in the form of the first law of
thermodynamics. The extension of this connection has also been carried out in
the braneworld cosmology Cai4 ; Shey1 ; Shey2 . The deep connection between
the gravitational equation describing the gravity in the bulk and the first
law of thermodynamics on the apparent horizon reflects some deep ideas of
holography.

Besides examining the validity of the thermodynamical interpretation of
gravity by expressing the gravitational field equations into the first law of
thermodynamics on the apparent horizon in different spacetimes, it is of great
interest to examine other thermodynamical principles if the thermodynamical
interpretation of gravity really holds and is a generic feature. This is
especially interesting in the braneworld. In the braneworld, the gravity is no
longer Einstein gravity so that the horizon entropy does not satisfy the usual
area law. The entropy on the apparent horizon in the braneworld was extracted
through connecting the gravitational equation to the first law of
thermodynamics on the apparent horizon Shey1 ; Shey2 . Whether this obtained
entropy satisfies general thermodynamical principles is a question to be
asked. In this work we are going to investigate the generalized second law of
thermodynamics by examining the evolution of the apparent horizon entropy
deduced through the connection between gravity and the first law of
thermodynamics together with the matter fields’ entropy inside the apparent
horizon. If the thermodynamical interpretation of gravity is correct, the
extracted apparent horizon entropy from the profound connection between
gravitational equations and the first law of thermodynamics should satisfy the
generalized second law of thermodynamics.

The generalized second law of thermodynamics is an important principle in
governing the nature. Recently the generalized second law in the accelerating
universe enveloped by the apparent horizon has been investigated in wang1 ;
wang2 . Using the general expression of temperature at apparent horizon of FRW
universe, it has been shown that the generalized second law holds in Einstein,
Gauss-Bonnet and more general lovelock gravity akbar . Other studies on the
generalized second law of thermodynamics have been done in Pavon2 ; Other ;
Setare . Here we will consider a general braneworld models with correction
terms, such as a 4D scalar curvature from induced gravity on the brane, and a
5D Gauss-Bonnet curvature term in the bulk. With these correction terms,
especially including a Gauss-Bonnet correction to the 5D action, we have the
most general action with second-order field equations in 5D lovelock , which
provides the most general model for the braneworld scenarios RS ; DGP . In an
effective action approach to the string theory, the Gauss-Bonnet term
corresponds to the leading order quantum corrections to gravity, its presence
guarantees a ghost-free actionzwiebach . We will first adopt the method
developed in Shey2  to extract the entropy expression associated with the
apparent horizon from the first law of thermodynamics in Gauss-Bonnet
braneworld. We will check the time evolution of the total entropy, including
the entropy of the apparent horizon together with the entropy of the matter
fields inside the apparent horizon.

We start with the following action

$\displaystyle S$ $\displaystyle=$
$\displaystyle\frac{1}{2{\kappa_{5}}^{2}}\int{d^{5}x\sqrt{-{g}}\left({R}-2\Lambda+\alpha\mathcal{L}_{GB}\right)}$
$\displaystyle+\frac{1}{2{\kappa_{4}}^{2}}\int{d^{4}x\sqrt{-\widetilde{g}}\widetilde{R}}+\int{d^{4}x\sqrt{-\widetilde{g}}(\mathcal{L}_{m}-2\lambda)},$

where $\Lambda<0$ is the bulk cosmological constant and

$\displaystyle{\mathcal{L}}_{GB}=R^{2}-4R^{AB}R_{AB}+R^{ABCD}R_{ABCD},$ (1)

is the Gauss-Bonnet correction term. Quantities on the brane are expressed
with tilde to be distinguished from those in the bulk. $\kappa_{4}$ and
$\kappa_{5}$ are the gravitational constants on the brane and in the bulk and
can be written respectively as

$\kappa_{4}^{2}=8\pi G_{4},\ \ \kappa_{5}^{2}=8\pi G_{5}.$ (2)

The last term in the action corresponds to the matter content.
$\mathcal{L}_{m}$ is the Lagrangian density of the brane matter fields, and
$\lambda$ is the brane tension. Hereafter we assume that the brane
cosmological constant is zero (if it does not vanish, one can absorb it to the
stress-energy tensor of matter fields on the brane). We assume that there are
no sources in the bulk other than $\Lambda$.

The field equations can be obtained by varying the action with respect to the
bulk metric $g_{AB}$. The result is

$G_{AB}+\Lambda g_{AB}+2\alpha H_{AB}={\kappa_{5}}^{2}T_{AB}\delta(y),$

where we have assumed that the brane is located at $y=0$ with $y$ being the
coordinate in the extra dimension, and

$H_{AB}=RR_{AB}-2R_{A}{}^{C}R_{BC}-2R^{CD}R_{ACBD}+R_{A}{}^{CDE}R_{BCDE}-\textstyle{1\over
4}g_{AB}{\cal L}_{GB},$ (3)

is the second-order Lovelock tensor. The energy-momentum tensor can be
decomposed into two parts
${T}_{AB}=\widetilde{T}_{AB}-\frac{1}{{\kappa_{4}}^{2}}\widetilde{G}_{AB}$,
where $\widetilde{T}_{AB}$ is the energy-momentum tensor describing the matter
confined to the 3-brane which we assume in the form of a perfect fluid,
$\widetilde{T}_{AB}=(\rho+P)u_{A}u_{B}+P\widetilde{g}_{AB}$, in the homogenous
and isotropic universe on the brane, where $u^{A}$, $\rho$, and $P$, are the
fluid velocity ($u^{A}u_{A}=-1$), energy density and pressure respectively.
The possible contribution of nonzero brane tension $\lambda$ will be assumed
to be included in $\rho$ and $P$. A homogeneous and isotropic brane at fixed
coordinate position $y=0$ in the bulk is described by the line element

$ds^{2}=-N^{2}(t,y)dt^{2}+A^{2}(t,y)\gamma_{ij}dx^{i}dx^{j}+B^{2}(t,y)dy^{2},$

where $\gamma_{ij}$ is a maximally symmetric $3$-dimensional metric for the
surface ($t$=const., $y$=const.), whose spatial curvature is parameterized by
$k=-1,0,1$. On every hypersurface ($y$=const), we have the metric of a FRW
cosmological model. The metric coefficient $N$ is chosen so that, $N(t,0)=1$
and $t$ is the cosmic time on the brane. The presence of the four-dimensional
curvature scalar in the gravitational action does not affect the bulk
equations. If we define

$\Phi=\frac{1}{N^{2}}\frac{\dot{A}^{2}}{A^{2}}-\frac{1}{B^{2}}\frac{A^{\prime\,2}}{A^{2}}+\frac{k}{A^{2}}$
(4)

then the field equation reduces to germani

$\displaystyle\frac{\dot{A}}{A}\frac{N^{\prime}}{N}+\frac{A^{\prime}}{A}\frac{\dot{B}}{B}-\frac{\dot{A}^{\prime}}{A}=0,$
$\displaystyle\Phi+2\alpha\Phi^{2}+\frac{1}{\ell^{2}}-\frac{\mathcal{C}}{A^{4}}=0,$
(5)

where $\ell^{2}\equiv-6/\Lambda$ and $\mathcal{C}$ is an integration constant
which is related to the mass of the bulk black hole. In the above equations
dot and prime denote derivatives with respect to $t$ and $y$, respectively.
Integrating the $(00)$ component of the field equation across the brane and
imposing $\mathbb{Z}_{2}$ symmetry, we have the jump across the brane kofin

$\displaystyle\frac{2{\kappa_{4}}^{2}}{{\kappa_{5}}^{2}}\left[1+4\alpha\left(H^{2}+\frac{k}{a^{2}}-\frac{A^{\prime\,2}_{+}}{3a^{2}b^{2}}\right)\right]\frac{A^{\prime}_{+}}{ab}=-\frac{\kappa^{2}_{4}}{3}\rho+H^{2}+\frac{k}{a^{2}},$
(6)

where $a=A(t,0)$, $b=B(t,0)$ and $2A^{\prime}_{+}=-2A^{\prime}_{-}$ is the
discontinuity of the first derivative. $H=\dot{a}/a$ is the Hubble parameter
on the brane. Inserting $\Phi$ into Eq. (6) we can obtain the generalized
Friedmann equation on the brane kofin

$\displaystyle\epsilon\frac{2{\kappa_{4}}^{2}}{{\kappa_{5}}^{2}}\left[1+\frac{8}{3}\alpha\left(H^{2}+{k\over
a^{2}}+{\Phi_{0}\over 2}\right)\right]\left(H^{2}+{k\over
a^{2}}-\Phi_{0}\right)^{1/2}=-\frac{\kappa^{2}_{4}}{3}\rho+H^{2}+{k\over
a^{2}},$ (7)

where $\Phi_{0}=\Phi(t,0)$ and $\epsilon=\pm 1$. For later convenience we
choose $\epsilon=-1$. This Friedmann equation contains various special cases
discussed extensively in the literature. The DGP braneworld is the limiting
case when $\alpha=0$, while the RS II braneworld can be reproduced in the
limit $\kappa_{4}\to\infty$ and $\alpha=0$. The pure Gauss-Bonnet braneworld
is the case with $\kappa_{4}\to\infty$. To have further understanding about
the nature of the apparent horizon we rewrite more explicitly, the metric of
homogenous and isotropic FRW universe on the brane in the form

$ds^{2}={h}_{\mu\nu}dx^{\mu}dx^{\nu}+\tilde{r}^{2}d{\Omega_{2}}^{2},$ (8)

where $\tilde{r}=a(t)r$, $x^{0}=t,x^{1}=r$, the two-dimensional metric
$h_{\mu\nu}$=diag $(-1,a^{2}/(1-kr^{2}))$, and $d\Omega_{2}$ is the metric of
two-dimensional unit sphere. Then, the dynamical apparent horizon, a
marginally trapped surface with vanishing expansion, is determined by the
relation $h^{\mu\nu}\partial_{\mu}\tilde{r}\partial_{\nu}\tilde{r}=0$, which
implies that the vector $\nabla\tilde{r}$ is null on the apparent horizon
surface. The apparent horizon has been argued to be a causal horizon for a
dynamical spacetime and is associated with gravitational entropy and surface
gravity Hay2 ; Bak . The explicit evaluation of the apparent horizon for the
FRW universe gives the apparent horizon radius

$\tilde{r}_{A}=\frac{1}{\sqrt{H^{2}+k/a^{2}}}.$ (9)

The associated surface gravity on the apparent horizon can be defined as

$\kappa=\frac{1}{\sqrt{-h}}\partial_{a}\left(\sqrt{-h}h^{ab}\partial_{ab}\tilde{r}\right).$
(10)

Then one can easily show that the surface gravity at the apparent horizon of
FRW universe can be written as

$\kappa=-\frac{1}{\tilde{r}_{A}}\left(1-\frac{\dot{\tilde{r}}_{A}}{2H\tilde{r}_{A}}\right).$
(11)

The associated temperature on the apparent horizon can then be defined as

$T_{h}=\frac{|\kappa|}{2\pi}=\frac{1}{2\pi\tilde{r}_{A}}\left(1-\frac{\dot{\tilde{r}}_{A}}{2H\tilde{r}_{A}}\right).$
(12)

where $\frac{\dot{\tilde{r}}_{A}}{2H\tilde{r}_{A}}<1$ ensures that the
temperature is positive. Recently the connection between temperature on the
apparent horizon and the Hawking radiation has been observed in cao . Hawking
radiation is an important quantum phenomenon of black hole, which is closely
related to the existence of event horizon of black hole. The cosmological
event horizon of de Sitter space has the Hawking radiation with thermal
spectrum as well. Using the tunneling approach proposed by Parikh and Wilczek,
the authors of cao  showed that there is indeed a Hawking radiation with a
finite temperature, for locally defined apparent horizon of Friedmann-
Robertson-Walker universe with any spatial curvature. This gives more solid
physical implication of the temperature associated with the apparent horizon.

Now we turn to apply these results to investigate the validity of the
generalized second law of thermodynamics in the Gauss-Bonnet braneworld
scenario. We apply the approach we developed in Shey1  to find the expression
of entropy associated with the horizon geometry in Gauss-Bonnet braneworld.
First of all, we assume that there is no black hole in the bulk and thus
$\mathcal{C}=0$. Inserting this condition into Eq. (5), we get
$\Phi=\Phi_{0}=\frac{1}{4\alpha}\left(-1+\sqrt{1-\frac{8\alpha}{\ell^{2}}}\right)=\rm{const}.$
This condition also implies $\alpha<\ell^{2}/8$. In terms of the apparent
horizon radius, we can rewrite the Friedmann equation (7) into

$\displaystyle\rho$ $\displaystyle=$ $\displaystyle\frac{3}{8\pi
G_{4}}\frac{1}{{\tilde{r}_{A}}^{2}}+\frac{3}{4\pi
G_{5}}\left(\frac{1}{{\tilde{r}_{A}}^{2}}-\Phi_{0}\right)^{1/2}\times\left[1+\frac{8}{3}\alpha\left(\frac{1}{{\tilde{r}_{A}}^{2}}+{\Phi_{0}\over
2}\right)\right].$ (13)

where we have used Eq. (2). Now, differentiating Eq. (13) with respect to the
cosmic time and using the continuity equation

$\dot{\rho}+3H(\rho+P)=0,$ (14)

we get

$\dot{\tilde{r}}_{A}=4\pi
H(\rho+P){\tilde{r}_{A}^{2}}\left[\frac{1}{G_{4}\tilde{r}_{A}}+\frac{1}{G_{5}\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}+\frac{4\alpha}{G_{5}\tilde{r}_{A}^{2}}\left(\frac{2-\Phi_{0}\tilde{r}_{A}^{2}}{\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}\right)\right]^{-1}.$
(15)

One can see from the above equation that $\dot{\tilde{r}}_{A}>0$ provided that
the dominant energy condition, $\rho+P>0$, holds. In our previous work Shey2 ,
we showed that the Friedmann equation in a general braneworld model with
curvature correction terms can be written in the form of the first law of
thermodynamics on the apparent horizon of the brane,

$dE=T_{h}dS_{h}+WdV,$ (16)

where $W=(\rho-P)/2$ is the matter work density Hay2  which is regarded as the
work done by the change of the apparent horizon, $E=\rho V$ is the total
energy of the matter fields on the brane inside a 3-ball with the volume
$V=\Omega_{3}\tilde{r}_{A}^{3}$. The area of the apparent horizon is the area
of the ball on a 2-sphere of radius $\tilde{r}_{A}$, which is expressed as
$A=3\Omega_{3}\tilde{r}_{A}^{2}$ TT . Using the first law we can extract the
expression for the entropy on the apparent horizon in the general Gauss-Bonnet
braneworld Shey2

$\displaystyle S_{h}$ $\displaystyle=$
$\displaystyle\frac{3\Omega_{3}}{2G_{4}}{\displaystyle\int^{\tilde{r}_{A}}_{0}\tilde{r}_{A}d\tilde{r}_{A}}+\frac{3\Omega_{3}}{2G_{5}}{\displaystyle\int^{\tilde{r}_{A}}_{0}\frac{\tilde{r}_{A}^{2}d\tilde{r}_{A}}{\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}}$
(17) $\displaystyle+$
$\displaystyle\frac{6\alpha\Omega_{3}}{G_{5}}{\displaystyle\int^{\tilde{r}_{A}}_{0}\frac{2-\Phi_{0}\tilde{r}_{A}^{2}}{\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}d\tilde{r}_{A}}.$

The explicit form of the entropy can be obtained by integrating (17), which
reads

$\displaystyle S_{h}$ $\displaystyle=$
$\displaystyle\frac{3\Omega_{3}{\tilde{r}_{A}}^{2}}{4G_{4}}+\frac{2\Omega_{3}{\tilde{r}_{A}}^{3}}{4G_{5}}\times{}_{2}F_{1}\left(\frac{3}{2},\frac{1}{2},\frac{5}{2},\Phi_{0}{\tilde{r}_{A}}^{2}\right)$
(18)
$\displaystyle+\frac{6\alpha\Omega_{3}{\tilde{r}_{A}}^{3}}{G_{5}}\left[\Phi_{0}\times{}_{2}F_{1}\left(\frac{3}{2},\frac{1}{2},\frac{5}{2},\Phi_{0}{\tilde{r}_{A}}^{2}\right)+\frac{\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}{{\tilde{r}_{A}}^{2}}\right],$

where ${}_{2}F_{1}(a,b,c,z)$ is a hypergeometric function. The expression
looks complicated. But its physical meaning is clear and includes various
special cases Shey2 . Some remarks are as follows. The first term in (18) has
the form of Bekenstein-Hawking entropy of the 4D Einstein gravity. In fact,
the Einstein-Hilbert term ($\tilde{R}$) on the brane contributes to this area
term. The second term is due to the Einstein-Hilbert term ($R$) in the bulk.
The Gauss-Bonnet correction term in the bulk contributes to the last term. In
the limit $\alpha\rightarrow 0$, this expression reduces to the entropy of
apparent horizon in warped DGP braneworld embedded in a $AdS_{5}$ bulk, while
in the limit $\alpha\rightarrow 0$ and $\Phi_{0}\rightarrow 0$ (or equality
$\ell\rightarrow\infty$), it reduces to the entropy of apparent horizon in
pure DGP braneworld with a Minkowskian bulk Shey1 . Furthermore, taking the
limit $\alpha\rightarrow 0$ and $G_{4}\rightarrow\infty$, while keeping
$G_{5}$ finite, the first and the last terms in (18) vanish and we obtain the
entropy associated with the apparent horizon in RSII braneworld Shey1 .
Finally, keeping $\alpha$ finite, and taking the limit
$G_{4}\rightarrow\infty$ and $\Phi_{0}\rightarrow 0$, one can extract from Eq.
(18) the entropy associated with the apparent horizon on the brane in the
Gauss-Bonnet braneworld with a Minkowskian bulk

$\displaystyle
S_{h}=\frac{2\Omega_{3}{\tilde{r}_{A}}^{3}}{4G_{5}}\left(1+\frac{12\alpha}{{\tilde{r}_{A}}^{2}}\right).$
(19)

This gives an expression of horizon entropy in the Gauss-Bonnet gravity GB ;
Cai2 ; Cai3 ; CaiKim ; Cvetic . This is an expected result. Indeed, because of
the absence of scalar curvature term on the brane and the negative
cosmological constant in the bulk, no localization of gravity happens on the
brane. As a result, the gravity on the brane is still $5$D Gauss-Bonnet
gravity and the brane looks like a domain wall moving in a Minkowski
spacetime. Therefore, the entropy of apparent horizon on the brane still obeys
the entropy formula in the bulk. Note that the factor $2$ in the entropy comes
from the $\mathbb{Z}_{2}$ symmetry of the bulk. These results give a self-
consistency check for the entropy expression (18).

Next we turn to find out $T_{h}\dot{S_{h}}$. It is a matter of calculation to
show that

$T_{h}\dot{S_{h}}=\frac{3\Omega_{3}}{4\pi}\left(1-\frac{\dot{\tilde{r}}_{A}}{2H\tilde{r}_{A}}\right)\left[\frac{1}{G_{4}}+\frac{\tilde{r}_{A}}{G_{5}\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}+\frac{4\alpha}{G_{5}\tilde{r}_{A}}\left(\frac{2-\Phi_{0}\tilde{r}_{A}^{2}}{\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}\right)\right]\dot{\tilde{r}_{A}}.$
(20)

Substituting Eq. (15) and doing some simplifications, we obtain

$T_{h}\dot{S_{h}}=3\Omega_{3}H(\rho+P){\tilde{r}_{A}}^{3}\left(1-\frac{\dot{\tilde{r}}_{A}}{2H\tilde{r}_{A}}\right).$
(21)

As we argued above that the physical positive temperature $T_{h}$ requires the
term $\left(1-\frac{\dot{\tilde{r}}_{A}}{2H\tilde{r}_{A}}\right)$ to be
positive. However, in the accelerating universe the dominant energy condition
may be violated, $\rho+P<0$, which can make the second law of thermodynamics
,$\dot{S_{h}}\geq 0$, do not hold in general. Then the question arises, “will
the generalized second law of thermodynamics, $\dot{S_{h}}+\dot{S_{m}}\geq 0$,
still holds on the brane?” According to the generalized second law the entropy
of matter fields inside the horizon together with the entropy on the apparent
horizon cannot decrease with time. The entropy of the matter fields inside the
apparent horizon, $S_{m}$, can be related to its energy $E=\rho V$ and
pressure $P$ in the horizon by the Gibbs equation Pavon2

$T_{m}dS_{m}=d(\rho V)+PdV=Vd\rho+(\rho+P)dV,$ (22)

where $T_{m}$ is the temperature of the energy inside the horizon. We limit
ourselves to the assumption that the thermal system enveloped by the apparent
horizon remains in equilibrium so that the temperature of the system must be
uniform and the same as the temperature of its boundary. This requires that
the temperature $T_{m}$ of the energy inside the apparent horizon should be in
equilibrium with the temperature $T_{h}$ associated with the apparent horizon,
so we have $T_{m}=T_{h}$. This expression holds in the local equilibrium
hypothesis. If the temperature of the fluid differs much from that of the
horizon, there will be spontaneous heat flow between the horizon and the fluid
inside and the local equilibrium hypothesis will no longer hold. Therefore
from the Gibbs equation (22) we can obtain

$T_{h}\dot{S_{m}}=(n-1)\Omega_{n-1}{\tilde{r}_{A}}^{n-2}\dot{\tilde{r}}_{A}(\rho+P)-(n-1)\Omega_{n-1}{\tilde{r}_{A}}^{n-1}H(\rho+P).$
(23)

To check the generalized second law of thermodynamics, we have to examine the
evolution of the total entropy $S_{h}+S_{m}$. Adding equations (21) and (23),
we get

$T_{h}(\dot{S_{h}}+\dot{S_{m}})=\frac{3\Omega_{3}}{2}{\tilde{r}_{A}}^{2}\dot{\tilde{r}}_{A}(\rho+P)=\frac{A}{2}(\rho+P)\dot{\tilde{r}}_{A}.$
(24)

Substituting $\dot{\tilde{r}}_{A}$ from Eq. (15) into (24) we get

$T_{h}(\dot{S_{h}}+\dot{S_{m}})=2\pi
A{\tilde{r}_{A}}^{2}H(\rho+P)^{2}\left[\frac{1}{G_{4}\tilde{r}_{A}}+\frac{1}{G_{5}\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}+\frac{4\alpha}{G_{5}\tilde{r}_{A}^{2}}\left(\frac{2-\Phi_{0}\tilde{r}_{A}^{2}}{\sqrt{1-\Phi_{0}\tilde{r}_{A}^{2}}}\right)\right]^{-1}.$
(25)

It is worth noticing that $1-\Phi_{0}\tilde{r}_{A}^{2}>0$, therefore the right
hand side of the above equation cannot be negative throughout the history of
the universe. Hence we have $\dot{S_{h}}+\dot{S_{m}}\geq 0$ indicating that
the generalized second law of thermodynamics is fulfilled in a general
braneworld model with curvature correction terms in a region enclosed by the
apparent horizon.

In summary, we have investigated the validity of the generalized second law of
thermodynamics in a general braneworld model with curvature correction terms,
such as a 4D scalar curvature from induced gravity on the brane, and a 5D
Gauss-Bonnet curvature term in the bulk. Following the method developed in
Shey2  we have extracted the entropy associated with the apparent horizon in a
general Gauss-Bonnet braneworld by relating the gravitational equation to the
first law of thermodynamics on the apparent horizon. We have examined the
total entropy evolution with time, including the derived apparent horizon
entropy and the entropy of matter fields inside the apparent horizon on the
brane. We have shown that the generalized second law of thermodynamics on the
$3$-brane embedded in the $5$D spacetime with curvature correction terms is
fulfilled throughout the history of the universe. The satisfaction of the
generalized second law of thermodynamics provides further confidence on the
thermodynamical interpretation of gravity.

###### Acknowledgements.

This work has been supported financially by Research Institute for Astronomy
and Astrophysics of Maragha, Iran. The work of B. W. was partially supported
by NNSF of China and Shanghai Science and Technology Commission and Shanghai
Education Commission.

## References

  * (1) T. Jacobson, Phys. Rev. Lett. 75, 1260 (1995).
  * (2) C. Eling, R. Guedens, and T. Jacobson, Phys. Rev. Lett. 96, 121301 (2006).
  * (3) M. Akbar and R. G. Cai, Phys. Lett. B 635, 7 (2006) ;
M. Akbar and R. G. Cai, Phys. Lett. B 648, 243 (2007).

  * (4) T. Padmanabhan, Class. Quant. Grav. 19, 5387 (2002);
T. Padmanabhan, Phys. Rept. 406, 49 (2005);
T. Padmanabhan, gr-qc/0606061;
A. Paranjape, S. Sarkar and T. Padmanabhan, Phys. Rev. D 74, 104015 (2006);
D. Kothawala, S. Sarkar and T. Padmanabhan, gr-qc/0701002;
T. Padmanabhan and A. Paranjape, Phys. Rev. D 75 (2007) 064004;
S. F. Wu, B. Wang, and G. H Yang, Nucl. Phys. B 799 (2008) 330.

  * (5) M. Akbar and R. G. Cai, Phys. Rev. D 75, 084003 (2007).
  * (6) R. G. Cai and L. M. Cao, Phys.Rev. D 75, 064008 (2007).
  * (7) R. G. Cai and S. P. Kim, JHEP 0502, 050 (2005).
  * (8) A. V. Frolov and L. Kofman, JCAP 0305, 009 (2003);
U. K. Danielsson, Phys. Rev. D 71, 023516(2005) ;
R. Bousso, Phys. Rev. D 71, 064024 (2005);
G. Calcagni, JHEP 0509, 060 (2005).

  * (9) B. Wang, E. Abdalla and R. K. Su, Phys.Lett. B 503, 394 (2001);
B. Wang, E. Abdalla and R. K. Su, Mod. Phys. Lett. A 17, 23 (2002);
R. G. Cai and Y. S. Myung, Phys. Rev. D 67, 124021 (2003).

  * (10) R. G. Cai and L. M. Cao, Nucl. Phys. B 785 (2007) 135.
  * (11) A. Sheykhi, B. Wang and R. G. Cai, Nucl. Phys. B 779 (2007)1.
  * (12) A. Sheykhi, B. Wang and R. G. Cai, Phys. Rev. D 76 (2007) 023515.
  * (13) Bin Wang, Yungui Gong, Elcio Abdalla, Phys.Rev. D 74 (2006) 083520.
  * (14) Jia Zhou, Bin Wang, Yungui Gong, Elcio Abdalla, Phys. Lett. B 652 (2007) 86.
  * (15) M. Akbar, gr-qc/0808.0169;
M. Akbar, gr-qc/0808.3308.

  * (16) G. Izquierdo and D. Pavon, Phys. Lett. B 633 (2006) 420\.
  * (17) E. Babichev, V. Dokuchaev, Yu. Eroshenko, Phys. Rev. Lett. 93 (2004) 021102;
M. D. Pollock, T. P. Singh, Class. Quantum Grav. 6 (1989) 901;
P. C. W. Davies, Class. Quantum Grav. 4 (1987) L225;
Izquierdo, D. Pavon, Phys. Lett. B 639 (2006) 1.

  * (18) M. R. Setare and S. Shafei, JCAP 09 (2006) 011;
M. R. Setare, Phys. Lett. B 641 (2006) 130;
H. Mohseni Sadjadi, Phys. Rev. D 73 (2006) 063525;
H. Mohseni Sadjadi, Phys. Rev. D 76 (2007) 104024;
H. Mohseni Sadjadi, Phys. Lett. B 645 (2007) 108;
S. F. Wu, B. Wang, G. H. Yang and P. M. Zhang, arXive: 0801.2688.

  * (19) D. Lovelock, J. Math. Phys. 12, 498 (1971).
  * (20) L. Randall, R. Sundrum, Phys. Rev. Lett. 83 (1999) 3370;
L. Randall, R. Sundrum, Phys. Rev. Lett. 83 (1999) 4690\.

  * (21) G. Dvali, G. Gabadadze, M. Porrati, Phys. Lett. B 485, 208 (2000) ;
G. Dvali, G. Gabadadze, Phys. Rev. D 63 065007 (2001).

  * (22) B. Zwiebach, Phys. Lett. B 156, 315 (1985);
B. Zumino, Phys. Rept. 137, 109 (1986);
D. J. Gross and J.H. Sloan, Nucl. Phys. B291, 41 (1987).

  * (23) R.G. Cai, L.M. Cao, Y.P. Hu, arXiv:0809.1554;
R. Li, J. R. Ren, D. F. Shi, Phys. Lett. B 670 (2009) 446.

  * (24) C. Germani and C. Sopuerta, Phys. Rev. Lett. 88, 231101 (2002).
  * (25) G. Kofinas, R. Maartens, E. Papantonopoulos, JHEP 0310, 066 (2003) .
  * (26) S. A. Hayward, S.Mukohyana, andM. C. Ashworth, Phys. Lett. A 256, 347 (1999);
S. A. Hayward, Class. Quantum Grav. 15, 3147 (1998).

  * (27) D. Bak and S. J. Rey, Class. Quantum Grav. 17, L83 (2000).
  * (28) $\Omega_{3}=\frac{\pi^{3/2}}{\Gamma(5/2)}=4\pi/3$. For the benefit to generalize our results to the $(n-1)$-brane embedded in the $(n+1)$-dimensional spacetime, we use $\Omega_{3}$ here.
  * (29) R. C. Myers and J. Z. Simon, Phys. Rev. D 38, 2434 (1988);
R. G. Cai, Phys. Rev. D 65, 084014 (2002) ;
R. G. Cai and Q. Guo, Phys. Rev. D 69, 104025 (2004).

  * (30) M. Cvetic, S. Nojiri, S. D. Odintsov, Nucl. Phys. B 628, 295, (2002).


# “Vironment”:
An Art of Wearable Social Distancing

Steve Mann  Cayden Pierce  Christopher Tong  Christina Mann We wish to thank
AMD, SwimOP, and Swim Drink Fish

###### Abstract

“Vironment” is a series of art pieces, social commentary, technology, etc.,
based on wearable health technologies of social-distancing, culminating in a
social-distancing device that takes the familiar world of security and
surveillance technologies that surround us and re-situates it on the body of
the wearer (technologies that become part of us). This piece also introduces a
conceptual framework for (1) the sensing of the self together with (2) sensing
of others and (3) sensing of the environment around us.

wearables, health, wearable computing, signal processing, signal
reconstruction, high-dynamic-range (HDR), self-sensing, vironment,
invironment, environment, sonar, wearable sensing, facial recognition,
surveillance, sousveillance

Figure 1: 3D printed version of Vironment 3.0 with rigidly aligned sensors and
OpenEyeTap computer vision and display system.

##  1 Background and Introduction

Invironment is a new concept defined in contrast to the environment. Whereas
the environment is our surroundings, the invironment is us, ourselves, and in
particular, includes that part of us that we consider to be ourselves, e.g.
our shoes, clothing, eyeglasses, and the like, along with a certain space, or
social-distance around us, often defined in the context of wearable
computing[1, 2]. The concept of invironment is of particular relevance in the
era of health pandemics and pandemic awareness as global society asks
questions about health, safety, wellness, and the individual’s right to
utilize technologies to assist in these areas[3, 4, 5, 6]. Technologies for
health and wellness are rapidly becoming viewed as extensions of one’s body,
which raises questions about human rights to technology, the human condition,
and the extension of “the self [7]” into the space around the body.
Specifically, with the introduction of social-distancing, questions arise
regarding the area in which “the self” exists and where “the self” ends.

In further inquiry of these concepts, a series of interactive art pieces are
constructed to highlight the good and bad of social-sistancing, i.e. some of
its benefits, downsides, and absurdities of social distancing, based on the
idea of a social-distancing necklace, inspired by the commonly worn spiked
necklace shown in Fig. 3. We develop wearable computing systems which promote
and consider the health practice of social distancing from a health and
wellness perspective, a realistic sociological perspective, and a
technological perspective.

Surveillance is well-known in the areas of smart buildings, smart streets,
smart cities, etc. from an “Internet of Things” (IoT) perspective. This is the
traditional situation in which sensors exist in the environment around us.
More recently, “wearables” (wearable technology) has emerged as a new
discipline in which sensors are affixed to people rather than things [8, 9,
2], giving rise to “WearableAI[1]” (Wearable Artificial Intelligence”). What
is most important about wearable technology is not so much the proximity to
the body, but, more importantly, the ability for this technology to function
as an agent of the mind and body’s freewill and self-determination over own
one’s own destiny, i.e. “sousveillance”[10, 11, 12, 13, 14, 15, 16, 17]
(inverse surveillance, sensors-on-people) and self-sensing e.g. Quantified
Self-Sensing (QSS)[18, 19], as outlined in Fig 1 of the ITTI paper[20].

Here, we present a series of art pieces, social commentary, technology,
inventions, etc. which raise questions deeply connected to the paradigm shifts
of social-distancing and wearable computing and how they interact.

##  2 Self and Technology, Society, and Environment

In the past cities were more important than countries in terms of boundaries,
e.g. walled cities thousands of years ago. Next countries emerged as
important. Finally in the era of global pandemics, we’re seeing world
governance, and the reduced autonomy of countries. What matters now is
clothes, i.e. our individual selves become in some sense the boundary of
greatest importance. So in regards to “crossing borders”, we’ve witnessed the
evolution from cities, to countries, to clothes.

Vironment is an exploration of the following three elements:

  * •

Self and technology (e.g. the combination of human and machines, “cyborg”,
“humachine”, augmented human, etc.);

  * •

Self and society (interaction between humans, augmented humans, etc.);

  * •

Self and the environment (interaction between the augmented human and the
natural or built environment, e.g. cyborg-city interaction, etc.)[21].

##  3 Vironment 1.0/Social-Bubble

Vironment is a series of art installations, design interventions, and
inventions aimed at understanding and reconstructing social distance as the
sometimes soft and fuzzy boundary between the invironment and the environment.
Through these creations arises a deeper inquiry into the relationship between
the body and the spacial sphere around the body, which together form much of
the concept of “self”. The design pieces presented here explore the extension
of the self into the environment, an area of space which has become of utmost
social significance in the era of health pandemics as individuals individuate
and define themselves by the “social bubble” that exists around them.

One piece we name “Vironment/Social-Bubble”, or, simply, “Vironment 1.0”, is
shown in Fig 2. It is based on the use of a ball commonly used as a water-
based amusement ride, called the “waterball ride” or “water ball ride”. In the
context of a vessel surrounding the rider, when used on water, it represents
an example of the interaction between humans, water, and technology. (The
field of “WaterHCI = Water-Human-Computer Interaction” originated in Ontario
in the 1960s and 1970s, and relates to the concept of “Fluidic User
Interface”[22]).

Here we explore the use of such a ball anywhere, such as on land, and not just
on water only.

Vironment 1.0 takes the idea of social-distancing quite literally. Many of the
parks in our neighbourhood post signs warning us to keep our social distance
of two metres or six feet. Vironment 1.0 is a two metre diameter plastic
bubble in which a participant is placed, and the bubble is filled through a
HEPA (High Efficiency Particulate Air) filter connected to a blower supplied
by a solar-powered battery system. The filter and blower assembly was
constructed from an automobile heater blower and runs on a small 12 volt
battery which is charged by solar power.

Two individuals crossing paths will be forced to maintain a proper 2m social
distance. This demonstrates an ideal situation for an individual wishing to
avoid contact with pandemic diseases by providing the wearer with clean,
filtered air and a visible “social bubble” barrier. Also demonstrated is the
inherent impossibility of social-distancing in some situations, as the
individual in the bubble cannot fit through doorways, or even walk down a
narrow sidewalk whilst encapsulated in an impenetrable social bubble.

This piece (Vironment 1.0) raises questions about what space “belongs” to an
individual. Does an individual encompass only their brain? Their whole body?
The brain, body, and the clothing on the body, and maybe also some of the
space that surrounds the body and its clothes? This is a vital question to ask
in times when the health and well-being of individuals and society is
predicated on everyone existing in and maintaining their own invironmental
space. Vironment 1.0 serves as a visceral example and visualization of the
social space that has become normal (and often mandatory) in the era of health
pandemics and pandemic awareness.

Figure 2: “Vironment/Social-Bubble”, or simply “Vironment 1.0” is a social
commentary on social-distancing to help us rethink the natural and social
environment and our natural and social ecosystems in times of pandemics.

##  4 Vironment 2.0

Vironment 2.0 borrows from a theatre of the absurd and the détournement
(Situationist) visual art tradition, as something so hideous as to hopefully
make itself unnecessary once it has raised the awareness needed to make it
unnecessary. It is important to note that spiked chokers are not very socially
acceptable to begin with. Here the location of this wearable compliments the
off-putting aesthetic[23].

Vironment 2.0 is an extension of the common spiked necklace, in which the
spikes are simply made longer, as shown in Fig. 4. The necklace was 3D printed
and the spike extensions were cut from steel wire, tipped with the same spikes
as in a common spiked necklace.

This piece more assertively defines the space around oneself. Instead of
simply blocking out anyone from entering into one’s social bubble, Vironment
2.0 threatens consequences for anyone entering into one’s social bubble. This
touches on the real-world health risks associated with entering into others’
social bubbles, and the physically detrimental ramifications that can be dealt
to those who disrespect maintenance of the social bubble.

Figure 3: Inspiration for Vironment 2.0. Figure 4: Vironment 2.0: Social
distancing spikes.

##  5 Vironment 3.0

Further, we present a virtual version of Vironment/Invironment, see Fig 1, Fig
5, and Fig 7. The virtual Vironment uses sonar to alert the wearer in regards
to violation of social-distancing. In particular, we use 12 HC-SR04 Ultrasonic
sensors which each have a detectable range from 2cm to 4 metres (approx. 13.1
feet). These 12 sensors are arranged around a circle, at 30 degree intervals,
analogous to a clock face, i.e. at the cardinal 12-hour clock face directions
and worn around the neck of the individual. Vironment 3.0 is a distance sensor
with a 360° veillance field surrounding the wearer. A blinding bright LED will
light up and a car horn (worn in a backpack) will sound whenever this
apparatus detects an individual entering into the 2m social radius existing
around the wearer. This serves as a barrier-less and non-violent approach to
social-distancing that maintains the importance of the social bubble, while
maintaining a notification for the wearer and the individual imposing upon the
wearer’s personal space.

Figure 5: Vironment 3.0 with softband. Figure 6: CAD of Vironment 3.0 made in
Fusion 360 for 3D printing

(a)

(b)

Figure 7: The Vironment 3.0 necklace with hardband.

A 3-D printed version of Vironment 3.0 was made in which the necklace is
hollowed out in order to route wires for the sonars more discretely. The
circuit board was also moved to the back of the necklace to act as a
counterweight. In previous designs the necklace was not level, with the
forward facing sonars pitched down. Adding a counterweight allows the necklace
to align in parallel with the ground, ensuring the sensors can sense the
presence of any imposing individuals within a 360° field around the wearer.

In order to increase the ease and speed at which the apparatus is mounted on
the neck, a hinge was placed to connect the two 3D printed parts of the
necklace, allowing it to open and close for easy wearability. A double latch
system was designed and included as a lock to keep the necklace shut around
the user’s neck. See Fig 6 for a CAD model of the Vironment 3.0 piece.

The electrical system is composed of 12 HC-SR04 Ultrasonic sensors, with all
Vcc pins on a 5 Volt bus, and all Gnd pins on a Ground bus. Each sensor has
two data wires, one “Trigger”, which triggers the ultrasonic wave, and one
‘Echo”, which outputs information regarding the time delay (and thus, distance
measured). Because there are 12 sonars each with two data lines (24 lines),
four MC14051B analog multiplexers are used to switch between the sonars, with
2 multiplexers chained for “Trigger”s from the sensors, and 2 multiplexers
chained for “Echo”s. The system is controlled by an Espressif ESP-32 DevKit C
microcontroller, which controls the multiplexers, provides power to the
sensors, and processes the data. These few components are mounted and soldered
to a piece of perfboard. The necklace interfaces with the display system (the
EyeTap) using a direct USB connection.

In the next, future iteration of Vironment, Vironment 4.0, instead of 12 pairs
of ultrasound transducers, there will be 12 individual transducers in which
each is both a transmitter and a receiver. This will decrease the weight of
the wearable as well as its height (the double stacked sonar hits the user’s
chin), from about 48mm to 16mm, i.e. about one third the height of Vironment
3.0.

###  5.1 PPI (Plan Position Indicator)

Finally, we combine two concepts: (1) the concept of the PPI (Plan Position
Indicator) from radar [24] (see Fig 8), and (2) the use of a clockface,
especially in shooter-based reconnaissance where a shooter is instructed as to
the location of enemy targets in reference to a clock face. The necklace was
updated to reflect the layout of a clock, with a sonar placed at every hour
position. In egocentric coordinates, the sonar at 12 o’clock points forward
and the sonar at 6 o’clock points backwards. The numbers of the clock are also
engraved on the inside of the collar right behind their respective sonars, to
aid the user in putting the necklace on properly.

As such, rather than quadrants or octants, we use dodecants, dividing the
space into 12 equally sized sectors.

This layout of PPI and clockface is displayed live on AR glasses worn by the
user. The AR view is achieved by a microdisplay UI running on an EyeTap [25]
smart glasses display, as shown in Fig 1. The processor on the necklace (an
ESP-32) communicates sonar information directly to the EyeTap’s on-board
Raspberry Pi Zero W Single Board Computer (SBC).

Thus, the wearer of Vironment 3.0 can visualize (in AR) social distancing
information using a Natural User Interface (NUI).

Figure 8: The P.P.I. live user interface representing the social bubble around
the user. The distance of the dodecant boundary to the center represents the
distance of another individual to the wearer, at that specific location. The
color is also modulated from black (infinite) to bright green (2cm) to
represent the distance.

(a)

(b)

(c)

Figure 9: Social Distancer app: A wearable indicator for social distance. The
application is run on a smartphone in the shirt pocket (with the screen facing
out). The screen is green if proper social distance is maintained, yellow if
on the verge of danger, and red if social distance is compromised.

##  6 Social Distancer app

Many of the widely used and discussed social distancing technologies are
surveillant - they rely on a centralized authority collecting data on citizens
in order to maintain a social distancing and contact-tracing database.
“Vironment” is different in its ability to encourage, or enforce, social
distancing without the requirement of surveillant systems.

To continue exploration in this direction, we developed a social distancing
smartphone app with the ability to be immediately deployed to billions of
people. It runs on consumer hardware and presents a means of social distancing
that is sousveillant, open, and available to everyone to use without
extraneous hardware and without reliance on any central authority. The Social
Distancer app can be seen in Fig 9.

The Social Distancer app runs on a smart phone equipped with a forward facing
camera. The application collects a live stream of video data from the front
facing camera which it runs through an object detection neural network
(MobileNet pretrained on Coco dataset [26]). A filter is placed on the output
of the network to only accept predictions of people in the image frame
(ignoring recognition of other objects). The bounding box around the person is
combined with the average height of a person ($\sim$1.65 meters [27]) to
estimate the distance between the Social Distancer app and the individual
being detected by the camera. The screen of the Social Distancer app is green
when social distancing is maintained, yellow when individuals are 6 to 7 feet
away (almost breaking social distancing), and red when social distancing has
been compromised (i.e. another individual has intruded into the wearer’s
inviroment).

The user of the system places the Social Distancer app in the breast pocket of
their clothing with the screen and front camera facing outwards. In this
configuration, the Social Distancer app serves as a way to remind others to
maintain a healthy and safe distance.

##  7 Conclusion

We have introduced a series of wearable health systems collectively named
“Invironment”. Vironment 1.0, and 2.0 were forms of social commentary
highlighting design questions which were answered through a social-distancing
sonar system, Vironment 3.0. Vironment 3.0 consists of an array of sonar
sensors arranged in a circle at 30 degree intervals, i.e. as the 12 cardinal
directions of the hours of a 12-hour clock face. In this system we designed a
PPI (Plan Position Indicator) on an EyeTap display to show the wearer a top-
down “map” of their risk space in regards to violators of social distance.
Finally, the Social Distancer app is a deployable, sousveillant system which
promotes social distancing while avoiding the use of centralized surveillant
data tracking.

## Acknowledgment

The authors would like to thank Kyle Simmons for his feedback on the
electrical design of the sonar social distancer “Vironment 3.0” system.

21

## References

  * [1] S. Mann, L.-T. Cheng, J. Robinson, K. Sumi, T. Nishida, S. Matsushita, Ömer Faruk Özer, O. Özün, C. Öncel Tüzel, V. Atalay, A. E. Çetin, J. Anhalt, A. Smailagic, D. P. Siewiorek, F. Gemperle, D. Salber, S. Weber, J. Beck, J. Jennings, and D. A. Ross, “Wearable ai,” _Intelligent Systems, IEEE_ , vol. 16, no. 3, pp. 1–53, May/June 2001.
  * [2] S. Mann, “Wearable computing: A first step toward personal imaging,” _IEEE Computer_ , vol. 30, no. 2, pp. 25–32, 1997.
  * [3] B. Nayak and S. S. Bhattacharyya, “The changing narrative in the health insurance industry: Wearables technology in health insurance products and services for the covid-19 world,” _Journal of Health Management_ , vol. 22, no. 4, pp. 550–558, 2020.
  * [4] O. Amft, L. Lopera, P. Lukowicz, S. Bian, and P. Burggraf, “Wearables to fight covid-19: From symptom tracking to contact tracing,” _IEEE Annals of the History of Computing_ , vol. 19, no. 04, pp. 53–60, 2020.
  * [5] B. Nestor, J. Hunter, R. Kainkaryam, E. Drysdale, J. B. Inglis, A. Shapiro, S. Nagaraj, M. Ghassemi, L. Foschini, and A. Goldenberg, “Dear watch, should i get a covid-19 test? designing deployable machine learning for wearables,” _medRxiv_ , 2021.
  * [6] A. R. Linares, K. A. Bramstedt, M. M. Chilukuri, and P. M. Doraiswamy, “Physician perceptions of surveillance: Wearables, apps, and bots for covid-19,” _medRxiv_ , 2021.
  * [7] M. P. Fitzgerald, “Self-disclosure and expressed self-esteem, social distance and areas of the self revealed,” _The Journal of Psychology_ , vol. 56, no. 2, pp. 405–412, 1963.
  * [8] R. T. Azuma, “Augmented reality: Approaches and technical challenges,” in _Fundamentals of wearable computers and augmented reality_ , W. Barfield and T. Caudell, Eds. New Jersey: Lawrence Erlbaum Press, 2001, ch. 2, pp. 27–63.
  * [9] T. Starner, S. Mann, B. Rhodes, J. Levine, J. Healey, D. Kirsch, R. Picard, and A. Pentland, “Augmented reality through wearable computing,” _PRESENCE_ , vol. 6, no. 4, August 1997, mIT Press.
  * [10] S. Mann, “Sousveillance, not just surveillance, in response to terrorism,” _Metal and Flesh_ , vol. 6, no. 1, pp. 1–8, 2002.
  * [11] S. Mann, J. Nolan, and B. Wellman, “Sousveillance: Inventing and using wearable computing devices for data collection in surveillance environments.” _Surveillance & Society_, vol. 1, no. 3, pp. 331–355, 2003\.
  * [12] M. G. Gordon Fletcher and M. Kutar, “A day in the digital life: a preliminary sousveillance study,” _SSRN, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1923629_ , September 7, 2011\.
  * [13] R. Janzen and S. Mann, “Vixels, veillons, veillance flux: An extramissive information-bearing formulation of sensing, to measure surveillance and sousveillance,” _IEEE CCECE_ , pp. 1–10, 2014.
  * [14] K. Michael and M. Michael, “Sousveillance and point of view technologies in law enforcement: An overview,” 2012.
  * [15] V. Bakir, _Sousveillance, media and strategic political comm…_ Continuum, 2010.
  * [16] E. Bradshaw, “Sousveillance, direct action …” _Critical Criminology_ , vol. 21, no. 4, pp. 447–461, 2013. [Online]. Available: http://dx.doi.org/10.1007/s10612-013-9205-4
  * [17] D. Freshwater, P. Fisher, and E. Walsh, “Revisiting the panopticon: professional regulation, surveillance and sousveillance,” _Nursing Inquiry_ , May 2013, pMID: 23718546. [Online]. Available: http://dx.doi.org/10.1111/nin.12038
  * [18] S. Mann, _Intelligent Image Processing_. John Wiley and Sons, Nov. 2 2001.
  * [19] K. Kelly and G. Wolf, “What is the quantified self,” _The Quantified Self_ , vol. 5, p. 2007, 2007.
  * [20] S. Mann, “Can humans being clerks make clerks be human? $k\>{o}nnen$ menschen, die sich wie angestellte benehmen, angestellte zu menschlichem verhalten bewegen?” _Informationstechnik und Technische Informatik_ , vol. 43, no. 2, pp. 97–106, 2001.
  * [21] S. Mann, C. Pierce, A. Bhargava, C. Tong, K. Desai, and K. O’Shaughnessy, “Sensing of the self, society, and the environment,” in _2020 IEEE Sensors_. IEEE, 2020, pp. 1–4.
  * [22] S. Mann, “fl huge uid streams: fountains that are keyboards with nozzle spray as keys that give rich tactile feedback and are more expressive and more fun than plastic keys,” in _Proceedings of the 13th annual ACM international conference on Multimedia_. ACM, 2005, pp. 181–190.
  * [23] C. Zeagler, “Where to wear it: functional, technical, and social considerations in on-body location for wearable technology 20 years of designing for wearability,” in _Proceedings of the 2017 ACM International Symposium on Wearable Computers_ , 2017, pp. 150–157.
  * [24] G. Harold, “Plan position indicator system,” Mar. 5 1946, uS Patent 2,395,966.
  * [25] S. Mann and J. Fung, “EyeTap devices for augmented, deliberately diminished, or otherwise altered visual perception of rigid planar patches of real-world scenes,” _Presence: Teleoperators & Virtual Environments_, vol. 11, no. 2, pp. 158–175, 2002.
  * [26] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” _CoRR_ , vol. abs/1801.04381, 2018. [Online]. Available: http://arxiv.org/abs/1801.04381
  * [27] N. R. F. Collaboration _et al._ , “A century of trends in adult human height,” _Elife_ , vol. 5, p. e13410, 2016.