# Tag-based annotation creates better avatars

Minghao Liu
UC Santa Cruz
<EMAIL_ADDRESS>
Zeyu Cheng
San Jose State University
@outlook.com
Shen Sang
ByteDance Inc.
<EMAIL_ADDRESS>
Jing Liu
ByteDance Inc.
<EMAIL_ADDRESS>
James Davis
UC Santa Cruz
<EMAIL_ADDRESS>

###### Abstract

Avatar creation from human images allows users to customize their digital
figures in different styles. Existing rendering systems like Bitmoji,
MetaHuman, and Google Cartoonset provide expressive rendering systems that
serve as excellent design tools for users. However, twenty-plus parameters,
some including hundreds of options, must be tuned to achieve ideal results.
Thus it is challenging for users to create the perfect avatar. A machine
learning model could be trained to predict avatars from images, however the
annotators who label pairwise training data have the same difficulty as users,
causing high label noise. In addition, each new rendering system or version
update requires thousands of new training pairs. In this paper, we propose a
Tag-based annotation method for avatar creation. Compared to direct annotation
of labels, the proposed method: produces higher annotator agreements, causes
machine learning to generates more consistent predictions, and only requires a
marginal cost to add new rendering systems.

##  1 Introduction

Well-designed avatar creation tools like Bitmoji [3], Google Cartoonset [8],
and Metahuman [24] provide expressive tools for users to create digital
figures based on themselves. However, customizing the ideal avatar involves
laborious selection and adjustment of parameters. Such a process consumes a
significant amount of time from an average user without necessarily resulting
in their ideal design. Training a learning-based algorithm for avatar auto-
creation is needed.

Supervised learning requires the collection of pairwise training data. Given
human face images, annotators manually create the corresponding avatars by
selecting the best assets. Unfortunately, there are inherent issues with this
direct annotation method. During the creation process, some parameters such as
hairstyle include hundreds of options with only minor differences. It is
almost impossible for the annotators to consistently select a single optimal
choice, resulting in low agreement with other annotators. When collected in
this way, the dataset has high label noise, and majority vote aggregation does
little to help.

Figure 1: Avatar creation aims to automatically customize a stylized figure
for a user. Hairstyle prediction is shown here because hair is the hardest
attribute to pick automatically with hundreds of ambiguous options. We propose
a Tag-based annotation method for avatar creation, which results in better
labels (Sec 4.1), better predictions (Sec 4.2), and better generalizability
(Sec 4.3).

Instead, we propose a Tag-based annotation method for avatar creation. A
semantically meaningful set of tags is designed as an intermediate
representation which applies to both photographs and avatar renderings.
Annotators label both training photographs and the set of stylized assets
using these tags. For example, annotating hair length, hair curly level, and
hair direction instead of simply find the best option out of two hundred
hairstyles. This design provides better instructions and encourages annotators
to search for more detailed features when labeling. The proposed Tag-based
annotation results in higher annotator agreements.

Given a photograph, Tag annotations do not directly provide an answer for
which asset is a best match. We use a search algorithm to evaluate the
similarity of image tags and the tags of each possible asset. The asset with
maximum similarity is selected as the best match. In addition to providing a
best match, this search provides a score for each asset, so additional
plausible options are available. In contrast, direct annotation produces a
single best label with no information about the quality of any other asset.

In order to understand how this change in annotation style affects the final
system, we compared supervised learning models trained on Tag-based labels
with models trained on Direct labels. Experimental results show that models
trained with the Tag-based system produces better and more consistent
predictions. Example hairstyles predicted by the tag-based system are shown in
Fig. 1.

Finally, we demonstrate that the Tag-based system helps with generalizability
by showing results on several different avatar rendering systems. When
shifting to a new rendering system, direct annotation requires a new large
training set of human-avatar pairs. In contrast, the tag-based labels of human
images can be reused, and the only new labels required are for the relatively
small set of new assets. In a typical system, assets number in the hundreds
while training photographs number in the tens of thousands, so the advantage
is significant.

This paper contributes a new method for avatar creation using tag-based
annotation. The advantages of this method include:

  * •

Cleaner labels with higher annotator agreement

  * •

Better predictions from the ML model

  * •

Lower cost generalization to new rendering systems

##  2 Related work

Image Stylization: Creating a virtual character from an input human portrait
image needs to overcome the domain gaps between the real world and the target
styles. Gatys et al.[11] matched feature information from CNN models to
achieve style transfer. Cycle consistency loss was used for image-to-image
transfer with paired-wise data supervision [15] and with self-supervision [45,
26]. Recently, the development of GAN inversion methods [29, 37] results in
excellent image decomposition and high-quality reconstructions, which has been
applied to image stylization [33, 6, 46]. However, all of these methods
focused on creating high-quality images in pixel space, as opposed to
selecting assets in an avatar rendering system.

Avatar creation using non-semantic parameters: Creating avatars in parameter
spaces without semantic meaning has been well-studied for many years.
Extremely high quality methods for photorealistic avatars using stereo vision
[2, 42], and single input images exist [4, 28, 9, 41, 7], with multiple good
survey papers [10, 47].

Stylized avatar systems also exist. Some methods utilize sketches as the prior
condition for generation [12, 13]. Other methods are guided by position and
landmarks, extracting human facial features used to deform textures and meshes
[40, 5, 20, 38]. Recently a conditional GAN has been applied in the generation
process [21, 43]. However, these methods all utilized parameters without
semantic meaning, making them inapplicable to avatar systems designed to
provide user level customization of asset choice.

Avatar creation using semantic parameters: To provide tools for customization
of avatar creation, excellent rendering tools like Bitmoji [3], Metahuman
[24], and Google Cartoon Set [8] were created. These rendering systems provide
explicit semantic meanings to each parameter and focus primarily on manual
user creation.

Avatar prediction has been explored using self-supervised methods to avoid the
difficulty of manual labeling. When the avatar is semi-photorealistic, F2P
[31] utilizes neural imitators to mimic the behaviors of the rendering system,
improving efficiency and applying textures for more photorealistic visual
quality [32, 22]. In the stylized domain, AgileAvatar [30] introduced a domain
transfer module to the avatar creation pipeline. However, these self-
supervised methods rely heavily on carefully tuning each style. We provide a
comparison to these methods in our results section.

Human face datasets: Training neural engines require the collection of human
face datasets. FFHQ [18] provides a collection of high-quality human face
images without annotation. CelebA [23] and MAAD[35, 36] datasets include some
basic tags of facial attributes. FairFace [17] includes ethnic tags and
provides a racially balanced set. Hairstyle specific datasets also exist.
Figaro-1k [34] provides a limited set of samples, Hairstyle-30k [44] treats
the task as an end-to-end classification task, while K-hairstyle [19] focus on
Korean hairstyles. None of these datasets has labels matching the specific
avatar rendering systems we use in our work. We make use of the FairFace
dataset for photographs of human faces.

##  3 Method

Direct annotation for avatar creation and the challenges it introduces are
discussed in Sec. 3.1. Our tag-based annotation system is introduced in Sec.
3.2. The search algorithm which relates tags to specific assets is discussed
in Sec. 3.3. The vision backbone and training loss we selected is provided in
Sec. 3.4.

Figure 2: Direct annotation is a challenging task: There are hundreds of
options in the selection pool, and for a given human portrait multiple
plausible answers may exist. Each of the plausible answers may have only very
minor differences, and none may be a perfect match for the image. This
ambiguity leads to poor agreement between annotators.

###  3.1 Direct annotation

Customizing a stylized avatar using rendering systems like Bitmoji requires
users to tune more than twenty parameters. For example, hairstyle, face shape,
and nose type. Some parameters include hundreds of options to choose from and
we notice that some parameters dominate the perception of avatar quality. To
enable detailed comparison between avatar annotation and prediction methods,
we select a single parameter to demonstrate results in this paper. We choose
hairstyle, because it is the attribute with the most variations, and assets
often contain high ambiguity between options. Hairstyle also has high visual
impact on an avatar, since it affects a large spatial region. We keep other
parameters fixed to the extent possible in each rendering system we use.

Fig. 2 shows some sample hairstyles from the Bitmoji rendering system with the
grey-scaled, gender-neutral, default face. There are many hairstyles to choose
from and direct annotation simply asks the annotator to pick the ground-truth
asset class which best matches the input photograph. However decision making
is complicated. Some options have very subtle differences between them. Often
none are an ideal match and more than one hairstyle is a plausible option. A
few examples of this “no single best answer" phenomena are shown on the left
side of the figure. Simply asking the annotators to directly label the best
one out of two hundred options results in high label noise, and since
annotators have low agreement majority vote aggregation does not often
succeed.

###  3.2 Tag-based annotation

In this paper, we propose Tag-based annotation. The goal is to map both human
face images and avatar hairstyles to a tag space with semantic meaning. We
defined our tags as in Tab. 1. Instead of providing the annotators with a
massive number of options, we specifically ask them to annotate tag attributes
from each region, for example Hair direction on the top of the head, or Hair
curliness level on the side of the head. Detailed descriptions of each tag are
included in the supplemental material.

Region | Annotation Tags | Distance calculation
---|---|---
Attributes | # Options | Weight | Type
Top and front | Length | 6 | 2.25 | Continuous
Direction | 8 | 2 | Discrete
Curly level | 4 | 1 | Continuous
On the side | Length | 5 | 2.25 | Continuous
Curly level | 4 | 1 | Continuous
Braid | Yes / No | 2 | 5 | Discrete
Count | 4 | 2 | Discrete
Position | 3 | 1 | Discrete
Type | 5 | 1 | Discrete

Table 1: Attributes designed for Tag-based annotation. The design process is
described in Sec 3.2. Additional detailed descriptions of each tag is included
in the supplementary material.

Designing the appropriate tags to describe the hairstyle requires domain
knowledge. Each tag requires a clear definition. For example, short hair and
medium-short hair is insufficient description for consistent labeling. We use
an iterative design process to arrive at our final tag definitions. The
researchers first designed tags to describe the hairstyles by simply looking
at a set of human images and avatars. An annotator tagged all the avatar
hairstyles using this tag design. A different annotator tagged a set of human
images. Using the tags, the best matched avatar to each photo is retrieved.
The researchers then evaluate the agreement between annotators, and the
expressibility of the tag design, to make modifications to the set of tags.
The process was repeated until tag design was considered sufficient.

After arriving at a tag designing, we perform the complete run of data
annotation. Note that our tag design pipeline allows researchers to focus on
iteratively improving their tag designs while not requiring them to work as
annotators. By going through such a design process, researchers verify their
designs, so that higher agreement between researchers and annotators is
achieved.

###  3.3 Search algorithm

Our designed tag system has 460,800 permutations, making it impossible to
design a hairstyle for each permutation. This implies that for many human
images there are no perfect hairstyle match. To address this issue, we search
through all existing hairstyles, computing tag similarity. The overall
distance of a particular asset is computed as a weighted sum of individual tag
distances. The weight of each attribute is listed in Tab. 1. To measure the
tag distance for each attribute, we used L1 loss for continuous variables, and
zero-one loss for discrete variables.

The distance score from the search provides ranking information for all the
hairstyles, while direct label annotation only provides the Top-1 result. Fig.
3 shows visual samples of low and high-distance pairs. Note that the low-
distance hairstyles have better visual similarity with the inputs, while the
high-distance samples are visually dissimilar.

Figure 3: Visualization of low and high distance samples: The distance score
is calculated by a search algorithm described in Sec 3.3, which measures the
visual similarity between human images and avatar hairstyles. Low distance
cases have good visual similarity to the person, while high distance cases
appear visually dissimilar.

###  3.4 ML training

We trained our learning-based models in a supervised manner. To extract
feature information from the image, we used the open-source pre-trained
Resnet-50 [14] from the Pytorch [27] library as our vision backbone and the
initial training checkpoint. During training, we used L2 loss for continuous
variables and cross-entropy loss for discrete classifications.

##  4 Results and Experiments

In this section, we demonstrate the advantage of Tag-based annotation with
experimental results. Sec. 4.1 shows the advantage at the annotation level,
Sec. 4.2 shows the advantage on model convergence and consistent predictions,
and Sec. 4.3 shows that a Tag-based system can easily be compatible with new
rendering systems.

###  4.1 Annotation Quality

|  | Direct annotation | Tag-based annotation
---|---|---|---
Chance Agreement exists | Tag level | NA | 96.7%
Final Top-1 | 31.0% | 52.0%
Final Top-2 | NA | 67.2%
Final Top-3 | NA | 73.5%
Final Top-4 | NA | 80.3%
Time | Skilled annotators | 25.1s | 23.6s
Random Turker | 48.4s | 112.6s
User study | Matching | 89.4% | 92.7%
Preference | 306 : 306

Table 2: Numerical comparisons of annotation methods:  Agreement between
annotators is a measure of quality. It is also correlated with the chance that
majority vote aggregation helps reduce label noise. Tags have much higher
agreement than direct labels. Using tags to predict the Top-1 hairstyle also
produces better agreement than asking annotators to directly label the best
hairstyle. Tag-based annotation additionally provides multiple Top-K matches
for each annotator, where direct annotation only results in the Top-1 option.
The agreement level increased to 80.3% (+49.3%) by aggregating answers from
the Top-4 matches from Tag-based annotation. Time analysis shows a skilled
annotator spent a similar amount of time to annotate a face using both
annotation methods. A user study showed that Tag-based annotation results in
similar visual quality.  Figure 4: Visual comparison of annotation results:
Both direct and Tag-based annotation methods provide plausible answers,
however Tag-based annotation provides additional backup recommendations.

Annotator Agreement: Label noise is a common problem in supervised learning
models [39]. Collecting multiple copies of annotation for aggregation is often
required to create a high-quality dataset. Using a majority vote is the most
common way to aggregate labels and reduce label noise. However, given a large
number of hairstyle options, there might exist multiple plausible answers, or
alternatively, no ideal match and only partially correct answers. These
situations cause low agreements between annotators. Tab. 2 provides evidence
of the severity of the problem when 3 annotators provide independent labels
for each target. Agreement exists between annotators in only 31.0% of cases
when using direct annotation. In the majority of cases all three annotators
provide different answers. On the other hand, annotator agreement on tags
exists in 96.7% of cases when using tag annotations. In order to fairly
compare against direct annotation, we compute agreement on the Top-1 hairstyle
chosen by each set of tag labels. Because multiple similar hairstyles exist,
agreement is lower than on raw tag prediction at only 52.0%, but this is
substantially better (+21.0%) than the agreement when labelers provide direct
annotations. Unlike direct annotation, Tag-based annotation provides
additional plausible answers which can be used to improve agreement and enable
aggregation. Aggregating over the Top-4 matches from each of 3 labelers,
reaches an agreement level of 80.39% (+49.37%). We conclude that Tag-based
labels are substantially more valuable in creating a “clean" set of labels.

Annotation cost: The total cost of annotation is a function of labeling time
for a individual image, and the total number of labels required. The cost of
individual annotation is measured in time as shown in Tab. 2. We asked
annotators from two different skill levels to annotate images and recorded
annotation time. Our experiment shows well-trained professional annotators
need marginally less time to annotate a face when using the Tag-based system.
In contrast, untrained workers obtained from Amazon Mechanical Turk need more
time when using tag-based labels. In either case, the differences in
individual labeling time are bounded and not the most significant factor.

Direct annotation requires a completely new set of labels each time a change
is made to the set of available hairstyles, with associated retraining of the
prediction model given the new labels. In contrast, tag-based systems only
need to label the new hairstyles, with no new tag labels on the much larger
set of training images, and no retraining of the tag predictor. Since most
avatar rendering systems will be updated with new artistic assets
occasionally, we conclude that there is a substantial savings on label cost
using tag-based annotation.

Visual Quality comparison: To compare the visual quality between direct and
Tag-based annotation, we conduct two user studies through Amazon Mechanical
Turk [1]: Matching and Preference. In the Matching task, we evaluate whether
visual similarity is sufficiently close that evaluators can tell which avatar
goes with which human. A human image is shown with the corresponding avatar
hairstyle and three random distractor hairstyles. The evaluator is required to
match the human image and avatar image and is scored as a correct match if the
evaluator correctly picks the original annotation. A high matching score
indicates the avatar represents the human well. A total of 1,224 judgments
were collected. As the result shows in Tab. 2, our proposed Tag-based
annotation preserves user identity marginally better compared to direct
annotation. In the Preference task, avatar results from both methods are
presented for comparison with the human image. The evaluators were asked to
provide their preferences by choosing one of the results or indifferent, a
total of 612 judgments were collected. The results showed a precise split of
306 judgements for each method. The combined results of both studies indicate
that our tags are sufficiently expressive to act as a replacement for direct
annotation in terms of visual quality.

A visual comparisons of the two annotations methods is provided in Fig. 4.
Both annotation methods result in plausible answers. In addition to the Top-1
match, a Tag-based system can provide ranking information for all other
hairstyles. The figure also shows Top-5 results from the Tag-based
annotations. The visual quality of these results is reduced, but they remain
plausible.

###  4.2 ML Prediction Quality

Tag-based labels result in better ML models when used for selecting avatar
assets, in terms of both visual quality and consistency.

####  4.2.1 Baseline methods and dataset

We compare several methods to understand the effects of tag-based annotation.
We choose two state of the art baseline methods using self-supervised models,
a model supervised by direct labels, and a model supervised by Tags.

Self-supervised baselines

  * •

F2P [31]: an optimization-based method designed for realistic game character
creation.

  * •

AgileAvatar [30]: a state-of-the-art learning-based method for stylized avatar
creation.

Supervised baselines: Both supervised baselines use an identical vision back
bone with similar loss functions.

  * •

Direct pred: Treat the task as a classification task, predict the best
hairstyle, and calculate training loss against direct annotation.

  * •

Tag pred: Treat the task as a classification task, predict the tags, and
calculate training loss against the Tag-based annotations. The final hairstyle
prediction is determined using the search described in Sec. 3.3.

Dataset and annotation: We used human face images from the FairFace [17]
dataset. Compared to the commonly used CelebA [23] dataset, FairFace is
racially balanced, which ensures our trained model works well on diverse
faces. FairFace also includes blurry images, which requires the trained model
to be robust on lower-quality images.

Annotation: We collected Tag-based annotation for 17k images from FairFace
using professional annotators. We randomly choose 14.5k for training, 2.8k for
testing, and 204 images as a hold out set for human evaluation studies.

Inter-annotator agreement is higher for tag-based annotation than direct
annotation. To avoid bias due to the quality of annotations collected, we
create a set of direct labels using the tag-based labels. We use the tags for
each image to find the single best hairstyle. We treat these hairstyles as the
training targets for Direct Pred. Thus both supervised learning methods share
the same Top-1 training targets.

####  4.2.2 Better prediction quality

Figure 5: Visual comparisons of model predictions:  Tag pred and Direct pred
are supervised learning methods, which results in better visual avatars
compared to self-supervised learning models F2P[31] and AgileAvatar[30]. F2P
was designed for photorealistic avatar creation and fails to produce good
results on stylized avatars. AgileAvatar produces better results with the help
of stylization modules, though it captures less feature information than
supervised learning methods.

| Trained methods | Matching
---|---|---
Self-supervised | F2P[31] | 34.73%
AgileAvatar [30] | 67.22%
Supervised (ours) | Direct pred | 70.79%
Tag pred | 83.92%

Table 3: Quality comparisons of learning methods: A matching test asks
evaluators to identify the best avatar when the ML produced avatar is
presented with 3 alternates as distractors. A higher matching score indicates
better representations of the input human images. Tag-based prediction
performs better than existing self-supervised state of the art, and in
comparison to direct annotations.

Visual comparisons for models: We visually compare all four learning-based
methods in Fig. 5. F2P [31] was designed for realistic human-style avatars and
fails frequently on stylized avatars. AgileAvatar [30] has a human stylization
module to overcome the style domain gap and has significant improvements over
F2P; however it is still not as good as the supervised learning methods.
Comparing the Direct pred and Tag pred models, training a model using the Tag-
based annotation helps the model to capture the detailed features of the
hairstyles. For example, capturing the double pony-tail for the girl in the
third row, and the curliness of the hair for the man in the sixth row.

Numerical comparison for models: We conducted user studies to support the
visual comparison with results shown in Tab. 3. The Matching test requires
evaluators to match a randomly selected human against 4 avatar results. Such a
test measures how well the model captures the human identity. If the avatar
looks very similar to the photo, then identifying it is easy. If the avatar
does not look similar it is hard to distinguish from randomly chosen
distractors. Higher matching scores are better. F2P is designed for
photorealistic avatars and performs poorly. The carefully tuned self-
supervised AgileAvatar reaches a similar matching score to supervised Direct
pred, only 3.57% lower. However, Tag-based annotation performs best,
preserving more user identity than all other baselines.

####  4.2.3 Better convergence, More consistent predictions

Figure 6: Prediction consistency: Visual comparisons of the Top-5 predictions
from supervised learning models trained using tag-based and direct
annotations. While both methods produce plausible Top-1 answer, The Top-5
predictions from Tag Pred have better consistency than those from Direct Pred.
Notice for example that direct prediction sometimes includes both short and
long hair in its top five predictions.

|  | Top-1
---
Accuracy
| Top-5
---
Accuracy
| Tag pred
---
Accuracy
| Distance
---
Top-1
| Distance
---
Top-5
Direct pred | 10.29 % | 32.84 % | NA | 6.09 | 8.25
Tag pred |  | 17.16%
---
(+6.87%)
| 42.16%
---
(+9.32%)
95.72% | 2.51 | 3.76
Annotation |  |  |  | 2.29 | 3.39

Table 4: Convergence comparisons:  Tag annotations can be used to build better
ML models. Using the final avatar target to calculate Top-1 and Top-5 accuracy
for each supervised learning method shows a better convergence on the Tag-pred
model. The distance score is an alternate measure of similarity between a
human face and avatar, and also shows better performance for tag-based
prediction.

Convergence: During training, both supervised learning models have the same
best-matching hairstyle for each human image. In the case of direct prediction
this hairstyle is explicitly used in the target loss, while in the case of tag
prediction it is implicitly defined by the target tags. Higher accuracy on the
test set indicates converging to a more optimal point. Tab. 4 provides
prediction accuracies. Naive comparison shows that direct prediction finds
correct answer 10.29% of the time while tag prediction is correct 95.72% of
the time. However this is not an entirely fair comparison because direct
prediction must choose among hundreds of classes, while tag prediction has
many fewer classes available. To create a fair comparison we require both
methods to choose a final hairstyle among the hundreds available and compare
this to the “ground-truth" target. We compare model convergence toward the
target loss using Top-K accuracy. Training the model on the Tag-based systems,
results in a better Top-1 (+6.87%) and Top-5 (+9.32%) accuracy compared to
model training on direct labeling.

The low Top-1 (17.16%) accuracy seems to contradict our claim of good quality
predictions in Tab 3; however, in reality, Top-1 accuracy only tells a partial
story. There are hundreds of hairstyles, and some options have very minor
difference from other options. Given the potential existence of multiple
correct answers, it’s not always wrong to predict something other than the
“ground truth" option. Thus Top-1 and Top-5 are good indicators of only the
relative quality of two prediction models, not the absolute quality.

As an alternative metric for quality, we used our search algorithm described
in Sec. 3.3 to calculate the distance between the human face and the final
avatar predictions, using the tags originally provided by annotators. A low
distance score indicates greater similarity between the human face and avatar
at the feature level. The result in Tab. 4 shows Tag pred has a lower average
distance for the Top-1(2.51) and Top-5(3.16) predictions. Note that most human
face images don’t have a perfectly matching avatar so a distance score of 0.0
is impossible. We include the distance for “ground truth" hairstyles as chosen
by annotators as an indication of a lower bound on prediction quality.

Consistency: We visual the Top-5 predictions of both supervised learning
methods in Fig. 6. While both methods produce plausible Top-1 answers, the
Top-5 predictions from Tag Pred have better consistency compared to Direct
Pred. The Direct Pred model treats each hairstyle as an independent class
without considering their visual similarities to the human image. Tag pred, on
the other hand, is trained to predict the visual features defined by human
researchers. Thus the model was encouraged to focus on the features that are
important to human observers, thus resulting in consistent Top-K predictions.
Notice for example that even when direct prediction correctly predicts a short
hairstyle as the Top-1 result, the next best prediction might be long hair.

###  4.3 Generalizability

Figure 7: Easily expandable to other systems: Models trained on Tag-based
annotation can easily be reused with other rendering systems. The tags
originally predicted for use with the Bitmoji system were used to select
avatars in three alternate systems.

Annotating a dataset costs time and effort from both researchers and
annotators. In our case, about 17k sets of tags for human images and avatar
hairstyles. To make our trained model compatible with any other rendering
system, the model requires an entirely new set of 17k labels if using direct
annotation.

Using a Tag-based system, on the other hand, can significantly reduce this
cost. Since the tags for the human images are independent of rendering
systems, human tags can be reused. We are only required to collect new tags
for the avatar assets in the new rendering systems. Most rendering systems
provide “only" 200 hairstyle variations, costing less than 2% of relabeling
the human images. In addition the original tag prediction ML model remains
valid, and does not need to be retrained.

To demonstrate the generalizability of the Tag-based system, we collected tags
for avatar samples from Bitmoji [3], Google Cartoonset [8], MetaHuman [24],
and NovelAI [25]. For all these systems, we only controlled the hairstyles:

  * •

Bitmoji: A cartoon avatar rendering system. We used the gender-neutral, gray-
scaled default faces.

  * •

Google Cartoon set: A dataset of the results from a cartoon avatar rendering
system. We controlled the hairstyles and left other attributes set random.

  * •

Metahuman: A realistic human avatar rendering system. Hairstyle is varied
while leave other parameters constant. We choose between males and female
default avatars using the gender tags in the FairFace dataset.

  * •

NovelAI: A diffusion-based text-to-image generation model specialized for
cartoon-style figures. We asked artists to select text prompts to generate
cartoon figures with different hairstyles and treat the resulting images as a
set of avatars.

Fig. 7 shows the model-predicted visual results. Given the tag predictions
from our supervised learning model, the search algorithm finds the closest
matching hairstyle in each system based on the avatar tags. Not every system
contains every hairstyle so some matches are closer than others, but overall
the avatars selected are good approximations of the input photograph.

##  5 Limitations and conclusion

Limitations: To benefit from the Tag-based system, a set of carefully designed
tags with clear definitions is required, along with the cost in time and
effort to produce it.

The search algorithm is sensitive to tag prediction errors. While tag
prediction is substantially more accurate than direct prediction, a wrong
prediction will mislead the search.

As with all avatar prediction methods, our work lacks a universally accepted
benchmark or evaluation metric. Commonly used L2 loss, perceptual loss[16], or
Top-K accuracy do not well represent actual user preference. This paper
conducted user studies for result evaluations, but this also has shortcomings
since evaluators are not the same as actual users.

Conclusion: In summary, we present a Tag-based annotation method for avatar
creation. Our approach results in higher annotation quality with higher
annotator agreements. Supervised learning models guided by the proposed
annotation converge to a more optimal point, and output more consistent
predictions. Our proposed method can easily be applied to new rendering
systems with marginal cost. Experimental results including numerical, visual,
and user studies support our claims.

## References

  * [1] Amazon mechanical turk.  https://www.mturk.com.  Accessed: 2022-11-10.
  * [2] Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross.  High-quality single-shot capture of facial geometry.  In ACM SIGGRAPH 2010 papers, pages 1–9. ACM New York, NY, USA, 2010\.
  * [3] Bitmoji.  https://www.bitmoji.com/.  Accessed: 2022-11-10.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194, 1999.
  * [5] Hongrui Cai, Yudong Guo, Zhuang Peng, and Juyong Zhang.  Landmark detection and 3d face reconstruction for caricature using a nonlinear parametric model.  Graphical Models, 115:101103, 2021.
  * [6] Kaidi Cao, Jing Liao, and Lu Yuan.  Carigans: Unpaired photo-to-caricature translation, 2018.
  * [7] Zhixiang Chen and Tae-Kyun Kim.  Learning feature aggregation for deep 3d morphable models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13164–13173, 2021.
  * [8] Forrester Cloe, Inbar Mosseri, Dilip Krishnan, Aaron Sarna, Aaron Maschinot, Bill Freeman, and Shiraz Fuman.  Cartoon set, 2022.  data retrieved from Google Machine Perception organization, https://google.github.io/cartoonset/people.html.
  * [9] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [10] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.  3d morphable face models—past, present, and future.  ACM Transactions on Graphics (TOG), 39(5):1–38, 2020.
  * [11] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  * [12] Xiaoguang Han, Chang Gao, and Yizhou Yu.  Deepsketch2face: a deep learning based sketching system for 3d face and caricature modeling.  ACM Transactions on graphics (TOG), 36(4):1–12, 2017.
  * [13] Xiaoguang Han, Kangcheng Hou, Dong Du, Yuda Qiu, Shuguang Cui, Kun Zhou, and Yizhou Yu.  Caricatureshop: Personalized and photorealistic caricature sketching.  IEEE transactions on visualization and computer graphics, 26(7):2349–2361, 2018.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [16] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [17] Kimmo Karkkainen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548–1558, 2021.
  * [18] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [19] Taewoo Kim, Chaeyeon Chung, Sunghyun Park, Gyojung Gu, Keonmin Nam, Wonzo Choe, Jaesung Lee, and Jaegul Choo.  K-hairstyle: A large-scale korean hairstyle dataset for virtual hair editing and hairstyle classification.  In 2021 IEEE International Conference on Image Processing (ICIP), pages 1299–1303. IEEE, 2021.
  * [20] Thomas Lewiner, Thales Vieira, Dimas Martínez, Adelailson Peixoto, Vinícius Mello, and Luiz Velho.  Interactive 3d caricature from harmonic exaggeration.  Computers & Graphics, 35(3):586–595, 2011.
  * [21] Song Li, Songzhi Su, Juncong Lin, Guorong Cai, and Li Sun.  Deep 3d caricature face generation with identity and structure consistency.  Neurocomputing, 454:178–188, 2021.
  * [22] Jiangke Lin, Yi Yuan, and Zhengxia Zou.  Meingame: Create a game character face from a single portrait.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 311–319, 2021.
  * [23] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [24] Unreal engine metahuman.  https://www.unrealengine.com/en-US/metahuman.  Accessed: 2022-11-10.
  * [25] Novelai.  https://novelai.net/.  Accessed: 2022-11-10.
  * [26] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu.  Contrastive learning for unpaired image-to-image translation.  In European Conference on Computer Vision, pages 319–345. Springer, 2020.
  * [27] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [28] Weilong Peng, Zhiyong Feng, Chao Xu, and Yong Su.  Parametric t-spline face morphable model for detailed fitting in shape subspace.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6139–6147, 2017.
  * [29] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [30] Shen Sang, Tiancheng Zhi, Guoxian Song, Minghao Liu, Chunpong Lai, Jing Liu, Xiang Wen, James Davis, and Linjie Luo.  Agileavatar: Stylized 3d avatar creation via cascaded domain bridging.  ACM SIGGRAPH Asia 2022 Conference Proceedings, 2022.
  * [31] Tianyang Shi, Yi Yuan, Changjie Fan, Zhengxia Zou, Zhenwei Shi, and Yong Liu.  Face-to-parameter translation for game character auto-creation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 161–170, 2019.
  * [32] Tianyang Shi, Zhengxia Zuo, Yi Yuan, and Changjie Fan.  Fast and robust face-to-parameter translation for game character auto-creation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1733–1740, 2020.
  * [33] Guoxian Song, Linjie Luo, Jing Liu, Wan-Chun Ma, Chunpong Lai, Chuanxia Zheng, and Tat-Jen Cham.  Agilegan: stylizing portraits by inversion-consistent transfer learning.  ACM Transactions on Graphics (TOG), 40(4):1–13, 2021.
  * [34] Michele Svanera, Umar Riaz Muhammad, Riccardo Leonardi, and Sergio Benini.  Figaro, hair detection and segmentation in the wild.  In 2016 IEEE International Conference on Image Processing (ICIP), pages 933–937. IEEE, 2016.
  * [35] Philipp Terhörst, Daniel Fährmann, Jan Niklas Kolf, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper.  Maad-face: A massively annotated attribute dataset for face images.  IEEE Trans. Inf. Forensics Secur., 16:3942–3957, 2021.
  * [36] Philipp Terhörst, Marco Huber, Jan Niklas Kolf, Ines Zelch, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper.  Reliable age and gender estimation from face images: Stating the confidence of model predictions.  In 10th IEEE International Conference on Biometrics Theory, Applications and Systems, BTAS 2019, Tampa, FL, USA, September 23-26, 2019, pages 1–8. IEEE, 2019.
  * [37] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  * [38] Roberto C Cavalcante Vieira, Creto A Vidal, and Joaquim Bento Cavalcante-Neto.  Three-dimensional face caricaturing by anthropometric distortions.  In 2013 XXVI Conference on Graphics, Patterns and Images, pages 163–170. IEEE, 2013.
  * [39] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu.  Learning with noisy labels revisited: A study using real-world human annotations.  In International Conference on Learning Representations, 2022.
  * [40] Qianyi Wu, Juyong Zhang, Yu-Kun Lai, Jianmin Zheng, and Jianfei Cai.  Alive caricature from 2d to 3d.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7336–7345, 2018.
  * [41] Sicheng Xu, Jiaolong Yang, Dong Chen, Fang Wen, Yu Deng, Yunde Jia, and Xin Tong.  Deep 3d portrait from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7710–7720, 2020.
  * [42] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao.  Facescape: A large-scale high quality 3d face dataset and detailed riggable 3d face prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [43] Zipeng Ye, Mengfei Xia, Yanan Sun, Ran Yi, Minjing Yu, Juyong Zhang, Yu-Kun Lai, and Yong-Jin Liu.  3d-carigan: an end-to-end solution to 3d caricature generation from normal face photos.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [44] Weidong Yin, Yanwei Fu, Yiqing Ma, Yu-Gang Jiang, Tao Xiang, and Xiangyang Xue.  Learning to generate and edit hairstyles.  In Proceedings of the 25th ACM international conference on Multimedia, pages 1627–1635, 2017.
  * [45] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [46] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka.  Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks, 2021.
  * [47] Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, and Christian Theobalt.  State of the art on monocular 3d face reconstruction, tracking, and applications.  In Computer Graphics Forum, volume 37, pages 523–550. Wiley Online Library, 2018.


# Dispersionless Flat Mode and Vibrational Anomaly in Active Brownian
Vibrators Induced by String-like Dynamical Defects

Cunyuan Jiang1,2,3 Zihan Zheng1,4 Yangrui Chen1,4 Matteo Baggioli1,2,3
<EMAIL_ADDRESS>Jie Zhang1,4<EMAIL_ADDRESS>1 School of
Physics and Astronomy, Shanghai Jiao Tong University, Shanghai 200240, China
2 Wilczek Quantum Center, School of Physics and Astronomy, Shanghai Jiao Tong
University, Shanghai 200240, China  3 Shanghai Research Center for Quantum
Sciences, Shanghai 201315,China  4 Institute of Natural Sciences, Shanghai
Jiao Tong University, Shanghai 200240, China

###### Abstract

In recent years, active Brownian particles have emerged as a prominent model
system for comprehending the behaviors of active matter, wherein particles
demonstrate self-propelled motion by harnessing energy from the surrounding
environment. A fundamental objective of studying active matter is to elucidate
the physical mechanisms underlying its collective behaviors. Drawing
inspiration from advancements in molecular glasses, our study unveils a low-
energy “flat mode” within the transverse spectrum of active Brownian vibrators
– a nearly two-dimensional, bi-disperse granular assembly. We demonstrate that
this collective excitation induces an anomalous excess in the vibrational
density of states (VDOS) beyond the phononic Debye contribution. Additionally,
we establish through empirical evidence that string-like dynamical defects,
discerned via the spatial distribution of each particle’s contribution to the
reduced transverse VDOS, serve as the microscopic origin of the flat mode and
its associated anomalies. These findings underscore the pivotal role of
string-like dynamical defects in elucidating the vibrational and mechanical
properties of active Brownian particles.

Introduction – Active Brownian particles have recently gained popularity as a
modeling system for investigating microorganisms such as bacteria, cells, and
microswimmers Marchetti  _et al._ (2013); Cates and Tailleur (2015); Zheng
_et al._ (2013); Kümmel  _et al._ (2013); Reichhardt and Reichhardt (2014);
Henkes  _et al._ (2020); Sprenger  _et al._ (2020); Caprini and Löwen (2023);
Bröker  _et al._ (2023). Unlike passive Brownian particles, active Brownian
particles do not exist in thermal equilibrium; they exhibit self-propelled
motion by harnessing energy from their surrounding environment Marchetti  _et
al._ (2013); Cates and Tailleur (2015). Understanding the behavior of these
active particles is crucial for deciphering processes like cell migration in
intricate tissue and organ formation Marchetti  _et al._ (2013), as well as
wound healing Safferling  _et al._ (2013). Furthermore, a deeper understanding
of active Brownian particles is imperative for the development of artificial
micro- and nanorobots intended for future applications in drug delivery and
environmental protection.

Chen et al. recently conducted an experiment involving a quasi-two-dimensional
granular system of uniformly driven active Brownian vibrators Chen and Zhang
(2022, 2023). In this system, the statistics of single-particle velocity and
rotation adhere to Gaussian distributions with zero means Chen and Zhang
(2022). This experimental setup closely mirrors the active Brownian particles
scrutinized in theoretical studies, as it lacks inherent particle-scale
alignment interactions Caprini and Löwen (2023). The system comprises numerous
hard-disk-like particles, each devoid of any inherent orientation in shape and
without preferred translational or rotational directions, owing to the absence
of sub-particle scale built-in asymmetries, in contrast to the earlier studies
Kudrolli  _et al._ (2008); Narayan  _et al._ (2007); Deseigne  _et al._
(2010); Scholz and Pöschel (2017). Unlike microswimmers, which operate within
a viscous and overdamped surrounding medium, the underdamped dynamics observed
in this system offer a convenient experimental platform. This platform allows
for the direct observation and analysis of collective shear excitations in bi-
disperse systems Jiang  _et al._ (2024), all while mitigating issues
associated with crystallization.

In our surroundings, amorphous soft matter systems, including gels, structural
glasses, colloids, and granular assemblies, are commonplace. However, their
dynamic, thermodynamic, and mechanical properties remain poorly understood in
contrast to the well-explained behaviors of ideal crystals, predominantly
governed by elastic theory and harmonic lattice vibrations (phonons) Chaikin
and Lubensky (2000). These soft matter systems exhibit non-phononic degrees of
freedom, collectively termed as ‘defects,’ which contribute to the observed
anomalies alongside crystal-like acoustic waves. Within amorphous materials,
the concept of ‘defects’ manifests in various forms. For instance, they are
recognized as two-level systems P. w. Anderson and c. M. Varma (1972);
Phillips (1972) elucidating the low-temperature heat capacity Zeller and Pohl
(1971), quasilocalized non-phononic modes Lerner and Bouchbinder (2021)
accounting for the boson peak (BP) excess in the vibrational density of states
Ramos (2022), and as entities such as shear transformation zones Argon (1979),
Eshelby quadrupoles Eshelby and Peierls (1959), or topological defects
Baggioli (2023), providing insight into mechanical and plastic dynamics during
deformation Falk and Langer (2011).

In recent studies, there has been a notable focus on the significance of
string-like dynamical defects across a spectrum of materials, including
supercooled liquids, heated crystals, quasi-crystals, and glasses. These
defects bear resemblance to vortex lines observed in superfluids Kleinert
(1989) and play a crucial role in shaping relaxational and dynamical
properties Donati  _et al._ (1998); Pazmino Betancourt  _et al._ (2014); Zhang
_et al._ (2021a, 2015, 2013); Zhao  _et al._ (2024); Hung and Simmons (2020).
Simulation-based investigations have revealed that these quasi-one-dimensional
entities Novikov and Surovtsev (1999), primarily of transverse nature, emerge
as the most plausible candidates responsible for the manifestation of
nonphononic modes. It is furthermore postulated that these string-like
excitations serve as a microscopic origin of the boson peak (BP) phenomenon
typical of glassy materials Bianchi  _et al._ (2020); Hu and Tanaka (2022,
2023); Zhang  _et al._ (2021b); Jiang  _et al._ (2023); Jiang and Baggioli
(2024); Zhang  _et al._ (2024). Of particular interest are findings from
simulations by Hu and Tanaka Hu and Tanaka (2022, 2023), which establish a
close correlation between these string-like excitations and the emergence of a
“flat boson peak mode”. This mode, characterized by collective and
dispersionless low-energy excitation, has also been observed in experimental
studies on the boson peak within two-dimensional amorphous materials Tømterud
_et al._ (2023). Despite accumulating experimental evidence in various
amorphous systems Berardi  _et al._ (2010); Zhang  _et al._ (2009); Nagamanasa
_et al._ (2011), direct experimental validation of dynamical string-like
defects in amorphous active Brownian particles remains elusive. Enhanced
understanding of vibrational and collective dynamics in these particles holds
promise in elucidating fundamental characteristics pertaining to the mechanics
and hydrodynamics of active matter Marchetti  _et al._ (2013); Ramaswamy
(2010).

In this study, we have uncovered a distinctive mode within the low-frequency
spectrum of a two-dimensional layer comprising bi-disperse active Brownian
vibrators through an analysis of their dynamic matrices. Termed the
‘dispersionless flat mode’, this transverse mode manifests as an anomalous
augmentation in the vibrational density of states (VDOS) beyond the typical
phononic Debye contribution. Through a detailed examination of the VDOS at the
particle level, we have pinpointed the microstructural origin of this flat
mode and its associated anomalies: the string-like excitation within the
spectral domain. Notably, our discovery corroborates recent theoretical
propositions by Hu and Tanaka Hu and Tanaka (2022, 2023), despite the
substantial distinctions between our active Brownian particle system and
conventional molecular glasses.

Experimental setup and data analysis  – Our experimental system consists in a
horizontal layer of granular particles driven vertically by a sinusoidal
oscillation with a fixed frequency of $100$ Hz and amplitude of $0.062$ mm
induced by an electromagnetic shaker. To prevent crystallization, we utilize
bi-disperse particles with a size ratio of disk diameters of $1$:$1.4$ and a
number ratio of $2$:$1$ for small and large. The total particle number is
$798$ with packing fraction $\phi=0.822$. The motion of the particles, labeled
“granular Brownian vibrators”, is recorded with a Basler CCD camera
(acA2040-180kc) at $40$ frames/s for at least an hour. We refer to Refs. Chen
and Zhang (2022); Jiang  _et al._ (2024) for more details about the
experimental setup.

The analysis of the experimental data is based on the displacement correlation
matrix $\boldsymbol{C}$, defined as Ghosh  _et al._ (2010); Chen  _et al._
(2010)

$C_{ij}=\langle n(t)_{i}n(t)_{j}\rangle_{T},$ (1)

where $n_{i}(t)$ is the displacement of $i$th degree of freedom at time $t$,
and $T$ the time window on which the average $\langle\cdot\rangle$ is
performed. The dynamical matrix can then be calculated as,

$D_{ij}=\dfrac{\alpha\,C^{-1}_{ij}}{\sqrt{m_{i}m_{j}}},$ (2)

where $m_{i}$ is the mass of the $i$th degree of freedom, and $\alpha$ is a
dimensionful parameter which is fixed using the same procedure outlined in
Jiang  _et al._ (2024). By diagonalizing the matrix $\boldsymbol{D}$, one
obtains the eigenvalues $\kappa_{i}$ and the eigenfrequencies
$\omega_{i}=\sqrt{\kappa_{i}}$. The eigenvector fields $\boldsymbol{u}$ are
then defined by solving the eigenvalue problem,
$\boldsymbol{D}\boldsymbol{u}=\omega^{2}\boldsymbol{u}$. From the eigenvector
field $\boldsymbol{u}$, longitudinal ($L$) and transverse ($T$) dynamical
structure factor are extracted,

$S_{L,T}(\boldsymbol{k},\omega)\propto\dfrac{1}{k^{2}}\left|\sum_{i}F_{(L,T),i}(\boldsymbol{k})\delta_{\omega,\omega_{i}}\right|^{2},$
(3)

with

$\displaystyle
F_{L,i}(\boldsymbol{k})=\boldsymbol{k}\cdot\sum_{j}\boldsymbol{u}_{i}(\boldsymbol{r}_{j})e^{-i\boldsymbol{k}\cdot\boldsymbol{r}_{j}},$
(4) $\displaystyle
F_{T,i}(\boldsymbol{k})=\boldsymbol{z}\cdot\boldsymbol{k}\times\sum_{j}\boldsymbol{u}_{i}(\boldsymbol{r}_{j})e^{-i\boldsymbol{k}\cdot\boldsymbol{r}_{j}}.$
(5)

Here, $\boldsymbol{z}$ is the vector perpendicular to the two-dimensional
experiment platform, while the index $j$ runs over all particles and
$\boldsymbol{r}_{j}$ is the position of $j$th particle. The vibrational
density of states (VDOS) can be obtained by
$g(\omega)=\sum_{i}\delta_{\omega,\omega_{i}}$, or equivalently
$g(\omega)_{L,T}=\int S_{L,T}(\boldsymbol{k},\omega)d\boldsymbol{k}$, showing
perfect agreement. Finally, the particle level VDOS is given by
$g_{L,T}(\boldsymbol{r}_{j},\omega)=\sum_{i}|\boldsymbol{u}_{i,L,T}(\boldsymbol{r}_{j})|^{2}\delta_{\omega,\omega_{i}}$
where $\boldsymbol{u}_{i,L,T}(\boldsymbol{r}_{j})$ is the longitudinal or
transverse eigenvector field of $j$th particle under eigenfrequency
$\omega_{i}$ Hu and Tanaka (2022, 2023).

Dynamical response and vibrational anomalies  – The transverse dynamical
structure factor is shown as a functon of frequency and for different wave-
vectors in Fig.1. The experimental data are visualized with red symbols. The
thick red lines are the results of a fit with the following function,

$S_{T}(\omega,k)\propto\dfrac{\omega^{2}}{(\omega^{2}-\Omega_{T}^{2})^{2}+\omega^{2}\Gamma_{T}^{2}}+S_{\text{extra}}(\omega).$
(6)

with

$S_{\text{extra}}(\omega)=\dfrac{1}{\omega\sigma\sqrt{2\pi}}\mathrm{exp}\left(-\dfrac{(\mathrm{ln}\omega-\mu)^{2}}{2\sigma^{2}}\right),$
(7)

while the thin lines are the two contributions in Eq. (6). The first term
corresponds to a damped harmonic oscillator with linewidth $\Gamma_{T}(k)$ and
characteristic frequency $\Omega_{T}(k)$. The second term represents a log-
normal function Malinovsky  _et al._ (1991) that describes additional low-
energy modes as commonly utilized in amorphous systems Hu and Tanaka (2022,
2023).

The results of Fig.1 outline the presence of two distinct excitations,
captured by the two terms in Eq.(6). The first mode, whose dispersion
$\Omega_{T}(k)$ is indicated with a black dashed line, corresponds to
propagating shear waves (transverse phonons) with a speed of $v_{T}=71.7$ ds/s
(in unit of the small particle diameter $d_{s}$). At this specific value of
the packing fraction, the system is in a dense liquid state and the collective
shear waves present a small gap in wave-vector Jiang  _et al._ (2024), known
as $k$-gap Baggioli  _et al._ (2020), that emerges nevertheless at smaller
wave-vector and is not shown in Fig.1.

Figure 1: The transverse dynamical structure factor $S_{T}(\omega,k)$. The red
symbols are the experimental data. The solid lines represent the fit using
Eq.(6). The dashed line marks the dispersion of the transverse phonon mode
$\Omega_{T}(k)$, while the gray vertical region indicates the energy of the
dispersionless flat mode plus/minus its uncertainty.

The second excitation appears at frequencies below those of the transverse
phonon and corresponds to the peak whose position is highlighted by the
vertical gray area. Interestingly, we find that the energy of this collective
mode does not depend on the wave-vector $k$. Hence, we label this excitation
the “dispersionless flat mode” (DFM). As the wave-vector becomes larger, the
intensity of the peak associated to the DFM becomes weaker with respect to
that of the concomitant transverse phonon.

Importantly, a DFM can only be found in the transverse dynamical structure
factor, indicating its transverse nature. As demonstrated in the Supplementary
Information (SI), the longitudinal dynamical structure factor
$S_{L}(\omega,k)$ displays a single peak, corresponding to a longitudinal
phonon mode with linear dispersion relation, $\Omega_{L}(k)=v_{L}k$, and speed
$v_{L}=116.9$ ds/s Jiang  _et al._ (2024).

In order to confirm the existence of the DFM and study its impact on the
vibrational properties of our system, we move to the analysis of the VDOS,
$g(\omega)$. To better visualize our results, we normalize the VDOS by the
two-dimensional Debye scaling $g_{\text{Debye}}\propto\omega$. Our
experimental results are shown in Fig.2, where the blue, red and gray lines
correspond respectively to the longitudinal, transverse and total density of
states.

At low frequency, the longitudinal part of the VDOS is approximately constant,
indicating the validity of the Debye approximation, and consistent with the
observation of a linearly dispersing longitudinal phonon. It then shows a
pseudo-Van Hove peak at approximately $12$ Hz, that agrees with the energy at
which $\Omega_{L}(k)$ flattens (see SI), and then decays at higher
frequencies, vanishing above $\approx 30$ Hz.

Figure 2: The reduced VDOS for packing fraction $\phi=0.822$. The inset shows
the original VDOS. The blue, red, and gray colors correspond to the
longitudinal, transverse, and total VDOS. The vertical gray area indicates the
energy of the dispersionless flat mode observed in the dynamical structure
factor, Fig.1. The dashed lines denote the position of the longitudinal and
transverse Debye levels, $C_{L,T}$, determined from the corresponding sound
speeds.

The transverse component of the VDOS (red curve in Fig.2), presents a more
complex behavior that can be divided in three different regions. Below $1$ Hz,
the transverse VDOS decays monotonically indicating the presence of a strong
diffusive component that can be also seen from a large nonzero value of
$g_{T}(\omega)$ at zero frequency (see inset of Fig.2). Above $\approx 3$ Hz,
the transverse VDOS is similar to the longitudinal one, with the difference
that the pseudo-Van Hove peak is much weaker. This indicates that the
transverse dynamics are more overdamped than the longitudinal ones. In order
to test the validity of Debye theory, we focus on the low frequency range in
which the transverse and longitudinal reduced VDOS display an approximate
plateau. We mark the corresponding plateaus with horizontal dashed lines in
Fig.2. We verify that the ratio of the values of the two plateaus,
$C_{L}/C_{T}$, agrees well with the prediction of the Debye model,
$v_{T}^{2}/v_{L}^{2}$ where the speeds are extracted from the dispersion
relations obtained from the dynamical structure factor.

More interestingly, between $\approx 1$-$3$ Hz, we observe an excess anomaly
in the Debye reduced transverse VDOS whose energy is compatible with the DFM
observed in the dynamical structure factor, Fig.1, and whose structure is
similar to the boson peak observed in several amorphous materials Ramos
(2022).

The connection between a dispersionless flat mode and the boson peak anomaly
has been highlighted in simulations Hu and Tanaka (2022, 2023) and observed
experimentally Tømterud  _et al._ (2023), and its a direct manifestation of
the presence of additional (and likely localized) low-energy degrees of
freedom responsible for the BP.

String-like dynamical defects  – After proving a direct link between the flat
mode and the BP excess anomaly in the VDOS, we are now in a position to
investigate its microscopic origin at the particle-level motion. To do so, we
consider the particle-level reduced transverse VDOS, which determines how much
each particle contributes to the reduced VDOS at a given frequency.

The results of this analysis are shown in Fig.3 for six different frequency
values, from $0.4$ to $12$ Hz. A darker red indicates a more significant
contribution to the reduced transverse VDOS at that frequency.

Panels (a)-(c) are for frequencies around the characteristic energy of the
DFM, as indicated with the vertical gray region in Figs.1-2. In particular,
panel (b) lies precisely at the frequency of the flat mode corresponding to
the BP energy. In this range of frequencies, one-dimensional ($1$D) string-
like structures emerge, indicating particles’ heterogeneous dynamics and
tendency to form collective excitations of approximately ($1$D) nature. These
string-like dynamical defects are most substantial at the flat mode frequency,
panel (b).

However, for higher frequencies, as shown in panels (d)-(f), the spatial
distribution of the particle-level VDOS becomes more uniform, and no evident
string-like structure is observed anymore. On the other hand, as shown
explicitly in SI, the longitudinal particle-level VDOS does not display any
string-like structure. This confirms the DFM’s transverse nature and the
localized string-like excitations.

Figure 3: The spatial distribution of transverse particle-level reduced VDOS
at frequencies $0.4,1.2,2,4,8,12$ Hz from panel (a) to (f). A darker color
indicates a more significant contribution to the reduced VDOS at a given
frequency. The thick black line in panel (b) shows the extent of the average
length of string-like dynamical defects, which is approximately $12d_{s}$.

To provide a stronger connection between the string-like defect structures,
the BP anomaly in the VDOS and the properties of the DFM, we follow the idea
of associating a length scale with the boson peak introduced by Hong et al.
Hong  _et al._ (2011a, b) and Kalampounias et al. Kalampounias  _et al._
(2006) on a heuristic basis (see also Malinovsky  _et al._ (1988); Malinovsky
and Sokolov (1986); Martin and Brenig (1974)). We then define a cooperative
length scale $l$,

$l\approx\frac{v_{T}}{\omega_{BP}}=\frac{v_{T}}{\omega_{\text{DFM}}},$ (8)

whose value is determined up to an unknown $\mathcal{O}(1)$ coefficient. Using
$\omega_{\text{DFM}}=\omega_{BP}\approx 2$ Hz, we find $l\approx 35\,d_{s}$.
This value is approximately $3$ times the average length of the string-like
structures $\lambda\approx 12\,d_{s}$ observed in Fig.3.

This result is in qualitative agreement with recent theoretical studies Jiang
_et al._ (2023); Jiang and Baggioli (2024). At zero temperature, or
equivalently in the absence of damping, the length-scale $l$ extracted from
the BP and the average string-like defects length $\lambda$ were found to
match to a good approximation. In overdamped dynamics, the BP-like anomaly is
softened by damping mechanisms, resulting in a longer length scale. As
observed in glass-forming liquids above the glass transition temperature Jiang
_et al._ (2023), in that case $l$ is slightly larger than $\lambda$,
consistent with our findings.

Finally, we notice that previous works connected the scale of dynamic
heterogeneities $\xi$ to the boson peak frequency, $\xi\approx
cv_{T}/\omega_{BP}$, where $c$ is a constant found to be between $1/2$ and $1$
Schirmacher  _et al._ (2008); Duval  _et al._ (1990); Sokolov  _et al._
(1992); Elliott (1992, 1992); Schmid and Schirmacher (2008); Tanguy  _et al._
(2002). Therefore, our result suggests that the length-scale defined by the
string-like defects is strongly related to the scale of dynamic heterogeneity,
but possibly also to the scale defined by the fluctuations of the shear
modulus or by low-energy quasi-localized structures, that are found to
coincide in certain simulation glass models Mahajan and Ciamarra (2021).

Conclusions – We conducted an experiment utilizing a quasi-two-dimensional
disordered active Brownian particle system. Our investigation unveiled a
robust correlation among a collective dispersionless flat mode, the boson peak
anomaly in the vibrational density of states, and the emergence of string-like
dynamical defects. Our findings affirm that string-like structures,
predominantly of a transverse nature, may underlie the low-energy vibrational
anomalies observed in active Brownian particles. This hypothesis was initially
proposed in molecular glasses through simulations Hu and Tanaka (2022), and
the existence of the boson peak aligns with a dispersionless localized mode,
as evidenced experimentally in two-dimensional silica Tømterud  _et al._
(2023). Our discoveries indicate striking parallels in the low-energy
collective excitations of disordered active Brownian particles and molecular
glasses, despite the myriad of distinctions between these systems.

Acknowledgments – We would like to thank Jack Douglas, Yunjiang Wang, Wensheng
Xu, Haibin Yu, Peng Tan and Massimo Pica Ciamarra for numerous discussions
about string-like objects and their role in amorphous systems. CJ and MB
acknowledge the support of the Shanghai Municipal Science and Technology Major
Project (Grant No.2019SHZDZX01). MB acknowledges the support of the
sponsorship from the Yangyang Development Fund. ZZ, YC, and JZ acknowledge the
support of the NSFC (No. 11974238 and No. 12274291) and the Shanghai Municipal
Education Commission Innovation Program under No. 2021-01-07-00-02-E00138. ZZ,
YC, and JZ also acknowledge the support from the Shanghai Jiao Tong University
Student Innovation Center.

## References

  * Marchetti  _et al._ (2013) M. C. Marchetti, J. F. Joanny, S. Ramaswamy, T. B. Liverpool, J. Prost, M. Rao, and R. A. Simha, Rev. Mod. Phys. 85, 1143 (2013).
  * Cates and Tailleur (2015) M. E. Cates and J. Tailleur, Annual Review of Condensed Matter Physics 6, 219 (2015).
  * Zheng  _et al._ (2013) X. Zheng, B. ten Hagen, A. Kaiser, M. Wu, H. Cui, Z. Silber-Li, and H. Löwen, Phys. Rev. E 88, 032304 (2013).
  * Kümmel  _et al._ (2013) F. Kümmel, B. Ten Hagen, R. Wittkowski, I. Buttinoni, R. Eichhorn, G. Volpe, H. Löwen, and C. Bechinger, Physical review letters 110, 198302 (2013).
  * Reichhardt and Reichhardt (2014) C. Reichhardt and C. J. O. Reichhardt, Soft Matter 10, 7502 (2014).
  * Henkes  _et al._ (2020) S. Henkes, K. Kostanjevec, J. M. Collinson, R. Sknepnek, and E. Bertin, Nature communications 11, 1405 (2020).
  * Sprenger  _et al._ (2020) A. R. Sprenger, M. A. Fernandez-Rodriguez, L. Alvarez, L. Isa, R. Wittkowski, and H. Lowen, Langmuir 36, 7066 (2020).
  * Caprini and Löwen (2023) L. Caprini and H. Löwen, Physical Review Letters 130, 148202 (2023).
  * Bröker  _et al._ (2023) S. Bröker, J. Bickmann, M. Te Vrugt, M. E. Cates, and R. Wittkowski, Physical Review Letters 131, 168203 (2023).
  * Safferling  _et al._ (2013) K. Safferling, T. Sütterlin, K. Westphal, C. Ernst, K. Breuhahn, M. James, D. Jäger, N. Halama, and N. Grabe, Journal of Cell Biology 203, 691 (2013).
  * Chen and Zhang (2022) Y. Chen and J. Zhang, Phys. Rev. E 106, L052903 (2022).
  * Chen and Zhang (2023) Y. Chen and J. Zhang, arXiv preprint arXiv:2308.04219 (2023).
  * Kudrolli  _et al._ (2008) A. Kudrolli, G. Lumay, D. Volfson, and L. S. Tsimring, Phys. Rev. Lett. 100, 058001 (2008).
  * Narayan  _et al._ (2007) V. Narayan, S. Ramaswamy, and N. Menon, Science 317, 105 (2007).
  * Deseigne  _et al._ (2010) J. Deseigne, O. Dauchot, and H. Chaté, Phys. Rev. Lett. 105, 098001 (2010).
  * Scholz and Pöschel (2017) C. Scholz and T. Pöschel, Phys. Rev. Lett. 118, 198003 (2017).
  * Jiang  _et al._ (2024) C. Jiang, Z. Zheng, Y. Chen, M. Baggioli, and J. Zhang, “Experimental observation of gapped shear waves and liquid-like to gas-like dynamical crossover in active granular matter,” (2024), arXiv:2403.08285 .
  * Chaikin and Lubensky (2000) P. Chaikin and T. Lubensky,  _Principles of Condensed Matter Physics_ (Cambridge University Press, 2000).
  * P. w. Anderson and c. M. Varma (1972) B. I. H. P. w. Anderson and c. M. Varma, The Philosophical Magazine: A Journal of Theoretical Experimental and Applied Physics 25, 1 (1972).
  * Phillips (1972) W. A. Phillips, Journal of Low Temperature Physics 7, 351 (1972).
  * Zeller and Pohl (1971) R. C. Zeller and R. O. Pohl, Phys. Rev. B 4, 2029 (1971).
  * Lerner and Bouchbinder (2021) E. Lerner and E. Bouchbinder, The Journal of Chemical Physics 155, 200901 (2021).
  * Ramos (2022) M. A. Ramos,  _Low-temperature Thermal and Vibrational Properties of Disordered Solids: A Half-century of Universal” anomalies” of Glasses_ (World Scientific, 2022).
  * Argon (1979) A. Argon, Acta Metallurgica 27, 47 (1979).
  * Eshelby and Peierls (1959) J. D. Eshelby and R. E. Peierls, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 252, 561 (1959).
  * Baggioli (2023) M. Baggioli, Nature Communications 14, 2956 (2023).
  * Falk and Langer (2011) M. L. Falk and J. Langer, Annual Review of Condensed Matter Physics 2, 353 (2011).
  * Kleinert (1989) H. Kleinert,  _Gauge Fields in Condensed Matter: Vol. 1: Superflow and Vortex Lines (Disorder Fields, Phase Transitions) Vol. 2: Stresses and Defects (Differential Geometry, Crystal Melting)_ (World Scientific, 1989).
  * Donati  _et al._ (1998) C. Donati, J. F. Douglas, W. Kob, S. J. Plimpton, P. H. Poole, and S. C. Glotzer, Phys. Rev. Lett. 80, 2338 (1998).
  * Pazmino Betancourt  _et al._ (2014) B. A. Pazmino Betancourt, J. F. Douglas, and F. W. Starr, The Journal of Chemical Physics 140, 204509 (2014).
  * Zhang  _et al._ (2021a) H. Zhang, X. Wang, H.-B. Yu, and J. F. Douglas, Journal of Chemical Physics 154, 084505 (2021a).
  * Zhang  _et al._ (2015) H. Zhang, C. Zhong, J. F. Douglas, X. Wang, Q. Cao, D. Zhang, and J.-Z. Jiang, The Journal of Chemical Physics 142, 164506 (2015).
  * Zhang  _et al._ (2013) H. Zhang, M. Khalkhali, Q. Liu, and J. F. Douglas, The Journal of Chemical Physics 138, 12A538 (2013).
  * Zhao  _et al._ (2024) K. Zhao, M. Baggioli, W.-S. Xu, J. F. Douglas, and Y.-J. Wang, arXiv preprint arXiv:2402.10295 (2024).
  * Hung and Simmons (2020) J.-H. Hung and D. S. Simmons, The Journal of Physical Chemistry B 124, 266 (2020), pMID: 31886663.
  * Novikov and Surovtsev (1999) V. N. Novikov and N. V. Surovtsev, Phys. Rev. B 59, 38 (1999).
  * Bianchi  _et al._ (2020) E. Bianchi, V. M. Giordano, and F. Lund, Phys. Rev. B 101, 174311 (2020).
  * Hu and Tanaka (2022) Y.-C. Hu and H. Tanaka, Nature Physics 18, 669 (2022).
  * Hu and Tanaka (2023) Y.-C. Hu and H. Tanaka, Phys. Rev. Res. 5, 023055 (2023).
  * Zhang  _et al._ (2021b) H. Zhang, X. Wang, H.-B. Yu, and J. F. Douglas, The Journal of Chemical Physics 154, 084505 (2021b).
  * Jiang  _et al._ (2023) C. Jiang, M. Baggioli, and J. F. Douglas, arXiv preprint arXiv:2311.04230 (2023).
  * Jiang and Baggioli (2024) C. Jiang and M. Baggioli, arXiv preprint arXiv:2403.18221 (2024).
  * Zhang  _et al._ (2024) J. Zhang, H. Zhang, and J. F. Douglas, The Journal of Chemical Physics 160, 114506 (2024).
  * Tømterud  _et al._ (2023) M. Tømterud, S. D. Eder, C. Büchner, L. Wondraczek, I. Simonsen, W. Schirmacher, J. R. Manson, and B. Holst, Nature Physics 19, 1910 (2023).
  * Berardi  _et al._ (2010) C. R. Berardi, K. Barros, J. F. Douglas, and W. Losert, Phys. Rev. E 81, 041301 (2010).
  * Zhang  _et al._ (2009) H. Zhang, D. J. Srolovitz, J. F. Douglas, and J. A. Warren, Proceedings of the National Academy of Sciences 106, 7735 (2009).
  * Nagamanasa  _et al._ (2011) K. H. Nagamanasa, S. Gokhale, R. Ganapathy, and A. K. Sood, Proceedings of the National Academy of Sciences 108, 11323 (2011).
  * Ramaswamy (2010) S. Ramaswamy, Annu. Rev. Condens. Matter Phys. 1, 323 (2010).
  * Ghosh  _et al._ (2010) A. Ghosh, V. K. Chikkadi, P. Schall, J. Kurchan, and D. Bonn, Phys. Rev. Lett. 104, 248305 (2010).
  * Chen  _et al._ (2010) K. Chen, W. G. Ellenbroek, Z. Zhang, D. T. N. Chen, P. J. Yunker, S. Henkes, C. Brito, O. Dauchot, W. van Saarloos, A. J. Liu, and A. G. Yodh, Phys. Rev. Lett. 105, 025501 (2010).
  * Malinovsky  _et al._ (1991) V. Malinovsky, V. Novikov, and A. Sokolov, Physics Letters A 153, 63 (1991).
  * Baggioli  _et al._ (2020) M. Baggioli, M. Vasin, V. Brazhkin, and K. Trachenko, Physics Reports 865, 1 (2020), gapped momentum states.
  * Hong  _et al._ (2011a) L. Hong, V. N. Novikov, and A. P. Sokolov, Phys. Rev. E 83, 061508 (2011a).
  * Hong  _et al._ (2011b) L. Hong, V. Novikov, and A. Sokolov, Journal of Non-Crystalline Solids 357, 351 (2011b).
  * Kalampounias  _et al._ (2006) A. G. Kalampounias, S. N. Yannopoulos, and G. N. Papatheodorou, The Journal of Chemical Physics 125, 164502 (2006).
  * Malinovsky  _et al._ (1988) V. Malinovsky, V. Novikov, A. Sokolov, and V. Bagryansky, Chemical Physics Letters 143, 111 (1988).
  * Malinovsky and Sokolov (1986) V. Malinovsky and A. Sokolov, Solid State Communications 57, 757 (1986).
  * Martin and Brenig (1974) A. J. Martin and W. Brenig, physica status solidi (b) 64, 163 (1974).
  * Schirmacher  _et al._ (2008) W. Schirmacher, B. Schmid, C. Tomaras, G. Viliani, G. Baldi, G. Ruocco, and T. Scopigno, physica status solidi c 5, 862 (2008).
  * Duval  _et al._ (1990) E. Duval, A. Boukenter, and T. Achibat, Journal of Physics: Condensed Matter 2, 10227 (1990).
  * Sokolov  _et al._ (1992) A. P. Sokolov, A. Kisliuk, M. Soltwisch, and D. Quitmann, Phys. Rev. Lett. 69, 1540 (1992).
  * Elliott (1992) S. R. Elliott, Europhysics Letters 19, 201 (1992).
  * Schmid and Schirmacher (2008) B. Schmid and W. Schirmacher, Phys. Rev. Lett. 100, 137402 (2008).
  * Tanguy  _et al._ (2002) A. Tanguy, J. P. Wittmer, F. Leonforte, and J.-L. Barrat, Phys. Rev. B 66, 174205 (2002).
  * Mahajan and Ciamarra (2021) S. Mahajan and M. P. Ciamarra, Phys. Rev. Lett. 127, 215504 (2021).

## Supplementary Information

In this Supplementary Information (SI), we provide further analysis on our
experimental system.

The longitudinal dynamical structure factor as a function of frequency and for
different wave-vectors is shown in Fig.S1. It can be fitted by a single damped
harmonic oscillator function,

$S_{L}(\omega,k)\propto\dfrac{\omega^{2}}{(\omega^{2}-\Omega_{L}^{2})^{2}+\omega^{2}\Gamma_{L}^{2}}$
(S1)

that corresponds to an attenuated longitudinal phonon. The dispersion relation
of this mode, indicated with a black dashed line in Fig.S1, is well
approximated by a linear function $\Omega_{L}=v_{L}k$, with $v_{L}=116.9$
ds/s.

In Fig.S2, we present the particle level longitudinal vibrational density of
states. Contrary to the transverse component discussed in the main text, we do
no observe any clear evidence of string-like dynamical defects.

Figure S1: The longitudinal dynamical structure factor $S_{L}(\omega,k)$ as a
function of the frequency for different wave-vectors (labeled in figure). The
solid lines indicate the fit to the damped harmonic oscillator function, Eq.
(S1). The black dashed line indicates the dispersion of the longitudinal mode
$\Omega_{L}(k)$. Figure S2: The distribution of longitudinal particle level
VDOS with frequency $0.4,1.2,2,4,8,12$ Hz from panel (a) to (f). The darker
color the more contribution of the particle to VDOS.