# BeatDance: A Beat-Based Model-Agnostic Contrastive Learning Framework for
Music-Dance Retrieval

Kaixing Yang
Renmin University of China
<EMAIL_ADDRESS>Xukun Zhou
Renmin University of China
<EMAIL_ADDRESS>Xulong Tang
The University of Texas at Dallas
<EMAIL_ADDRESS>Ran Diao
Renmin University of China
<EMAIL_ADDRESS>Hongyan Liu
Tsinghua University
<EMAIL_ADDRESS>Jun He
Renmin University of China
<EMAIL_ADDRESS>Corresponding authors Zhaoxin Fan$\ast$
Renmin University of China
Psyche AI Inc
<EMAIL_ADDRESS>
###### Abstract

Dance and music are closely related forms of expression, with mutual retrieval
between dance videos and music being a fundamental task in various fields like
education, art, and sports. However, existing methods often suffer from
unnatural generation effects or fail to fully explore the correlation between
music and dance. To overcome these challenges, we propose BeatDance, a novel
beat-based model-agnostic contrastive learning framework. BeatDance
incorporates a Beat-Aware Music-Dance InfoExtractor, a Trans-Temporal Beat
Blender, and a Beat-Enhanced Hubness Reducer to improve dance-music retrieval
performance by utilizing the alignment between music beats and dance
movements. We also introduce the Music-Dance (MD) dataset, a large-scale
collection of over 10,000 music-dance video pairs for training and testing.
Experimental results on the MD dataset demonstrate the superiority of our
method over existing baselines, achieving state-of-the-art performance. The
code and dataset will be made public available upon acceptance.

Figure 1: This figure represents music, dance, and beat visualization from top
to bottom. Red dots indicate occurrence of dance beats, while purple vertical
lines represent occurrence of music beats. It is evident that there exists a
certain degree of correspondence between dance beats and music beats.

##  1 INTRODUCTION

Dance, as a significant art form, not only embodies human beauty and emotion
but also serves as a crucial medium for cultural inheritance and
communication. In recent years, with the rapid advancement of the Internet,
the availability and impact of dance videos have witnessed a remarkable surge,
providing audiences with diverse and captivating dance experiences.
Consequently, the demand for large scale music-dance retrieval has grown
exponentially, holding immense practical value for dance practitioners,
encompassing areas such as dance education, art creation, and sports training.

Existing approaches for obtaining music/dance from dance/music can be broadly
categorized into generation-based and retrieval-based methods. While
generation-based methods[16, 21, 37, 32] have shown significant progress in
recent years, they encounter certain inherent challenges such as unnatural
generation effects and limitations in generating diverse data types. For
instance, in mainstream Music2Dance methods, only human key points are
generated, neglecting factors like background and clothing. Similarly, in
Dance2Music approaches[38, 8, 10], models with better performance often
generate MIDI scores, overlooking the richness of human voice, background
sound, and other audio details. On the other hand, retrieval-based methods
naturally address these issues. Although music-dance retrieval has received
comparatively less attention, there have been notable advancements[36, 43],
but not fully explored correlation between dance and music.

Generally, dancers synchronize their body movements with the rhythm of music,
expressing their emotions and offering audiences a rich artistic experience,
where the ”beat” in dance and music serves as the most important information,
as illustrated in Figure 1. Motivated by this observation, we propose
BeatDance, a novel beat-based model-agnostic contrastive learning framework.
In BeatDance, the concept of beat alignment between music and dance is fully
utilized to enhance the model’s focus on individuals. By incorporating
temporal human pose information, representing the music beat, the model
becomes more attuned to capturing the nuances of dancers’ movements and allows
for a stronger connection between the rhythmic elements of dance and music.
Hence, it the retrieval performance could be significantly increased.

BeatDance comprises three key blocks: the Beat-Aware Music-Dance
InfoExtractor, the Trans-Temporal Beat Blender, and the Beat-Enhanced Hubness
Reducer. In the InfoExtractor block, pre-trained models and methods are
employed to extract rich information, including global
features(CLIP[29]/MERT[22]), music beat, and dance beat. The Feature Alignment
module is utilized to unify the dimensions of these extracted features. The
Beat Blender block involves sending the features to their respective Trans-
Temporal Process modules to obtain trans-temporal features. These trans-
temporal features are then blended with the global features using the Beat-
Enhanced Feature Fusion module, and beat-guided features are obtained through
the Beat-Guided Information Extraction module. To address the Hubness problem,
the Beat-Enhanced Hubness Reducer block employs a query bank to normalize the
similarity matrix during the inference phase, thereby alleviating issues
associated with hubness. Additionally, we introduce the Music-Dance dataset(MD
dataset), the first large-scale dataset specifically designed for the music-
dance retrieval task. This dataset is sourced from Bilibili[2], a popular
video-sharing platform in China, covering the period from May 2018 to
September 2023. It comprises 12,000 curated dance-music pairs with over
100,000 likes, encompassing various dance and music genres. Experimental
results on the MD dataset demonstrate the superiority of our method compared
to existing baselines, achieving state-of-the-art performance.

Our main contributions are summarized as below:

  * •

We introduce BeatDance, a novel beat-based model-agnostic contrastive learning
framework that effectively utilizes the beat alignment information between
music and dance to enhance the music-dance retrieval task.

  * •

To facilitate the learning of music-dance correlation, BeatDance incorporates
the Beat-Aware Music-Dance InfoExtractor, the Trans-Temporal Beat Blender, and
the Beat-Enhanced Hubness Reducer. These modules work synergistically to
jointly capture and leverage the relationship between music and dance.

  * •

To evaluate and benchmark existing methods, we present the MD dataset, the
first large-scale music-dance retrieval dataset. This dataset encompasses a
wide range of dance and music genres, providing a comprehensive evaluation
platform. Experimental results on the MD dataset demonstrate the superior
performance of our proposed method.

##  2 RELATED WORK

####  2.0.1 Music2Dance

Generating natural and realistic human motion from music is a challenging
problem. In recent years, significant progress has been made in the field of
music-to-dance motion generation using various neural network architectures
such as CNNs[44, 33, 42, 45], RNNs[1, 15, 33, 35], GCNs[9, 30], GANs[19, 33],
or Transformers[16, 21, 37, 32]. Typically, these music-to-dance methods are
conditioned on multimodal inputs and generate the future sequence of human
poses. However, these methods still face several challenges. First, they are
limited to generating only human poses and do not consider other important
factors in dance, such as costumes, backgrounds, and facial expressions.
Second, the generated motions often suffer from issues such as discontinuity
and teleportation. Third, research efforts have largely been focused on solo
dances while overlooking multi-person dances, despite their significant
importance in dance practice. Furthermore, with the abundance of internet
data, direct retrieval dance from music yields excellent results while
avoiding above issues. Therefore, this paper focuses on the research of music-
dance retrieval.

####  2.0.2 Dance2Music

Generating melodious and harmonious music for a given video is a challenging
task, and there are two main categories of methods to address this task: non-
symbolic based and symbolic based. Non-symbolic methods generate audio
directly in the waveform, which is the original form of audio[38, 8, 10].
However, a second of audio waveform covers a significant amount of data due to
its high frequency. Even utilizing intermediate audio representations[40, 18,
6], it is still computationally expensive and prone to generate noise.
Symbolic methods adopt a symbolic music modeling approach, such as 1D piano-
roll[7] and 2D event-based MIDI-like[14] music representations, etc.[25, 31].
However, harmonious resonance of different timbres of instruments is essential
to produce beautiful music, but symbolic methods often simplify the timbre,
resulting in relatively monotonous generated music. Moreover, given the wealth
of available internet data, performing direct retrieval music from video leads
to outstanding outcomes, circumventing above concerns. Consequently, our paper
delves into the exploration of dance-music retrieval.

Figure 2: Overview of BeatDance. We constructed a contrastive learning
framework consisting of three blocks: InfoExtractor, Beat Blender, Beat-
Enhanced Hubness Reducer. Specifically, given a music $m$ and a dance $d$,
InfoExtractor first returns aligned global feature $f^{D}$ and $f^{M}$, beat
feature of dance $f^{BD}$ and music $f^{BM}$. Then, Beat Blender processes
them and returns beat-enhanced feature of music $f^{M_{e}}$ and dance
$f^{D_{e}}$, beat-guided feature of music $f^{M_{g}}$ and dance $f^{D_{g}}$.
Finally, we construct two similarity matrix $m_{e}$ and $m_{g}$ between two
modality from beat-enhanced feature and beat-guided feature. In training
phase, we utilize $m_{e}$ and $m_{g}$ to calculate beat-enhance loss $L_{e}$
and beat-guided loss $L_{g}$ for contrastive learning; in inference phase, we
only send $m_{e}$, to Beat-Enhanced Hubness Reducer block and obtains
normalized $m_{qbnorm}$, and computed retrieved sequences.

####  2.0.3 Music-Dance Retrieval

Music-dance retrieval is a highly practical task in retrieval task, and music-
dance retrieval can be considered as a subtask of video-music retrieval. In
recent years, video-music retrieval have made significant progress[5, 13, 34,
28]. Typically, those above methods design a music and a video encoders to
project raw modalities into a high-dimensional feature space, followed by
contrastive learning training. However, video-music retrieval task primarily
focus more on the high-level semantic consistency between the two
modalities[20, 24], while ignoring the real-time matching requirements between
the two modalities. Relatively few to no researchers have paid attention to
the field of Music-Dance retrieval[36, 43], and those who have mostly followed
the traditional path of video-music retrieval, neglect strong beat
correspondence between dance and music, and do not fully explore the
correlation between music and dance. Moreover, we find there is no suitable
large-scale dataset to benchmark music-dance retrieval methods. In this paper,
we propose the BeatDance method and the MD dataset to solve the issue.

##  3 METHODOLOGY

###  3.1 Overview

Our study involves two tasks: Music-Dance retrieval and Dance-Music retrieval,
as Fig. 3 shows. For Music-Dance retrieval task, we take a piece of music $m$
as input, and output the matching sequence of dance
$\left\\{d_{1},d_{2}...d_{n}\right\\}$ from our database. For Dance-Music
retrieval task, we take a piece of dance $d$ as input, and output the matching
sequence of music $\left\\{m_{1},m_{2}...m_{n}\right\\}$ from our database.

To better explore correlation between the music and dance modalities, we
propose a Beat-Based Model-Agnostic contrastive learning framework called
BeatDance, as Fig. 2 shows. BeatDance consists of three blocks: Beat-Aware
Music-Dance InfoExtractor, Trans-Temporal Beat Blender, and Beat-Enhanced
Hubness Reducer.

In InfoExtractor block, we aim to acquire richer information and dimension
unification. We send music $m$ and dance $d$ to it, and then obtain unified:
music beat feature $f^{BM}$, dance beat feature $f^{BD}$, music global feature
$f^{M}$, dance global feature $f^{D}$.

$\displaystyle\begin{aligned} f^{D},f^{BD}&=InfoExtractor_{d}(d)\\\
f^{M},f^{BM}&=InfoExtractor_{m}(m)\end{aligned}$ (1)

In Beat Blender block, we aim to leverage the strong correspondence between
music beat and dance beat to better explore the correlation between Music and
Dance. We send unified feature $f^{BM},f^{BD},f^{M},f^{D}$ to it, and then get
beat-enhanced feature $f_{M_{e}},f_{D_{e}}$ and beat-guided feature
$f_{M_{g}},f_{D_{g}}$.

$\displaystyle\begin{aligned}
f^{D_{e}},f^{D_{g}}&=BeatBlender_{d}(f^{D},f^{BD})\\\
f^{M_{e}},f^{M_{g}}&=BeatBlender_{m}(f^{M},f^{BM})\end{aligned}$ (2)

In Hubness Reducer block, we aim to tackle the Hubness problem in retrieval
task by constructing a query bank to normalize similarity matrix. Beat-
Enhanced Hubness Reducer operates only during inference stage. We send our
similarity matrix $m_{e}$ to it, and get a normalized matrix $m_{qbnorm}$:

$m_{qbnorm}=HubnessReducer(m_{e})$ (3)

Finally, we can get ranked sequence by $m_{qbnorm}$ for music-to-dance or
dance-to-music retrieval task.

###  3.2 Beat-Aware Music-Dance InfoExtractor

To tackle the challenge of music-dance retrieval, it is crucial to extract
powerful features from both the dance video and the music, enabling the
identification of their similarities. However, a naive approach would involve
directly using global features extracted from CLIP[29] or MERT[22] for
retrieval purposes. While this approach seems straightforward, it has
limitations. Pretrained CLIP[29] and MERT[22] features are learned separately
from other tasks and primarily focus on capturing global representations of
images or music. Consequently, they may fail to capture the specific
correlation between music and dance, hindering the effectiveness of music-
dance retrieval. To overcome these limitations, we introduce the Beat-Aware
Music-Dance InfoExtractor.

####  3.2.1 DanceInfo Extractor

First, we calculate the CLIP[29] features for dance videos $d$. Then, we
evenly divide CLIP[29] feature into $L$ intervals, and perform averaging
operation on each interval. Finally, we obtain a dance feature $f^{d}\in
R^{L\times d_{C}}$, which can represent entire dance, where $d_{C}$ represents
dimension of CLIP feature. We denote process of obtaining CLIP feature as
$\Gamma_{C}$:

$f^{d}=\Gamma_{C}(d)$ (4)

Second, we obtain human pose by sequence Openpose[4], then we calculate the
dance beat $b^{d}\in R^{F_{d}}$ from pose sequence by Dance Beat Detector[32],
where $F_{d}$ represents frames number of dance video. To put it simple, the
main idea of Dance Beat Detector is to consider the moments when the
acceleration of movement is 0 as the beat points. We denote Dance Beat
Detector as $\Phi_{d}$:

$b^{d}=\Phi_{d}(Openpose(d))$ (5)

####  3.2.2 MusicInfo Extractor

First, we calculate the MERT[22] features for music $m$. Then, we execute
interval averaging operation as above CLIP features, and obtain a music
feature $f^{m}\in R^{L\times d_{M}}$, which can represent entire music, where
$d_{M}$ represents dimension of MERT[22] feature, We denote process of
obtaining CLIP[29] feature as $\Gamma_{M}$:

$f^{m}=\Gamma_{M}(m)$ (6)

Second, we directly obtain the dance beat $b^{d}\in R^{F_{m}}$ by Music Beat
Detector from Librosa[23], We denote Music Beat Detector as $\Phi_{m}$:

$b^{m}=\Phi_{m}(m)$ (7)

####  3.2.3 Feature Alignment

Since the dimensions of $f^{d}$, $f^{m}$, $b^{d}$, and $b^{m}$ are all
different, we need to implement a process of unification.

With respect to beat, $b^{m}$ or $b^{d}$ can only take two possible values, 0
or 1, where 1 represents the presence of a beat and 0 represents its absence.
Since beat is not a feature vector, segmenting and averaging as above methods
would result in significant loss of information. To solve this problem, we
first align the frame per second(fps) of $b^{m}$ and $b^{d}$, and then reshape
them into $f^{bm},f^{bd}\in R^{L\times d_{b}}$, respectively, where $d_{b}$ is
dimension of beat feature. Additionally, We have processed all dance and music
data to have equal durations, see Sec. 4.1 for more details.

Next, we use a two layers MLP to adjust their feature dimension of
$f^{bm},f^{bd},f^{m},f^{d}$, respectively, obtaining aligned features
$f^{BM},f^{BD},f^{M},f^{D}\in R^{L\times d_{u}}$, we denote this process as
$\zeta$:

$\displaystyle\begin{aligned} f^{D}&=\zeta_{D}(f^{d})\\\
f^{M}&=\zeta_{M}(f^{m})\\\ f^{BD}&=\zeta_{BD}(b^{d})\\\
f^{BM}&=\zeta_{BM}(b^{m})\end{aligned}$ (8)

###  3.3 Trans-Temporal Beat Blender

As shown in Fig. 2, for both music and dance modalities, we extract two
different kinds of features. However, simply concatenating or adding these
features may not fully utilize their advantages. Moreover, it is important to
consider capturing deep correlation between music and dance. To address these
issues, we introduce a novel and efficient fusion block named Trans-Temporal
Beat Blender.

####  3.3.1 Trans-Temporal Processing

Effective extraction of temporally spanning features significantly impacts the
final results in both dance and music domains. In recent years, transformers
have demonstrated remarkable success in extracting such features. Therefore,
we employ four multi-layer transformer architecture to construct the Trans-
Temporal Process module for $f^{D},f^{M},f^{BD},f^{BM}$ respectively, and then
obtain respective trans-temporal feature
$f^{D_{t}},f^{M_{t}},f^{BD_{t}},f^{BM_{t}}\in R^{L\times d_{u}}$, we denote
this process as $\eta$.

$\displaystyle\begin{aligned} f^{D_{t}}&=\eta_{D}(f^{D})\\\
f^{M_{t}}&=\eta_{M}(f^{M})\\\ f^{BD_{t}}&=\eta_{BD}(f^{BD})\\\
f^{BM_{t}}&=\eta_{BM}(f^{BM})\end{aligned}$ (9)

####  3.3.2 Beat-Enhanced Feature Fusion

Due to the relatively weak correlation between music and dance features, it
will introduce several challenges in retrieval tasks. However, it has been
observed that music beat and dance beat exhibit a strong correspondence,
indicating a potential avenue to resolve this problem.

To leverage this, a intuitive way is to use element-wise addition, but it
fails to effectively capture cross-impact and non-linear relationships between
features. Meanwhile, element-wise multiplication precisely addresses this
issue[12], but is highly susceptible to noise interference. Thus, we combine
above two method to achieve Beat-Enhanced Feature Fusion:

$\displaystyle\begin{aligned} f^{D_{e}}&=MLP([f^{D_{t}}\oplus
f^{BD_{t}},f^{D_{t}}\otimes f^{BD_{t}}])\\\ f^{M_{e}}&=MLP([f^{M_{t}}\oplus
f^{BM_{t}},f^{M_{t}}\otimes f^{BM_{t}}])\\\ \end{aligned}$ (10)

where $f^{M_{e}},f^{D_{e}}\in R^{L\times d_{u}}$, and $MLP$ is used to rectify
dimension.

####  3.3.3 Beat-Guided Information Extraction

On the one hand, after enhance the beat-related information in $f^{D_{t}}$ and
$f^{M_{t}}$ through Beat-Enhanced Feature Fusion, we next propose to guide the
learning of $f^{D}t$ and $f^{M_{t}}$ towards the direction containing beat-
related information, utilizing the Beat-Guided Information Extraction module.

We utilize a Multi-Head Attention layer to construct Beat-Guided Information
Extraction module. In this module, we can consider $f^{BM_{t}}$ and
$f^{BD_{t}}$ information to be a subset of $f^{M_{t}}$ and $f^{D_{t}}$
information, to get beat-guided feature, we can construct Key and Value from
$f^{M_{t}},f^{D_{t}}$, and Query from $f^{BM_{t}},f^{BD_{t}}$, as XPool[11],
we take dance part as example, and music part is similar:

$\displaystyle\begin{aligned}
Q_{b}&=\mathrm{LN}\left(f^{BD_{t}}\right)W_{Q}\\\
K_{d}&=\mathrm{LN}\left(f^{D_{t}}\right)W_{K}\\\
V_{d}&=\mathrm{LN}\left(f^{D_{t}}\right)W_{V}\end{aligned}$ (11)
$head_{i}=\mathrm{softmax}\left(\frac{Q_{b}K_{d}^{T}}{\sqrt{D_{p}}}\right)V_{d}$
(12)

$f^{D_{g}}=[head_{1},\ldots,head_{h}]W_{O}$ (13)

where $\mathrm{LN}$ is a Layer Normalization layer, and
$W_{Q},W_{K},W_{V},W_{O}$ are projection matrices, and $h$ is head number,
$f^{D_{g}},f^{M_{g}}\in R^{L\times d_{u}}$.

Figure 3: BeatDance can effectively capture the underlying correspondence
between dance and music. Given a piece of music/dance, the topk retrieved
musics/dances exhibit a high degree of semantic similarity, such as in terms
of dance/music style, emotional characteristics and etc.. It also demonstrates
the strong expressive capability of BeatDance in feature extraction.
Additionally, demonstration video of more experimental results can be found at
YouTube-URL.

###  3.4 Beat-Enhanced Hubness Reducer

Despite the previous block’s ability to effectively capture the correlation
between music and dance, similar to other retrieval tasks[3], a dance/music
may always be reasonably matched to multiple music/dance, the Hubness Problem
persists. Hubness problem refers to a phenomenon in which certain samples in
high-dimensional data become central hubs, attracting a disproportionate
number of nearest neighbors, which can lead to decreased retrieval accuracy,
biased results, and difficulties in generalization. To tackle this challenge,
we design the Beat-Enhanced Hubness Reducer block based on QBNorm[3].
Additionally, Beat-Enhanced Hubness Reducer only executes during inference
phase.

Specifically, we take music-dance retrieval as example. First, we construct a
QueryBank set $S_{QB}$ from music in training/validation/test set. Second, we
compute querybank-test similarity matrix $m_{qb\\_t}\in R^{N_{qb}\times
N_{t}}$ by query bank $S_{QB}$ and test dances set $S_{T^{d}}$, where $N_{t}$
and $N_{qb}$ represent number of test set and query bank, and then take the
intersection of all $m\in S_{d}{QB}$’s top 1 matching dance to construct the
Hubness-affecting dance set $S_{H_{d}}$. Third, we compute test similarity
matrix $m_{e}\in R^{N_{t}\times N_{t}}$ by test music set $S_{T^{m}}$ and test
dance set $S_{T^{d}}$, and then select all Hubness-affected music $m_{H}$,
whose top 1 matching dance is in Hubness-affecting dance set $S_{H_{d}}$, to
construct Hubness-affected music set $S_{H_{m}}$. Additionally, the
constructions of all similarity matrix stem from corresponding Beat-Enhanced
feature. Fourth, we update test similarity matrix $m_{e}$:

$m_{e}(i,j)=\frac{\exp\left(\beta\cdot
m_{e}(i,j)\right)}{\mathbf{1}^{T}\exp\left[\beta\cdot
m_{qb\\_t}(j)\right]}\quad\text{ if }music_{i}\in S_{H_{m}}$ (14)

where $i$ and $j$ represent the index of $S_{T^{m}}$ and $S_{T^{d}}$.

Finally, we rename new matrix as QBNorm similarity matrix $m_{qbnorm}$, and
can calculate ranked dance for each music from it. Hubness Reducer for dance-
music retrieval is operated similarly.

###  3.5 Training and Inference

####  3.5.1 Training

During training stage, we execute contrastive learning, which encourages
positive pairs to have a high similarity value, while vice versa.
$f^{D_{e}},f^{M_{e}}$ from the Beat-Enhanced Feature Fusion are used for
computing Beat-Enhanced similarity matrix $m_{e}$. Then, we obtain Beat-
Enhanced Loss $\mathcal{L}_{e}$ from $m_{e}$ by infoNCE[27] loss, and we
perform similar operation to obtain Beat-Guided Loss $\mathcal{L}_{g}$:

$\mathcal{L}_{e}^{m\rightarrow
d}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{e^{s\left(f^{M_{e}}_{i},f^{D_{e}}_{i}\right)\cdot\lambda}}{\sum_{j=1}^{B}e^{s\left(f^{M_{e}}_{i},f^{D_{e}}_{j}\right)\cdot\lambda}}$
(15) $\mathcal{L}_{g}^{m\rightarrow
d}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{e^{s\left(f^{M_{g}}_{i},f^{D_{g}}_{i}\right)\cdot\lambda}}{\sum_{j=1}^{B}e^{s\left(f^{M_{g}}_{i},f^{D_{g}}_{j}\right)\cdot\lambda}}$
(16) $\mathcal{L}^{m\rightarrow d}=\mathcal{L}_{e}^{m\rightarrow
d}+\beta\times\mathcal{L}_{g}^{m\rightarrow d}$ (17)

where $s(m,d)$ represents cosine similarity, $B$ is batch size, $\lambda$ is
temperature parameter, $\beta$ is a weighted hyperparameter.
$\mathcal{L}^{d\rightarrow m}$ is computed symmetrically, and
$\mathcal{L}=\mathcal{L}^{m\rightarrow d}+\mathcal{L}^{d\rightarrow m}$ is
used for training our model.

####  3.5.2 Inference

During inference stage, we only construct similarity matrix $m_{e}$ through
$f^{D_{e}}$ and $f^{M_{e}}$. Because Beat-Guided Information Extraction is
designed solely to guide $f^{D_{t}},f^{M_{t}}$ towards the direction that
contains $f^{BD_{t}},f^{BM_{t}}$ information during training phase, thus
unnecessary to consider its influence during inference phase. Then, we send
$m_{e}$ to Beat-Enhanced Hubness Reducer to get a normalized matrix
$m_{qbnorm}$. Finally, we can calculate a ranked sequence from $m_{qbnorm}$
for music-dance or dance-music retrieval task.

Table 1: Comparisons with state-of-the-art results on M-D dataset for music-
to-dance and dance-to-music retrieval. Compared models include: CBVMR[13],
XPool[11], SCFEM [26], MQVR [41], MVPt[34].

Method | Music $\Longrightarrow$ Dance | Dance $\Longrightarrow$ Music
---|---|---
| Recall@1/10/50/100$\uparrow$ | MeanR/MedianR$\downarrow$ | Recall@1/10/50/100$\uparrow$ | MeanR/MedianR$\downarrow$
CBVMR | 0.83/6.35/20.71/30.61 | 245.5/333.91 | 1.24/6.11/20.79/31.02 | 236.5/333.64
SCFEM | 0.99/7.76/23.10/35.81 | 196.0/306.05 | 0.91/8.25/23.27/35.31 | 192.0/305.65
MQVR | 1.65/8.91/26.90/39.60 | 152.5/263.80 | 1.24/9.49/26.90/39.11 | 152.0/265.36
MVPt | 1.57/8.25/26.24/38.78 | 162.5/258.15 | 1.23/9.46/27.81/39.42 | 166.0/254.81
XPool | 1.57/9.41/27.72/41.50 | 148.0/248.79 | 1.49/8.83/28.55/41.58 | 148.0/253.80
BeatDance | 2.48/13.12/32.26/44.06 | 128.0/239.81 | 2.97/13.04/32.34/44.55 | 127.0/238.77
|  |  |  |

##  4 EXPERIMENT

###  4.1 Dataset

To evaluate and benchmark existing methods in the Music-Dance retrieval task,
we introduce M-D dataset, which is the first large-scale open-source dataset
for this task. Fig. 3 illustrates some examples of this dataset. The dataset
is sourced from Bilibili[2], the most popular video-sharing platform among
young people in China. To ensure the quality and popularity of the dance
videos, we collect videos uploaded between May 2018 and September 2023 in the
dance category with over 100,000 likes. This ensures the excellence and
popularity of the dataset.

The Music-Dance dataset encompasses various types of dance videos, including
dance performances, tutorials, and practices in daily life. Through meticulous
manual selection, we curate approximately 12,000 high-quality dance
performance videos. The dataset is randomly shuffled and split into training,
validation, and test sets in an 8:1:1 ratio. Statistical analysis of the
dataset reveals that it contains both single-person and group dance
performances, covering a wide range of dance genres such as Ballet,
Contemporary, Hip-hop, Jazz, Tap, Latin, and more. It also includes a diverse
selection of music genres, including Pop, Rock, Hip-hop, Electronic, Jazz, and
others. Moreover, in addition to the dance and music video data, we provide
dance beats extracted by Openpose[4] and music beats extracted by Librosa[23].
These beats are uniformly sampled at 10 frames per second (fps) and
represented as binary values (1 for presence of beat, 0 for absence of beat).
To ensure consistency in the analysis and evaluation of beat-based approaches,
we consider a consistent 10-second segment from the middle of each dance video
in our task. This ensures that all videos in the dataset have the same
duration, allowing us to attribute any performance improvements solely to the
presence of beats, independent of duration information.

###  4.2 Evaluation

Similar to other multi-modal retrieval tasks, such as text-video retrieval[41,
11], video-music retrieval[34, 13], we introduce Recall@K (higher is better)
and Mean/Median Rank (lower is better) as evaluation metrics. To explore
whether our method fully utilizes beats, we also introduce BS@K[32](averaged
Beat Similarity between ground truth and top k query results). We take dance-
music retrieval as example:

$BS_{d\rightarrow m}=\frac{1}{\left|B^{m}\right|}\sum_{t^{m}\in
B^{m}}\exp\left\\{-\frac{\min_{t^{d}\in
B^{d}}\left\|t^{d}-t^{m}\right\|^{2}}{2\sigma^{2}}\right\\}$ (18)

where, $t^{m}$ represents the moment when music beats occur. Likewise,
$BS_{m\rightarrow d}$ is defined symmetrically in music-dance retrieval.

###  4.3 Implementation Detail

In our experiments, we employ CLIP’s ViT-B/32[29] image encoder and
MERT-95M[22] as the base feature extractors. We initialize all encoder
parameters using their pre-trained weights. The base features from CLIP[29]
and MERT[22] are precomputed, and the interval $L$ between base features is
set to 10. The beat dim $d_{b}$ is set to 10. The feature unification
dimension size is set to $d_{u}$=256. We initialize our logit scaling
parameter $\lambda$ using the value from the pre-trained CLIP[29] model. For
all transformers, we use a hidden dimension of 256, 6 layers, 4 heads, and a
dropout rate of 0.3 (except for Beat-Guided Information Extraction, which uses
a dropout rate of 0.3). During training, we set the batch size to 32 and the
learning rate for the model parameters to 1e-5. We optimize our model for 150
epochs using the AdamW optimizer with a weight decay of 0.2. The learning rate
is decayed using a cosine schedule. We use training set to construct query
bank. Loss weight $\beta$ is set 0.4 for constrastive learning.

###  4.4 Comparison

To evaluate the performance of our method, we compared it with recent related
works. Due to the limited availability of open-source code for video-music
retrieval, let alone music-dance retrieval, we only reproduced the classic
algorithms MVPt[34] and CBVMR[13] in this field. Additionally, we migrated
models from other multimodal retrieval fields, such as XPool[11] and MQVR[41]
in text-video retrieval and SCFEM[26] in image-music retrieval. Specifically,
for MVPt, since the music encoder(DeepSim) used in MVPt[34] is not open-
sourced, we replaced it with MERT[22]. For CBVMR, due to the age of CBVMR, we
replace its video encoder and music encoder with CLIP[29] and MERT[22]
respectively to ensure fairness. For XPool, we use averaged MERT[22] feature
of music instead of the CLIP feature of text. For MQVR, we first obtain
MERT[22]/CLIP[29] feature, and then uniformly divide it into 5 intervals,
Averaged MERT[22]/CLIP[29] feature of each interval represent one query in
multi-query scene in MQVR. For SCFEM, we average the feature obtained by
CLIP[29] over the time dimension to replace the original image feature.

As shown in Tab. 1, our proposed BeatDance obviously outperforms all baseline
methods, including CBVMR, SCFEM, MQVR, MVPt, and XPool, by a significant
margin across various evaluation metrics. Specifically, in the Music-to-Dance
task, BeatDance achieves superior performance compared to other models, with
Recall@1/10/50/100 values of 2.48/13.12/32.26/44.06, respectively.
Additionally, BeatDance obtains lower MeanR/MedianR values, specifically
128.0/239.81. These results indicate that BeatDance significantly improves the
accuracy of retrieving dance videos given music inputs. Similarly, in the
Dance-to-Music task, BeatDance continues to outperform the baseline models. It
achieves a Recall@1/10/50/100 of 2.97/13.04/32.34/44.55, surpassing all other
models. The MeanR/MedianR values for BeatDance in this task are 127.0/238.77,
which are lower compared to the baseline models. The superior performance of
BeatDance can be attributed to its ability to capture and learn the
correlation between music and dance videos more effectively. By considering
beat alignment, BeatDance leverages the temporal structure and rhythmic
patterns present in both the music and dance modalities. This allows the model
to better align and synchronize the representations of music and dance,
resulting in improved retrieval performance. The significant improvements
achieved by BeatDance across all evaluation metrics establish its superiority
over existing methods and position it as the current state-of-the-art (SOTA)
approach in the field of music-dance retrieval.

Table 2: Effect of each component of BeatDance on M-D datasets for music-to-
dance and dance-to-music retrieval.

Method | Music $\Longrightarrow$ Dance | Dance $\Longrightarrow$ Music
---|---|---
Recall@1/10/50/100$\uparrow$ | MeanR/MedianR$\downarrow$ | Recall@1/10/50/100$\uparrow$ | MeanR/MedianR$\downarrow$
Baseline | 2.15/11.87/29.29/42.33 | 142.0/256.28 | 2.48/12.05/28.38/41.50 | 145.0/257.73
w/o Trans-Temporal Processing |  2.56/11.80/27.81/40.43 | 166.0/281.25 | 2.56/11.88/27.48/39.93 | 163.5/284.41
w/o Beat-Enhanced Feature Fusion | 1.98/12.21/29.13/42.41 | 140.0/258.32 | 2.39/11.37/29.62/41.34 | 147.0/260.39
w/o Beat-Guided Information Extraction | 2.15/10.23/28.30/41.91 | 149.5/252.73 | 2.23/10.81/27.48/41.50 | 147.0/253.71
w/o Hubness Reducer | 2.48/12.29/32.01/43.89 | 136.5/240.21 | 2.48/12.71/30.86/44.31 | 130.0/239.58
Openpose→Mediapipe | 2.15/12.05/30.61/43.47 | 135.0/239.94 | 2.89/11.72/28.55/43.14 | 134.0/238.82
Full BeatDance | 2.48/13.12/32.26/44.06 | 128.0/239.81 | 2.97/13.04/32.34/44.55 | 127.0/238.77
|  |  |  |

Table 3: Effect of Fusion Mode in music-to-dance retrieval.

Method | Recall@1/10/50/100$\uparrow$ | MeanR/MedianR$\downarrow$
---|---|---
Baseline | 2.15/11.87/29.29/42.33 | 142.0/256.28
Beat Loss | 1.98/12.21/29.13/42.41 | 140.0/258.32
Beat-Enhanced Feature Fusion(B) | 1.65/9.82/25.91/40.35 | 155.5/258.76
Beat-Enhanced Feature Fusion(A) | 2.15/12.21/30.12/43.56 | 139.5/245.26
Beat-Guided Information Extraction | 2.15/10.23/28.30/41.91 | 149.5/252.73
BeatDance | 2.48/12.29/32.01/43.89 | 136.5/240.2
|  |

Table 4: Exploring BeatDance effect in music-to-dance retrieval. ”+” means
introduction of BeatDance.

Method | Recall@1/10/50/100$\uparrow$ | MeanR/MedianR$\downarrow$ | BS@1/5 $\uparrow$
---|---|---|---
CBVMR | 0.83/6.35/20.71/30.61 | 245.5/333.91 | 85.11/84.97
CBVMR+ | 0.99/8.33/23.93/37.21 | 179.5/276.02 | 85.32/85.13
XPool | 1.57/9.41/27.72/41.50 | 148.0/248.79 | 85.11/85.00
XPool+ | 2.15/10.40/29.21/42.57 | 140.5/239.08 | 85.26/85.04
Baseline | 2.15/11.87/29.29/42.33 | 142.0/256.28 | 85.15/85.16
Baseline+ | 2.56/11.88/31.60/44.22 | 129.0/234.11 | 85.30/85.18
|  |  |

###  4.5 Ablation Study

####  4.5.1 Trans-Temporal Processing

To better capture the trans-temporal information in music and dance related
feature, we propose Trans-Temporal Processing. As shown in Tab. 2, the
introduction of it makes great improvement in Recall@1/10/50/100(+2.55 in
average) and Median/Mean Rank(+40.40 in average), which demonstrates its great
effectiveness.

####  4.5.2 Beat-Enhanced Feature Fusion

To better enhance global information with corresponding beat information, we
propose Beat-Enhance Fusion. As shown in Tab. 2, the introduction of it makes
great improvement in Recall@1/10/50/100(+1.80 in average) and Median/Mean
Rank(+18.03 in average), which demonstrates its great effectiveness.

####  4.5.3 Beat-Guided Information Extraction

To better guided music and dance related feature training direction containing
beat information, we propose Beat-Guided Information Extraction. As shown in
Tab. 2, the introduction of it makes great improvement in
Recall@1/10/50/100(+2.53 in average) and in Median/Mean Rank(+17.34 in
average), which demonstrates its great effectiveness.

####  4.5.4 Beat-Enhanced Hubness Reducer

To address the Hubness problem, we design Beat-Enhanced Hubness Reducer. As
shown in Tab. 2, the introduction of it makes great improvement in
Recall@1/10/50/100(+0.47 in average) and in Median/Mean Rank(+3.18 in
average), which demonstrates its great effectiveness.

####  4.5.5 Pose Estimatior

In the process of generating Dance Beats, pose estimation plays an important
role, and we explore two popular methods Openpose and Mediapipe as our Pose
Estimators. From Tab. 2, it can be observed that the performance based on
Openpose is improved in Recall@1/10/50/100(+1.28 in average) and in
Median/Mean Rank(+3.55 in average), compared to Mediapipe. This is because our
dataset includes both multi-person and single-person dances, and in multi-
person dances, Mediapipe focuses only on one dancer, neglecting the influence
of others.

####  4.5.6 Fusion Mode

It is well known that beat information is crucial in dance and music. How to
effectively integrate beat information with related music and dance features
is an important problem. Thus, we also explore other feature fusion methods in
addition to BeatDance.In Tab. 3, Beat Loss represents the separate contrastive
learning training of global features and beat features after passing through
the Trans-Temporal Process module. Beat-Enhanced Process(B) denotes the
process in which global features and beats are first processed through the
Beat-Enhanced Feature Fusion module, followed by the Trans-Temporal Process
module, and then subjected to contrastive learning training. Beat-Enhanced
Feature Fusion (A) refers to the process where global features and beats are
initially processed through their respective Trans-Temporal Process modules,
followed by the Beat-Enhanced Feature Fusion module, and subsequently undergo
contrastive learning training. Beat-Guided Information Extraction signifies
the process in which global features and beats are processed through their
respective Trans-Temporal Process modules, followed by the Beat-Guided
Information Extraction module, before undergoing contrastive learning
training. BeatDance represents the BeatDance without utilizing the Beat-
Enhanced Hubness Reducer module.As Tab. 3 shows, BeatDance significantly
outperforms other fusion methods on all metrics and it can be observed that
the standalone use of Beat-Guided Information Extraction and Beat-Enhanced
Feature Fusion yields inferior results.

###  4.6 Model Analysis

####  4.6.1 Beat Similarity Analysis

BeatDance integrates beat information and global features, naturally enhancing
the correspondence between dance and music at the Beat level. BS@K can be a
effective metric for evaluating if beat information is effectively utilized.
As Tab. 4 shows, the introduction of BeatDance resulted in a improvement in
BS@K on all models. It is worth noting that the limited improvement can be
attributed to two factors: the minor role of beat in the retrieval task and
the inherent limitations of the computational formula18. Even when provided
with an beat array consisting entirely of ones, averaged Beat Similarity
between it and all ground truth can still reach 69.86%.

Table 5: Comparson with others on classification task, including CBVMR[13],
XPool[11], SCFEM [26], MVPt+[34].

Method | Genre | Instrument | Mood
---|---|---|---
CBVMR | 50.58 | 64.60 | 61.14
SCFEM | 53.88 | 69.22 | 61.22
MVPt | 54.37 | 67.90 | 62.05
XPool | 54.54 | 66.91 | 62.05
BeatDance | 57.10 | 70.38 | 63.86
|  |  |

(a) Music

(b) Dance

Figure 4: t-SNE[39] visulization of learned features. 2000 randomly sampled
data pairs are chosen. It can be observed that music representations and dance
representations exhibit a remarkably high degree of similarity in their
distribution.

####  4.6.2 Model Agnositic Analysis

It is worth noting that BeatDance is essentially a framework with good
generality, which is easy to extend to other models. Therefore, we conduct
extra experiments on CBVMR and XPool. As shown in Tab. 4, BeatDance greatly
improved efficiency of all models, demonstrating its strong generalizability.

####  4.6.3 Downstream Task Analysis

To validate the expressive power of feature vectors generated by BeatDance, we
introduce three classification tasks: music genre classification, music
emotion classification, and music instrument classification. We first employ
well-known PANN[17] to assign genre, mood, and instrument labels to music for
classification task. There are a total of 7 emotion categories, 23 genre
categories, and 18 instrument categories. Then, we append two MLP layers to
feature extracted by each model for subsequent classification. Accuracy was
used as the evaluation metric. As shown in Tab. 5, BeatDance outperformed
other models significantly in all three tasks, demonstrating its strong
information extraction capabilities.

####  4.6.4 Feature Distribution Analysis

To explore feature representation capabilities of BeatDance, we randomly
select 2000 instances from our dataset and obtain music representations and
dance representations after processing them with BeatDance. Firstly, we apply
K-Means clustering to assign cluster labels to all representations.
Subsequently, we employ t-SNE[39] for dimensionality reduction, projecting the
high-dimensional features into a two-dimensional space. Finally, we visualize
all 2000 data points. Fig. 4(a) and Fig. 4(b) illustrate the visualizations of
the music representations and video representations respectively. Remarkably,
it can be observed that the music representations and dance representations
exhibit a high degree of similarity in their distributions, providing evidence
for the efficient feature representation capabilities of BeatDance.

##  5 CONCLUSION

In this work, we have introduced BeatDance, a novel beat-based model-agnostic
contrastive learning framework designed to better explore correlation between
music and dance. In BeatDance, the Beat-Aware Music-Dance InfoExtractor, the
Trans-Temporal Beat Blender, and the Beat-Enhanced Hubness Reducer are
proposed to jointly facilitate the music-dance retrieval performance. To
facilitate future research endeavors, we have also introduced the M-D dataset,
the first large-scale open-source dataset specifically curated for the music-
dance retrieval task. This dataset encompasses a diverse range of dance and
music genres, providing a valuable resource for researchers in this field. Our
experimental results have demonstrated the superiority of our proposed method
compared to other baselines in the music-dance retrieval domain. We believe
that this pioneering work will inspire and encourage more researchers and
practitioners to explore and advance the capabilities of music-dance retrieval
systems.

## References

  * [1] Omid Alemi, Jules Françoise, and Philippe Pasquier.  Groovenet: Real-time music-driven dance movement generation using artificial neural networks.  networks, 8(17):26, 2017.
  * [2] Bilibili.  Bilibili, 2023.  2023.9.30.
  * [3] Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie.  Cross modal retrieval with querybank normalisation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5194–5205, 2022.
  * [4] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  * [5] Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, and Yuexian Zou.  Ssvmr: Saliency-based self-training for video-music retrieval.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [6] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.  Jukebox: A generative model for music.  arXiv preprint arXiv:2005.00341, 2020.
  * [7] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang.  Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [8] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts.  Gansynth: Adversarial neural audio synthesis.  In International Conference on Learning Representations, 2018.
  * [9] Joao P Ferreira, Thiago M Coutinho, Thiago L Gomes, José F Neto, Rafael Azevedo, Renato Martins, and Erickson R Nascimento.  Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio.  Computers & Graphics, 94:11–21, 2021.
  * [10] Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré.  It’s raw! audio generation with state-space models.  In International Conference on Machine Learning, pages 7616–7633. PMLR, 2022.
  * [11] Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu.  X-pool: Cross-modal language-video attention for text-video retrieval.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5006–5015, 2022.
  * [12] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  In Proceedings of the 26th international conference on world wide web, pages 173–182, 2017.
  * [13] Sungeun Hong, Woobin Im, and Hyun S Yang.  Cbvmr: content-based video-music retrieval using soft intra-modal structure constraint.  In Proceedings of the 2018 ACM on international conference on multimedia retrieval, pages 353–361, 2018.
  * [14] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck.  Music transformer: Generating music with long-term structure.  In International Conference on Learning Representations, 2018.
  * [15] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang.  Dance revolution: Long-term dance generation with music via curriculum learning.  arXiv preprint arXiv:2006.06119, 2020.
  * [16] Jinwoo Kim, Heeseok Oh, Seongjean Kim, Hoseok Tong, and Sanghoon Lee.  A brand new dance partner: Music-conditioned pluralistic dancing controlled by multiple dance genres.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3490–3500, 2022.
  * [17] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020.
  * [18] Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville.  Melgan: Generative adversarial networks for conditional waveform synthesis.  Advances in neural information processing systems, 32, 2019.
  * [19] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz.  Dancing to music.  Advances in neural information processing systems, 32, 2019.
  * [20] Bochen Li and Aparna Kumar.  Query by video: Cross-modal music retrieval.  In ISMIR, pages 604–611, 2019.
  * [21] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13401–13412, 2021.
  * [22] Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et al.  Mert: Acoustic music understanding model with large-scale self-supervised training.  arXiv preprint arXiv:2306.00107, 2023.
  * [23] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto.  librosa: Audio and music signal analysis in python.  In Proceedings of the 14th python in science conference, volume 8, pages 18–25, 2015.
  * [24] Daniel McKee, Justin Salamon, Josef Sivic, and Bryan Russell.  Language-guided music recommendation for video via prompt analogies.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14784–14793, 2023.
  * [25] Aashiq Muhamed, Liang Li, Xingjian Shi, Suri Yaddanapudi, Wayne Chi, Dylan Jackson, Rahul Suresh, Zachary C Lipton, and Alex J Smola.  Symbolic music generation with transformer-gans.  In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 408–417, 2021.
  * [26] Takayuki Nakatsuka, Masahiro Hamasaki, and Masataka Goto.  Content-based music-image retrieval using self-and cross-modal feature embedding memory.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2174–2184, 2023.
  * [27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [28] Yagya Raj Pandeya and Joonwhoan Lee.  Deep learning-based late fusion of multimodal information for emotion classification of music video.  Multimedia Tools and Applications, 80:2887–2905, 2021.
  * [29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [30] Xuanchi Ren, Haoran Li, Zijian Huang, and Qifeng Chen.  Self-supervised dance video synthesis conditioned on music.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 46–54, 2020.
  * [31] Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie-Yan Liu.  Popmag: Pop music accompaniment generation.  In Proceedings of the 28th ACM international conference on multimedia, pages 1198–1206, 2020.
  * [32] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu.  Bailando: 3d dance generation by actor-critic gpt with choreographic memory.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11050–11059, 2022.
  * [33] Guofei Sun, Yongkang Wong, Zhiyong Cheng, Mohan S Kankanhalli, Weidong Geng, and Xiangdong Li.  Deepdance: music-to-dance motion choreography with adversarial learning.  IEEE Transactions on Multimedia, 23:497–509, 2020.
  * [34] Dídac Surís, Carl Vondrick, Bryan Russell, and Justin Salamon.  It’s time for artistic correspondence in music and video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10564–10574, 2022.
  * [35] Taoran Tang, Jia Jia, and Hanyang Mao.  Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1598–1606, 2018.
  * [36] Shuhei Tsuchida, Satoru Fukayama, and Masataka Goto.  Query-by-dancing: a dance music retrieval system based on body-motion similarity.  In MultiMedia Modeling: 25th International Conference, MMM 2019, Thessaloniki, Greece, January 8–11, 2019, Proceedings, Part I 25, pages 251–263. Springer, 2019.
  * [37] Guillermo Valle-Pérez, Gustav Eje Henter, Jonas Beskow, Andre Holzapfel, Pierre-Yves Oudeyer, and Simon Alexanderson.  Transflower: probabilistic autoregressive dance generation with multimodal attention.  ACM Transactions on Graphics (TOG), 40(6):1–14, 2021.
  * [38] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  In 9th ISCA Speech Synthesis Workshop, pages 125–125.
  * [39] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [40] Sean Vasquez and Mike Lewis.  Melnet: A generative model for audio in the frequency domain.  arXiv preprint arXiv:1906.01083, 2019.
  * [41] Zeyu Wang, Yu Wu, Karthik Narasimhan, and Olga Russakovsky.  Multi-query video retrieval.  In European Conference on Computer Vision, pages 233–249. Springer, 2022.
  * [42] Zijie Ye, Haozhe Wu, Jia Jia, Yaohua Bu, Wei Chen, Fanbo Meng, and Yanfeng Wang.  Choreonet: Towards music to dance synthesis with choreographic action unit.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 744–752, 2020.
  * [43] Jiashuo Yu, Junfu Pu, Ying Cheng, Rui Feng, and Ying Shan.  Self-supervised learning of music-dance representation through explicit-implicit rhythm synchronization.  arXiv preprint arXiv:2207.03190, 2022.
  * [44] Haolin Zhuang, Shun Lei, Long Xiao, Weiqin Li, Liyang Chen, Sicheng Yang, Zhiyong Wu, Shiyin Kang, and Helen Meng.  Gtn-bailando: Genre consistent long-term 3d dance generation based on pre-trained genre token network.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [45] Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia.  Music2dance: Dancenet for music-driven dance generation.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2):1–21, 2022.


$i_{\mathbb{R}}(P)\subset\mathbb{R}^{n}$. Note that the value
$\operatorname{vol}_{L}(P)$ is independent of the choice of $i$.

######  Corollary B.4 ([SM20b]*Theorem 2).

Let $(\Gamma,L)$ be a strongly connected $n$-dimensional periodic graph, and
let $x_{0}\in V_{\Gamma}$. Let $f_{b}$ and $f_{s}$ be the quasi-polynomials
corresponding to the functions $b:i\mapsto b_{\Gamma,x_{0},i}$ and $s:i\mapsto
s_{\Gamma,x_{0},i}$ (Theorem 3.15). Then the following assertions hold.

  1. (1)

Each constituent of $f_{b}$ is a polynomial of degree $n$, and its leading
coefficient is $\\#(V_{\Gamma}/L)\cdot\operatorname{Vol}_{L}(P_{\Gamma})$.

  2. (2)

Let $Q_{0},\ldots,Q_{N-1}$ be the constituents of $f_{s}$. Then, for each
$i=0,\ldots,N-1$, we have $\deg Q_{i}\leq n-1$. Furthermore, let $a_{i}$
denote the coefficient of $Q_{i}$ of degree $n-1$. Then, we have
$\frac{1}{N}\sum_{i=0}^{N-1}a_{i}=n\cdot\\#(V_{\Gamma}/L)\cdot\operatorname{Vol}_{L}(P_{\Gamma})$.

###### Proof.

We fix an injective periodic realization $\Phi$ satisfying $\Phi(x_{0})=0$. By
$\Phi$, we identify $V_{\Gamma}$ with the subset of $L_{\mathbb{R}}$. We take
$C^{\prime}_{1}$ and $C^{\prime}_{2}$ to satisfy Theorem B.2. Then, for each
$d\in\mathbb{Z}_{\geq 0}$, we have

$\\#((d-C^{\prime}_{2})P_{\Gamma}\cap V_{\Gamma})\leq
b_{\Gamma,x_{0},d}\leq\\#((d+C^{\prime}_{1})P_{\Gamma}\cap V_{\Gamma}).$

Therefore, (1) follows from the following fact (cf. [BR]*Lemma 3.19):

$\\#(V_{\Gamma}/L)\cdot\operatorname{Vol}_{L}(P_{\Gamma})=\lim_{d\to\infty}\frac{1}{d^{n}}\cdot\\#\bigl{(}dP_{\Gamma}\cap
V_{\Gamma}\bigr{)}.$

(2) follows from (1). ∎

######  Remark B.5.

In crystallography, the invariant $\frac{1}{N}\sum_{i=0}^{N-1}a_{i}$ in
Corollary B.4(2) is called the topological density (cf. [GKBS96]).

######  Example B.6.

For the Wakatsuki graph (see Example 3.9), the invariant
$\frac{1}{N}\sum_{i=0}^{N-1}a_{i}$ in Corollary B.4(2) for the start point
$x_{0}=v^{\prime}_{0}$ is given by
$\frac{1}{2}\left(\frac{9}{2}+\frac{9}{2}\right)=\frac{9}{2}$ (see Example
3.17). The same invariant for the start point $x_{0}=v^{\prime}_{2}$ is also
given by $\frac{1}{2}(3+6)=\frac{9}{2}$. They are actually equal to
$2\cdot\\#(V_{\Gamma}/L)\cdot\operatorname{Vol}_{L}(P_{\Gamma})=2\cdot
3\cdot\frac{3}{4}=\frac{9}{2}$ as proved in Corollary B.4(2).

##  Appendix C Ehrhart theory

In this section, we discuss a variant of the Ehrhart theory (Theorem C.4),
which is necessary for the proof of Theorem 4.4. The difference from the usual
Ehrhart theory is that the center $v$ of the dilation need not be the origin,
and the dilation factor may be shifted by a constant $\alpha$.

######  Definition C.1.

Let $P\subset\mathbb{R}^{N}$ be a rational polytope, and let
$v\in\mathbb{R}^{N}$ and $\alpha\in\mathbb{R}$.

  1. (1)

We define a function $h_{P,v,\alpha}:\mathbb{Z}\to\mathbb{Z}$ as follows

$h_{P,v,\alpha}(d):=\\#\left((v+(d+\alpha)P)\cap\mathbb{Z}^{N}\right).$

Here, we define $tP=\emptyset$ for $t<0$.

  2. (2)

Let $\operatorname{relint}(P)$ denote the relative interior of $P$. Then, we
also define a function
$\overset{\circ}{h}_{P,v,\alpha}:\mathbb{Z}\to\mathbb{Z}$ as follows

$\overset{\circ}{h}_{P,v,\alpha}(d):=\\#\left((v+(d+\alpha)\cdot\operatorname{relint}({P}))\cap\mathbb{Z}^{N}\right).$

Here, we define $t\cdot\operatorname{relint}({P})=\emptyset$ for $t\leq 0$.

  3. (3)

Let

$H_{P,v,\alpha}(t):=\sum_{i\in\mathbb{Z}}h_{P,v,\alpha}(i)t^{i},\qquad\overset{\circ}{H}_{P,v,\alpha}(t):=\sum_{i\in\mathbb{Z}}\overset{\circ}{h}_{P,v,\alpha}(i)t^{i}$

denote their generating functions.

######  Remark C.2.

  1. (1)

In the definition above, we have defined $0\cdot P=\\{0\\}$ but
$0\cdot\operatorname{relint}({P})=\emptyset$. This definition is necessary for
the equations ($\spadesuit$) in the proof of Theorem C.4.

  2. (2)

The usual Ehrhart theory (cf. [BR]) treats the case where $v=0$ and
$\alpha=0$. In [dVY21]*Section 3, de Vries and Yoshinaga discuss
$h_{P,v,\alpha}$ when $\alpha=0$.

######  Lemma C.3.

We have $h_{P,v,\alpha}=h_{-P,-v,\alpha}$ and
$\overset{\circ}{h}_{P,v,\alpha}=\overset{\circ}{h}_{-P,-v,\alpha}$.

###### Proof.

The first assertion follows from

$\displaystyle\\#\left((v+(d+\alpha)P)\cap\mathbb{Z}^{N}\right)$
$\displaystyle=\\#\left((-(v+(d+\alpha)P))\cap\mathbb{Z}^{N}\right)$
$\displaystyle=\\#\left((-v+(d+\alpha)(-P)\cap\mathbb{Z}^{N}\right).$

The second assertion can be proved in the same way. ∎

######  Theorem C.4.

Let $P\subset\mathbb{R}^{N}$ be a rational polytope of dimension $M$, and let
$v\in\mathbb{R}^{N}$ and $\alpha\in\mathbb{R}$. Then, the following assertions
hold:

  1. (1)

$h_{P,v,\alpha}$ is a quasi-polynomial on $d\geq-\alpha$, and
$h_{P,v,\alpha}(d)=0$ holds for $d<-\alpha$.

  2. (2)

$\overset{\circ}{h}_{P,v,\alpha}$ is a quasi-polynomial on $d>-\alpha$, and
$\overset{\circ}{h}_{P,v,\alpha}(d)=0$ holds for $d\leq-\alpha$.

  3. (3)

$H_{P,v,\alpha}(1/t)=(-1)^{M+1}\overset{\circ}{H}_{P,-v,-\alpha}(t)$.

  4. (4)

Let $f_{P,v,\alpha}$ be the corresponding quasi-polynomial to the function
$h_{P,v,\alpha}$ on $d\geq-\alpha$. Then, we have
$f_{P,v,\alpha}(-i)=(-1)^{M}\overset{\circ}{h}_{P,-v,-\alpha}(i)$ for any
$i\in\mathbb{Z}_{>\alpha}$.

###### Proof.

For a subset $S\subset\mathbb{R}^{N+1}$, let

$\sigma_{S}({\bf z})=\sigma_{S}(z_{1},\ldots,z_{N+1}):=\sum_{{\bf m}\in
S\cap\mathbb{Z}^{N+1}}{\bf z}^{\bf m}$

denote the integer-point transform of $S$ (see [BR]*Section 3.3). This is a
formal sum of Laurent monomials ${\bf z}^{\bf m}=z_{1}^{m_{1}}\cdots
z_{N+1}^{m_{N+1}}$ for ${\bf m}=(m_{1},\ldots,m_{N+1})\in
S\cap\mathbb{Z}^{N+1}$.

Let $K:=\mathbb{R}_{\geq 0}\bigl{(}\\{1\\}\times
P\bigr{)}\subset\mathbb{R}^{N+1}$ be the cone over $P$. Then, we have

($\spadesuit$)
$\displaystyle\begin{split}H_{P,v,\alpha}(t)&=\sigma_{(-\alpha,v)+K}(t,1,\ldots,1),\\\
\overset{\circ}{H}_{P,-v,-\alpha}(t)&=\sigma_{(\alpha,-v)+\operatorname{relint}(K)}(t,1,\ldots,1).\end{split}$

Furthermore, we have

$\sigma_{(-\alpha,v)+K}(z_{1}^{-1},\ldots,z_{N+1}^{-1})=(-1)^{M+1}\sigma_{(\alpha,-v)+\operatorname{relint}(K)}(z_{1},\ldots,z_{N+1})$

by [BR]*Exercises 4.5 and 4.6. Therefore, we get the desired equality in (3).

The second assertions of (1) and (2) are obvious by the definition of
$h_{P,v,\alpha}$ and $\overset{\circ}{h}_{P,v,\alpha}$.

By taking a triangulation of $P$, the first assertions in (1) and (2) can be
reduced to the same assertions for simplicial polytopes $P$. Assume that $P$
is simplicial. Let $v_{1},\ldots,v_{M+1}$ be the vertices of $P$. Since $P$ is
rational polytope, we may write $v_{i}=u_{i}/a_{i}$ for some
$u_{i}\in\mathbb{Z}^{N}$ and $a_{i}\in\mathbb{Z}_{>0}$. We set
$D,D^{\circ}\subset\mathbb{R}^{N+1}$ by

$\displaystyle D$
$\displaystyle:=\left\\{\sum_{i=1}^{M+1}\alpha_{i}(a_{i},u_{i})\ \middle|\
\alpha_{1},\ldots,\alpha_{M+1}\in[0,1)\right\\},$ $\displaystyle D^{\circ}$
$\displaystyle:=\left\\{\sum_{i=1}^{M+1}\alpha_{i}(a_{i},u_{i})\ \middle|\
\alpha_{1},\ldots,\alpha_{M+1}\in(0,1]\right\\}.$

Then, by [BR]*Theorem 3.5, we have

$\displaystyle\sigma_{(-\alpha,v)+K}(z_{1},{\bf z})$
$\displaystyle=\frac{\sigma_{(-\alpha,v)+D}(z_{1},{\bf
z})}{\prod_{i=1}^{M+1}(1-z_{1}^{a_{i}}{\bf z}^{u_{i}})},$
$\displaystyle\sigma_{(-\alpha,v)+\operatorname{relint}(K)}(z_{1},{\bf z})$
$\displaystyle=\frac{\sigma_{(-\alpha,v)+D^{\circ}}(z_{1},{\bf
z})}{\prod_{i=1}^{M+1}(1-z_{1}^{a_{i}}{\bf z}^{u_{i}})},$

where ${\bf z}=(z_{2},\ldots,z_{N+1})$. Therefore, we have

$\displaystyle H_{P,v,\alpha}(t)$
$\displaystyle=\sigma_{(-\alpha,v)+K}(t,1,\ldots,1)=\frac{\sigma_{(-\alpha,v)+D}(t,1,\ldots,1)}{\prod_{i=1}^{M+1}(1-t^{a_{i}})},$
$\displaystyle\overset{\circ}{H}_{P,v,\alpha}(t)$
$\displaystyle=\sigma_{(-\alpha,v)+\operatorname{relint}(K)}(t,1,\ldots,1)=\frac{\sigma_{(-\alpha,v)+D^{\circ}}(t,1,\ldots,1)}{\prod_{i=1}^{M+1}(1-t^{a_{i}})}.$

Here, we have

$\displaystyle\deg\sigma_{(-\alpha,v)+D}(t,1,\ldots,1)$
$\displaystyle<-\alpha+\sum_{i=1}^{M+1}a_{i},$
$\displaystyle\deg\sigma_{(-\alpha,v)+D^{\circ}}(t,1,\ldots,1)$
$\displaystyle\leq-\alpha+\sum_{i=1}^{M+1}a_{i}.$

Therefore, we can conclude that $h_{P,v,\alpha}$ is a quasi-polynomial on
$d\geq-\alpha$, and that $\overset{\circ}{h}_{P,v,\alpha}$ is a quasi-
polynomial on $d>-\alpha$. We complete the proof of (1) and (2).

Finally, we prove (4). Since $f_{P,v,\alpha}$ is a quasi-polynomial, we have

$\sum_{i\in\mathbb{Z}_{<-\alpha}}{f_{P,v,\alpha}(i)}t^{i}=-\sum_{i\in\mathbb{Z}_{\geq-\alpha}}{f_{P,v,\alpha}(i)}t^{i}$

as rational functions (cf. [BR]*Exercise 4.7). Therefore, we have

$\displaystyle\sum_{i\in\mathbb{Z}_{>\alpha}}{f_{P,v,\alpha}(-i)}t^{-i}$
$\displaystyle=\sum_{i\in\mathbb{Z}_{<-\alpha}}{f_{P,v,\alpha}(i)}t^{i}$
$\displaystyle=-\sum_{i\in\mathbb{Z}_{\geq-\alpha}}{f_{P,v,\alpha}(i)}t^{i}$
$\displaystyle=-H_{P,v,\alpha}(t)$
$\displaystyle=(-1)^{M}\overset{\circ}{H}_{P,-v,-\alpha}(t^{-1})$
$\displaystyle=(-1)^{M}\sum_{i\in\mathbb{Z}}\overset{\circ}{h}_{P,-v,-\alpha}(i)t^{-i}.$

Here, the third equality follows from (1), and the fourth follows from (3). By
comparing the coefficients, we conclude that
$f_{P,v,\alpha}(-i)=(-1)^{M}\overset{\circ}{h}_{P,-v,-\alpha}(i)$ for
$i\in\mathbb{Z}_{>\alpha}$. ∎

## References