# bases equivalent to the unit vector basis of $c_{0}$ or $\ell_{p}$

Peter G. Casazza  Department of Mathematics, University of Missouri, Columbia,
MO 65211-4100<EMAIL_ADDRESS>
###### Abstract.

We will show that an unconditional basis in a Banach space is equivalent to
the unit vector basis of $c_{0}$ or $\ell_{p}$ for $1\leq p<\infty$ if and
only if all finitely supported blocks of the basis generated by a unit vector
and its dual basis are uniformly equivalent to the basis or all such blocks
are uniformly complemented.

The author was supported by NSF DMS 1906025

##  1\. Introduction

There are several classical results concerning when a basis for a Banach space
is equivalent to the unit vector basis of $c_{0}$ or $\ell_{p}$ for $1\leq
p<\infty$.

######  Definition 1.1.

Let $\\{x_{n}\\}_{n=1}^{\infty}$ be a basic sequence in a Banach space $X$. A
sequence of non-zero vectors $\\{u_{j}\\}_{j=1}^{\infty}$ in X of the form

$u_{j}=\sum_{n=p_{j}+1}^{p_{j+1}}b_{n}x_{n}$

with $\\{b_{n}\\}_{n=1}^{\infty}$ scalars and $p_{1}<p_{2}<\cdots$ an
increasing sequence of integers, is called a block basic sequence or briefly a
block basis of $\\{x_{n}\\}_{n=1}^{\infty}$.

######  Definition 1.2.

If $\\{x_{n}\\}_{n=1}^{\infty}$ and $\\{y_{n}\\}_{n=1}^{\infty}$ are bases for
Banach spaces $X,Y$ respectively, $\\{x_{n}\\}_{n=1}^{\infty}$ is K-equivalent
to $\\{y_{n}\\}_{n=1}^{\infty}$ if there is a constant $1\leq K<\infty$ so
that for all scalars $\\{a_{n}\\}_{n=1}^{\infty}$ we have

$\frac{1}{K}\|\sum_{n=1}^{\infty}a_{n}y_{n}\|\leq|\sum_{n=1}^{\infty}a_{n}x_{n}\|\leq
K\|\sum_{n=1}^{\infty}a_{n}y_{n}\|.$

Equivalently, if $T:X\rightarrow Y$ is given by $Tx_{n}=y_{n}$ then
$\|T\|,\|T^{-1}\|\leq K$. We say $\\{x_{n}\\}_{n=1}^{\infty}$ is equivalent to
$\\{y_{n}\\}_{n=1}^{\infty}$ if they are K-equivalent for some K.

In [2] we have:

######  Theorem 1.3.

let $X$ be a Banach space with a normalized basis
$\\{x_{n}\\}_{n=1}^{\infty}$. Then $\\{x_{n}\\}_{n=1}^{\infty}$ is equivalent
to the unit vector basis of $c_{0}$ or $\ell_{p}$, $1\leq p<\infty$ if and
only if $\\{x_{n}\\}_{n=1}^{\infty}$ is equivalent to all of its normalized
block bases.

######  Definition 1.4.

A basis $\\{x_{n}\\}_{n=1}^{\infty}$ of a Banach space X is called
subsymmetric if it is unconditional and, for every increasing sequence of
integers $\\{n_{i}\\}_{i=1}^{\infty}$, $\\{x_{n_{i}}\\}_{i=1}^{\infty}$ is
equivalent to $\\{x_{n}\\}_{n=1}^{\infty}$. By switching to an equivalent norm
we may assume the subsymmetric basis is 1-unconditional and 1-equivalent to
all of its subsequences.

######  Definition 1.5.

Let $\\{x_{n}\\}_{n=1}^{\infty}$ be a symmetric or subsymmetric basis for a
Banach space X. Let $0\not=\alpha=\sum_{j=1}^{\infty}b_{j}x_{j}\in X$. Given a
subset of $\mathbb{N}$ $\\{n_{1}<n_{2}<\ldots\\}$ partition this set into
infinitely many disjoint subsets $\\{(i,j)\\}_{i,j=1}^{\infty}$. Then the
block basis generated by $\alpha$ is $\\{U_{i}^{\alpha}\\}_{i=1}^{\infty}$
where

$U_{i}^{\alpha}=\sum_{j=1}^{\infty}b_{j}x_{ij}\mbox{ for all }i=1,2,\ldots.$

######  Remark 1.6.

We have to be careful with this. In the symmetric case, for different
$\\{n_{i}\\}_{i=1}^{\infty}$ and different partitions
$\\{(i,j)\\}_{i,j=1}^{\infty}$ the $\\{U_{i}^{\alpha}\\}_{i=1}^{\infty}$ are
symmetric and equivalent to each other. But for subsymmetric bases, neither of
these properties may hold.

Also in [2] we have

######  Theorem 1.7.

If $\\{x_{n}\\}_{n=1}^{\infty}$ is a symmetric basis for a Banach space X,
then $\\{x_{n}\\}_{n=1}^{\infty}$ and its dual basis
$\\{x_{n}^{*}\\}_{n=1}^{\infty}$ are equivalent to all their block bases
generated by a single vector if and only if $\\{x_{n}\\}_{n=1}^{\infty}$ is
equivalent to the unit vector basis of $c_{0}$ or $\ell_{p}$ for some $1\leq
p<\infty$.

Recently [1] the subsymmetric version of this was proved.

######  Theorem 1.8.

If $\\{x_{n}\\}_{n=1}^{\infty}$ is a subsymmetric basis for a Banach space X
then $\\{x_{n}\\}_{n=1}^{\infty}$ and $\\{x_{n}^{*}\\}_{n=1}^{\infty}$ are
both equivalent to all block bases generated by a single vector if and only if
$\\{x_{n}\\}_{n=1}^{\infty}$ is equivalent to the unit vector basis of either
$c_{0}$ or $\ell_{p}$ for some $1\leq p<\infty$.

######  Remark 1.9.

In the theorems above it is important that we work with possibly infinitely
supported vectors since the theorems fail if we can only work with finitely
supported vectors as we will see in the next section.

In this paper we look at finitely supported block bases generated by a single
vector and prove the corresponding results for this family but need the
additional requirement that the equivalences are uniform.

##  2\. Bases Equivalent to the unit vector basis of $c_{0}$ or $\ell_{p}$

######  Definition 2.1.

Let $\\{x_{n}\\}_{n=1}^{\infty}$ and $0\not=\alpha=\sum_{n=1}^{m}b_{n}x_{n}$.
The (finitely supported) block basis generated by $\alpha$ is the block basis
$\\{U_{i}^{\alpha}\\}_{i=1}^{\infty}$ where

$U_{i}^{\alpha}=\sum_{n=(i-1)m+1}^{im}b_{n-km}x_{n},\mbox{ for all
}i=1,2,\ldots.$

For subsymmetric bases, blocks generated by a single vector are always
equivalent to the basis, but as we will see, they may not be uniformly
equivalent.

######  Theorem 2.2.

Let $\\{x_{n}\\}_{n=1}^{\infty}$ be a subsymmetric basis for a Banach space X.
Assume $0\not=\alpha=\sum_{n=1}^{m}b_{n}x_{n}\in X$ and
$\\{U_{i}^{\alpha}\\}_{i=1}^{\infty}$ is the block basis generated by
$\alpha$. Then $\\{U_{i}^{\alpha}\\}_{i=1}^{\infty}$ is equivalent to
$\\{x_{n}\\}_{n=1}^{\infty}$.

###### Proof.

We may assume that the basis is 1-unconditional and 1-equivalent to all of its
subsequences. Assume $b_{j}\not=0$ for some fixed $1\leq j\leq m$. We compute
for any $\sum_{n=1}^{\infty}a_{n}x_{n}$:

$\displaystyle\|\sum_{n=1}^{\infty}a_{n}x_{n}\|$
$\displaystyle=\|\sum_{i=1}^{\infty}a_{i}x_{(i-1)m+j}\|$
$\displaystyle=\frac{1}{b_{j}}\|\sum_{i=1}^{\infty}a_{i}b_{j}x_{(i-1)m+j}\|$
$\displaystyle\leq\frac{1}{b_{j}}\|\sum_{i=1}^{\infty}a_{i}U_{i}^{\alpha}\|$
$\displaystyle\leq\frac{1}{b_{j}}\sum_{k=1}^{m}b_{k}\|\sum_{i=1}^{\infty}a_{i}x_{(i-1)m+k}\|$
$\displaystyle=\frac{1}{b_{j}}\left(\sum_{k=1}^{m}b_{k}\right)\|\sum_{i=1}^{\infty}a_{i}x_{i}\|.$

This completes the stated equivalence. ∎

Now we can prove the main result of the paper.

######  Theorem 2.3.

Let $\\{x_{n}\\}_{n=1}^{\infty}$ be a unconditional basis for a Banach space
X. The following are equivalent:

  1. (1)

$\\{x_{n}\\}_{n=1}^{\infty}$ is equivalent to the unit vector basis for
$c_{0}$ or $\ell_{p}$ for $1\leq p<\infty$.

  2. (2)

There is a constant $1\leq K$ so that whenever
$\alpha=\sum_{n=1}^{m}b_{n}x_{n}$ is a unit vector, the block basis generated
by $\alpha$ is K-equivalent to $\\{x_{n}\\}_{n=1}^{\infty}$, the the same
result holds for $\\{x_{n}^{*}\\}_{n=1}^{\infty}$.

  3. (3)

There is a constant $1\leq K$ so that whenever
$\alpha=\sum_{n=1}^{m}b_{n}x_{n}$ is a unit vector then the block basis
$\\{U_{i}^{\alpha}\\}_{i=1}^{\infty}$ generated by $\alpha$ is complemented in
the space by a projection $P$ with $\|P\|\leq K$, and the same is true for
$\\{x_{n}^{*}\\}_{n=1}^{\infty}$.

###### Proof.

$(1)\Rightarrow(2),\ (3)$: This is obvious.

$(3)\Rightarrow(2)$: We may assume the basis is 1-unconditional. Choose norm
one vectors $v_{1}=\sum_{i=1}^{m}b_{i}x_{i}$ and
$w_{1}=\sum_{i=m+1}^{n}c_{i}x_{i}$ and let $u_{1}=v_{1}+w_{1}$. Define
$u_{i}=v_{i}+w_{i}$ for $i=1,2,\ldots$ by

$v_{i}=\sum_{j=(i-1)n+1}^{(i-1)n+m}b_{j-(i-1)n}x_{i}\mbox{ and
}w_{i}=\sum_{j=(i-1)n+m+1}^{in}c_{j-(i-1)n}x_{i}.$

By assumption, there is a projection $Q$ onto the block basis
$\\{u_{i}\\}_{i=1}^{\infty}$ generated by the vector $u_{1}$ with $\|Q\|\leq
K$. For each $i=1,2,\ldots$ assume

$Qv_{i}=\sum_{j=1}^{\infty}b_{ij}u_{j}\mbox{ and
}Qw_{i}=\sum_{j=1}^{\infty}c_{ij}u_{j}.$

Then $b_{ii}+c_{ii}=1$ for all $i=1,2,\ldots$. Define an operator
$T:[Qv_{i}]\rightarrow[w_{i}]$ by $T(Qv_{i})=w_{i}$ for all $i=1,2,\ldots$. So
$\|TQ\|\leq K$ and the matrix of $T$ is $[b_{ij}+c_{ij}]$. And since the basis
is 1-unconditional, the norm of the diagonal operator $D$ of $TQ$ is
$\|D\|\leq\|TQ\|$. But the diangonal operator maps $v_{i}$ to $w_{i}$. So for
all scalars $\\{a_{i}\\}_{i=1}^{\infty}$ we have

$\|\sum_{i=1}^{\infty}a_{i}w_{i}\|=\|\sum_{i=1}^{\infty}a_{i}Tv_{i}\|\leq
K\|\sum_{i=1}^{\infty}a_{i}v_{i}\|.$

Switching the roles of $v_{i}$ and $w_{i}$ we have for all scalars
$\\{a_{i}\\}_{i=1}^{\infty}$

$\|\sum_{i=1}^{\infty}a_{i}v_{i}\|\leq K\|\sum_{i=1}^{\infty}a_{i}w_{i}\|$

So $\\{v_{i}\\}_{i=1}^{\infty}$ is equivalent to $\\{w_{i}\\}_{i=1}^{\infty}$.
Letting $v_{1}=x_{1}$, we discover that $\\{w_{i}\\}_{i=1}^{\infty}$ is
equivalent to $\\{x_{i}\\}_{i=1}^{\infty}$.

The same proof works for $\\{x_{n}^{*}\\}_{n=1}^{\infty}$.

$(2)\Rightarrow(1)$: We will do this in steps. For any $n\in\mathbb{N}$ define

$\lambda(n)=\|\sum_{i=1}^{n}x_{i}\|\mbox{ and
}\mu(n)=\|\sum_{i=1}^{n}x_{i}^{*}\|.$

Step 1: We show that either $X$ is isomorphic to $c_{0}$ or $\ell_{1}$ or
there is a $1<p$ so that for all $n\in\mathbb{N}$

$\frac{1}{K}\leq\frac{\lambda(n)}{n^{1/p}}\leq K.$

For this, fix $n\in\mathbb{N}$ and let
$\alpha_{n}=\frac{1}{\lambda(n)}\sum_{i=1}^{n}x_{i}$. Consider the block basis
generated by $\alpha_{n}$ as

$U_{i}^{\alpha_{n}}=\frac{1}{\lambda(n)}\sum_{j=(i-1)n+1}^{in}x_{i},\mbox{ for
all }i=1,2,\ldots.$

By (2), for every $m=1,2,\ldots$ we have

$\frac{1}{K}\lambda(m)\leq\|\sum_{i=1}^{m}U_{i}^{\alpha_{n}}\|=\frac{1}{\lambda_{n}}\lambda(mn)\leq
K\lambda(m).$

Hence,

$\frac{1}{K}\leq\frac{\lambda(mn)}{\lambda(n)\lambda(m)}\leq K.$

By the proof of [2] Theorem (2,a,9) either $sup_{n}\lambda(n)<\infty$, or we
have for some $1\leq p<\infty$,

(1) $\frac{1}{K}\leq\frac{\lambda(n)}{n^{1/p}}\leq K,\mbox{ for all
}n=1,2,\ldots.$

If $sup_{n}\lambda_{n}<\infty$ then X is isomorphic to $c_{0}$ and if $p=1$
then X is isomorphic to $\ell_{1}$ by [2] the remark following (3,a,7). So we
assume $1<p<\infty$ and choose $1<q$ so that $\frac{1}{p}+\frac{1}{q}=1$.

Step 2: We have for all $n\in\mathbb{N}$

$\frac{1}{2K}\leq\frac{\mu(n)}{n^{1/q}}\leq 2K.$

For this, we apply [2] Proposition (3,a,6), page 116:

$n\leq\lambda(n)\mu(n)\leq 2n.$

Hence

$1\leq\frac{\lambda(n)\mu(n)}{n}=\frac{\lambda(n)}{n^{1/p}}\frac{\mu(n)}{n^{1/q}}\leq
2.$

The result is now immediate from Equation 1.

Step 3: For all scalars $\\{a_{i}\\}_{i=1}^{\infty}$ we have

$\left\|\sum_{i=1}^{\infty}a_{i}x_{i}\right\|\leq
2K\left(\sum_{i=1}^{\infty}|a_{i}|^{p}\right)^{1/p}.$

The same is true for all $\\{a_{i}\\}_{i=1}^{\infty}$

$\|\sum_{i=1}^{\infty}a_{i}x_{i}^{*}\|\leq
2K\left(\sum_{i=1}^{\infty}|a_{i}|^{q}\right)^{1/q}.$

For this, we can copy line by line [2] Theorem (3,a,10), page121.

Step 5:We conclude the proof.

By steps (2), (3), and (4) we have for all scalars
$\\{a_{i}\\}_{i=1}^{\infty}$

$\left\|\sum_{i=1}^{\infty}a_{i}x_{i}^{*}\right\|\leq
2K\left(\sum_{i=1}^{\infty}|a_{i}|^{q}\right)^{1/q}.$

By duality, for all $\\{a_{i}\\}_{i=1}^{\infty}$ we have

$\left\|\sum_{i=1}^{\infty}a_{i}x_{i}\right\|\geq\frac{1}{2K}\left(\sum_{i=1}^{\infty}|a_{i}|^{p}\right)^{1/p}.$

This concludes the proof of the theorem. ∎

######  Remark 2.4.

We would hope to not have to assume the basis is unconditional since (3)
implies that every subsequence of the basis is uniformly complemented.
Unfortunately, this property does not imply unconditionality of the basis as
we see below.

Example: Lwt $\\{x_{i}\\}_{i=1}^{\infty}$ be the row vectors of the matrix:

$\begin{bmatrix}1&0&0&0&0&\cdots\\\ 1&1&0&0&0&\cdots\\\ 1&1&1&0&0&\cdots\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\end{bmatrix}$

The set $\\{x_{i}\\}_{i=1}^{\infty}$ is called the summing basis of $c_{0}$.
We will now show that every subsequence of this basis is 1-complemented. Fix
natural numbers $n_{0}=0<n_{1}<n_{2}<\cdots$. For
$x=\sum_{i=1}^{\infty}a_{i}x_{i}$ in terms of the unit vector basis this
vector is:

$x=\left(\sum_{i=1}^{\infty}a_{i},\sum_{i=2}^{\infty}a_{i},\sum_{i=3}^{\infty}a_{i},\ldots\right),$

and so the norm of this vector is:

$\|x\|=max_{i=1}^{\infty}|\sum_{j=i}^{\infty}a_{j}|.$

Define a projection $P$ by:

$P(x)=\sum_{i=0}^{\infty}\left(\sum_{j=n_{i}+1}^{n_{i+1}}a_{j}\right)x_{n_{i}}.$

It is clear that this is a projection and in terms of the unit vector basis
this is

$Px=\left(\sum_{i=1}^{\infty}a_{i},\ldots,\sum_{i=1}^{\infty}a_{i}(n_{1}),\sum_{i=2}^{\infty}a_{i}(n_{1}+1),\ldots,\sum_{i=2}^{\infty}a_{i}(n_{2}),\sum_{i=3}^{\infty}a_{i}(n_{2}+1),\ldots,\sum_{i=3}^{\infty}a_{i}(n_{3}),\ldots\right)$

Hence,

$\|Px\|=max_{i=1}^{\infty}|\sum_{j=i}^{\infty}a_{j}|=\|x\|.$

An obvious question is:

######  Problem 2.5.

Do we need the conditions in Theorem to include both
$\\{x_{n}\\}_{n=1}^{\infty}$ and $\\{x_{n}^{*}\\}_{n=1}^{\infty}$ or is just
one of these enough?

## References

  * [1] P.G. Casazza, S. J. Dilworth, D. Kutzarova, P. Motakis, On uniquencess and plentitude of subsymmetric sequences, preprint.
  * [2] J. Lindenstrauss and L. Tzafriri, Classical Banach Spaces 1: Sequence Spaces, Springer-Verlag, New York (1977).


###### Proof.

###### Proof.

.:

###### Federated Machine Reasoning for Resource Provisioning in 6G O-RAN

###### Abstract

O-RAN specifications reshape RANs with function disaggregation and open
interfaces, driven by RAN Intelligent Controllers. This enables data-driven
management through AI/ML but poses trust challenges due to human operators’
limited understanding of AI/ML decision-making. Balancing resource
provisioning and avoiding overprovisioning and underprovisioning is critical,
especially among the multiple virtualized base station(vBS) instances. Thus,
we propose a novel Federated Machine Reasoning (FLMR) framework, a
neurosymbolic method for federated reasoning, learning, and querying. FLMR
optimizes CPU demand prediction based on contextual information and vBS
configuration using local monitoring data from virtual base stations (vBS) on
a shared O-Cloud platform.This optimization is critical, as insufficient
computing resources can result in synchronization loss and significantly
reduce network throughput. In the telecom domain, particularly in the virtual
Radio Access Network (vRAN) sector, predicting and managing the CPU load of
vBSs poses a significant challenge for network operators. Our proposed FLMR
framework ensures transparency and human understanding in AI/ML decisions and
addresses the evolving demands of the 6G O-RAN landscape, where reliability
and performance are paramount. Furthermore, we performed a comparative
analysis using DeepCog as the baseline method. The outcomes highlight how our
proposed approach outperforms the baseline and strikes a better balance
between resource overprovisioning and underprovisioning. Our method notably
lowers both provisioning relative to the baseline by a factor of 6.

###### Index Terms:

FL, LTN, MR, Neuro-Symbolic, O-RAN, SLA

##  I Introduction

The transition from 5G to 6G networks emphasizes better connectivity and
diverse service support. The concept of open radio access networks (RAN) is
pivotal in this evolution, with the O-RAN Alliance leading the standardization
of Open RAN architecture for interoperability among multi-vendor elements [1].
In O-RAN, AI/ML optimizes CPU resources for efficient network performance
across diverse regions with multiple base stations, but data security
complexities challenge accurate predictive model development.Transparency and
interpretability in predictive models are critical for making informed
resource allocation decisions. This preserves the delicate balance between
overprovisioning and underprovisioning, ensuring optimal resource utilization,
flexibility to adapt to network changes, and cost optimization. [2] proposes
an online learning algorithm for resource allocation balancing throughput and
energy consumption, while [3] introduces DeepCog for cognitive resource
management in 5G, addressing the balance between resource overprovisioning and
service request violations. In [4], the authors evaluate resource allocation
in vRAN networks within the O-RAN paradigm, showing that misallocation can
harm performance. Given O-RAN’s distributed nature, federated reinforcement
learning [5] have been proposed for enhancing predictive models while
preserving privacy.These studies lack clarity in explaining the
interpretability of their predictive models, a common issue with AI/ML
adoption in the telco sector. Neural networks’ limitations in interpretability
and systematic generalization emphasize the significance of machine reasoning
(MR), a Neural-symbolic AI (NSAI) approach in AI [6]. In addressing ML
limitations, the authors [7] propose MR as an advanced solution for the
growing complexity in telecom networks. Thus, we propose integrating MR with
FL to tackle O-RAN challenges. FL enables collaborative learning across base
stations, improving AI forecasting with distributed data. This fusion of
interpretability from NSAI and privacy-preserving learning of FL builds trust
among O-RAN operators and stakeholders. In this paper, we present the
following contributions:

  * •

Addressing the challenge within the O-RAN framework by focusing on efficient
computing usage of vBS (gNB) instances on shared computing platforms in the
O-Cloud.

  * •

Developing accurate CPU forecasting models in O-RAN architecture entails
addressing data privacy and security and ensuring model transparency for
operators to understand high CPU utilization factors. Notably, this system-
agnostic approach ensures flexibility across O-RAN and non O-RAN environments,
enhancing the model’s utility for accurate forecasting amidst evolving telecom
demands.

  * •

Integrating Logical tensor networks (LTN) [8], a NSAI approach, with FL for
real-time resource allocation in the Near Real-Time RIC of O-RAN. LTN
introduces logical constraints to provide transparency into decision-making
tasks for resource allocation with specific conditions.

  * •

Contributing significantly to balance the trade-off between resource
overprovisioning and underprovisioning to enhance the efficiency of the
network.

Figure 1: Reasoning for learning [9]

##  II The LTN Paradigm in Machine Reasoning

Neural-symbolic learning systems combine the strengths of symbolic and neural
approaches. Symbolic systems excel with structured data, acquire solution
spaces through training and yield higher-level reasoning outputs. Conversely,
neural systems show proficiency in learning from unstructured data, producing
in lower-level learning outputs during training. So, by merging both
capabilities, it offers a unified technique for problem-solving [9]. In the
reasoning for learning category, shown in Fig. 1, this approach incorporates
symbolic knowledge into training process and leverage the capabilities of
neural systems for machine learning tasks. This enhances interpretability and
overall performance in challenging tasks by guiding or constraining the
learning process with symbolic knowledge, typically encoded for neural
networks. Based on this concept, the paper of [8], introduced LTN which has a
notable contribution in this domain. It is neuro-symbolic approach, a MR
framework that seamlessly integrates neural and symbolic approaches to
learning, reasoning, and querying concrete data and abstract knowledge. Using
neural computational graphs and first-order fuzzy logic semantics, it presents
tensor-based formalism and offers Real Logic, a fully differentiable logical
language [8]. It introduces logical constructs like predicates and axioms to
reasoning and interpretability in machine learning systems. Axioms express
general facts by logical assertions, whereas predicates reflect relationships
or properties, allowing LTN to hold complex relationships, facts, and rules
about entities in a domain. It enables the creation of AI models that
anticipate and explain why they predict [8].In telecommunications,
transparency is crucial for understanding AI systems’ decisions, particularly
in architectures like O-RAN. Integrating LTN into the proposed federated deep
learning architecture enhances interpretability and aligns with human
cognitive processes, vital for dynamic telecom environments.

Figure 2: Federated model in O-RAN architecture [10]

##  III Proposed Architecture and Datasets

###  III-A Architecture

As illustrated in Fig. 2, our O-RAN based system model, drawing inspiration
from [11], involves multiple vBS instances operating on a shared computing
platform and establishing connections with User Equipments (UEs). The O-RAN
system includes two RAN Intelligent Controllers (RICs): a near real-time RIC
with vBSs and a non-real-time RIC by SMO. They support programmable functions,
leveraging AI, and control activities through xApps and rApps. SMO manages
O-Cloud infrastructure via O1/O2 interfaces for vBS setup. Monitoring metrics
follow O-RAN standards, with a near-RT RIC retrieving data via E2 and passing
it to the non-RT RIC via A1. Our design expands the traditional O-RAN concept
by integrating proactive network resource management into each vBS. Utilizing
FL via the Near Real-Time RIC improves mobile communication services’
dependability. Independent local FL models in each vBS enhance adaptive
resource management, while a Global FL model in the Near Real-Time RIC’s xApps
ensures system stability and flexibility. Additionally, the system introduces
distributed applications, dApps [10], which complement xApps/rApps by
providing intelligence at Central Units (CUs)/Distributed Units (DUs). These
dApps support real-time inference at tighter timescales compared to the RICs.
In our system model, dApps are situated within the vBS instance. A distinct
component, the dApp Controller & Monitoring, is hosted in the near-RT RIC,
controls and monitors dApps executing at the vBSs/gNBs through xApps [10].
Noted that, the concept of dApps within the O-RAN framework has been proposed
in this work [10], although yet to be standardized by the O-RAN alliance [11].
However, we have incorporated this entity into our framework due to the unique
capabilities of dApps in addressing the specific challenges outlined in our
problem statement.

###  III-B Datasets

The dataset, as summarized in Table I, corresponds to research activities
documented in [12], focusing on the instantiation of varying vBS numbers on a
shared computing platform. These vBSs are instantiated in specific CPU core
sets with distinct time-sharing allocations. Each vBS is associated with
context, encompassing traffic demands and statistics on the used Modulation
and Coding Scheme (MCS) for both Uplink (UL) and Downlink (DL). Notably,
network parameters like the MCS index significantly impact CPU load,
particularly due to coding/decoding workloads. Moreover, this is a time-series
dataset where each row corresponds to a 20-second experiment with specific
contextual conditions.It includes per-core CPU utilization, calculated as the
average of samples collected every 200 milliseconds, indicating a temporal
dimension for analyzing CPU utilization patterns over time.

Table I: Dataset Features and Output Feature | Description
---|---
mcs$\\_$dl$\\_$i | Downlink MCS index of vBS i
mcs$\\_$ul$\\_$i | Uplink MCS index of vBS i
dl$\\_$kbps$\\_$i | Downlink traffic demand in kbps of vBS i
ul$\\_$kbps$\\_$i | Uplink traffic demand in kbps of vBS i
cpu$\\_$set | Computing set used by vBS i
|
Output | Description
cpu$\\_$i | Avg. measured CPU utilization between 0 and 1.
explode | Whether the experiment has run correctly or not

##  IV Federated machine reasoning (FLMR) model for transparent CPU usage
prediction

Here, we describe the different stages of the proposed FLMR model as
summarized in Fig. 3.

###  IV-A Closed-Loop Description

We propose a federated deep learning architecture where the local learning is
performed iteratively with run-time rule-based neuro symbolic reasoning scheme
in a closed loop way as shown in Fig. 3. Our model integrates LTN [8], capable
of both neural network-style learning and classical AI-style reasoning, into a
deep neural network FL model. For each local epoch, the Learner module feeds
the posterior symbolic model graph to the Tester block which yields the test
features and the corresponding predictions $\hat{y}_{k}^{(i)}$ to the
_Knowledge base Mapper_ at step 3. In the proposed ORAN based system model,
this function should be deployed in the xApps. xApps are responsible for
implementing specific functions or services and are part of the broader move
towards a more open, interpretable and flexible RAN environment. During
training, LTNs calculate the satisfaction level of logical constraints from
the Knowledge base information at step 4. The satisfaction level, defined as
$\phi$, is converted into a loss during training using the formulation
"$loss=1-\phi$." In the training process, the optimization algorithm, such as
the Adadelta optimizer, works to minimize this loss at step 5. It adjusts the
model’s parameters, including weights and biases, to simultaneously maximize
satisfaction of logical constraints and minimize the loss. The objective is to
find optimal parameter values that strike a balance between accurate
predictions and adherence to the logical rules encoded in the "eq" predicate
which we explain in the next subsection.

Indeed, for each local CL $(k)$, the predicted ML model
$\hat{y}_{k}^{(i)},\,(i=1,\ldots,D_{k})$, should minimize the main loss
function with respect to the ground truth $y_{k}^{(i)}$, while jointly
respecting some long-term logical constraints defined over its $D_{k}$
samples.

As shown in steps 1 and 6 of Fig. 3, the optimized local weights at round $t$,
$\mathbf{W}_{k}^{(t)}$, are sent to the server which generates a global FL
model as,

$\mathbf{W}^{(t+1)}=\sum_{k=1}^{K}\frac{D_{k}}{D}\mathbf{W}_{k}^{(t)},$ (1)

where $D=\sum_{k=1}^{K}D_{k}$ is the total data samples of all datasets. The
server then broadcasts the global model to all the $K$ CLs that use it to
start the next round of iterative local optimization.

Figure 3: Neuro symbolic FLMR block

###  IV-B Knowlegde base Mapper

However, in LTN, the knowledge base is a central component that stores and
represents information in a formalized manner. The knowledge base in LTN
typically consists of predicates, axioms, and other logical constructs that
capture relationships, facts, and rules about the entities in a given domain.
It forms the basis for logical reasoning and decision-making within the
framework, allowing for the representation and manipulation of complex
knowledge structures.

####  IV-B1 Predicates

_Predicates_ in the knowledge base represent relationships or properties
between entities. We define our predicates by inspiring the paper of [8] which
introduce a "smooth" version of the equality symbol "$=$" using a predicate
called "eq." The "eq" predicate is defined as:

$\textbf{{eq}}(f(x_{i}),y_{i})=\frac{1}{1+\alpha\sqrt{\sum_{j=1}^{N}(f(x_{i})_{j}-(y_{i})_{j})^{2}}}$
(2)

Here, $\textbf{{eq}}(f(x_{i}),y_{i})$ represents the output of the "eq"
predicate for predicted value $f(x_{i})$ and observed value $y_{i}$. $\alpha$
is a positive constant that controls the smoothness of the matching. This
formulation ensures output within the [0, 1] range, with higher values
indicating a closer matches. In our O-Cloud scenario, the "eq" predicate
provides crucial domain-specific knowledge by allowing for flexible matching
between predicted ($f(x_{i})$) and observed ($y_{i}$) CPU usage in a
continuous and smooth manner. This flexibility is essential for optimizing
vRAN network performance in O-RAN deployment, as traditional strict equality
constraints can be overly rigid, making optimization challenging.

####  IV-B2 Axioms

_Axioms_ are logical statements or rules that express general truths within
the knowledge base, guiding the process of making inferences based on the
predicates’ relationships. The logical relationships specified by predicates
within axioms enable the system to derive new information. During training,
LTNs calculate the satisfaction level of logical constraints from the
Knowledge base information, shown at step 4. This satisfaction level
quantifies how well the model adheres to the specified logical rules, such as
the "eq" predicate, during predictions.

A high satisfaction level (close to 1) corresponds to a low loss (close to 0),
adherence to logical constraints. Conversely, a low satisfaction level (close
to 0) results in a high loss (close to 1).

######  Definition 1 (Satisfaction level).

A continuous parameter called the satisfaction level ($\phi$) of logical
constraints in LTNs indicates how closely the model complies with predefined
logical rules, like the "eq" predicate, throughout training. Based on the data
from the Knowledge base at a certain phase, this measure is calculated.
Typically, $\phi$ ranges from 0 to 1, with higher values denoting a more
crucial adherence to the logical requirements.

###  IV-C Integration of LTN with FL for CPU Usage prediction

In order to solve a local regression problem for predicting CPU usage in
iterations specified by the FL rounds $t\,(t=0,\ldots,T-1)$ i.e.,

$\min_{\mathbf{W}_{k}^{(t)}}\,\frac{1}{D_{k}}\sum_{i=1}^{D_{k}}\ell\left(y_{k}^{(i)},\hat{y}_{k}^{(i)}\left(\mathbf{W}_{k}^{(t)},\mathbf{x}_{k}\right)\right),$
(3)

Indeed, for each local CL $(k)$, the predicted ML model
$\hat{y}_{k}^{(i)},\,(i=1,\ldots,D_{k})$, should minimize the main loss
function with respect to the ground truth $y_{k}^{(i)}$, while jointly
respecting some long-term logical constraints defined over its $D_{k}$
samples.

The loss is computed as $1-\phi$, where $\phi$ is the satisfaction level of
the axioms. Therefore, the loss function,
$\ell({y}_{k}^{(i)},\hat{y}_{k}^{(i)})$ can be written as:

$\ell({y}_{k}^{(i)},\hat{y}_{k}^{(i)})=1-\phi({y}_{k}^{(i)},\hat{y}_{k}^{(i)})$
(4)

The goal of training is to minimize this loss, which essentially maximizes the
satisfaction level of the logical axioms. The axioms are given by:

$\textbf{A}({y}_{k}^{(i)},\hat{y}_{k}^{(i)})=\textbf{{Forall}}(\textbf{{ltn.diag}}({y}_{k}^{(i)},\hat{y}_{k}^{(i)}),\textbf{{eq}}({y}_{k}^{(i)},\hat{y}_{k}^{(i)}))$
(5)

This equation represents a logical statement that involves quantifiers
(universal quantification with Forall), a diagonal operator (diag), an
equality predicate (eq), and the regressor function $f(x)$. The overall
expression asserts that for all pairs of variables ${y}_{k}^{(i)}$ and
$\hat{y}_{k}^{(i)}$, the equality predicate should hold between $f(x)$ and
${y}_{k}^{(i)}$. The Forall quantifier signifies that this condition should be
true for all possible combinations of ${y}_{k}^{(i)}$ and $\hat{y}_{k}^{(i)}$.
Additionally, a regression task requires a notion of equality. We, therefore,
define the predicate eq as a smooth version of the symbol "$=$" to turn the
constraint $\hat{y}_{k}^{(i)}={y}_{k}^{(i)}$ into a smooth optimization
problem. The overall steps are mentioned in the Algorithm 1.

Input: $K$, $\eta_{\lambda}$, $T$, $L$, $\alpha$. # See Table II

Server initializes $\mathbf{W}^{(0)}$ and broadcasts it to the $\mathrm{CL}$s

# Federated Learning

for  _$t=0,\ldots,T-1$_ do

parallel for $k=1,\ldots,K$ do

# Model graph from the local Model

Receive the graph $\mathcal{M}_{k}$ from the local model.

# Test local model and send results to Knowledge Base Mapper.

# Use "eq" predicate to measure the satisfaction level

$\textbf{{eq}}(f(x_{i}),y_{i})=\frac{1}{1+\alpha\sqrt{\sum_{j=1}^{N}(f(x_{i})_{j}-(y_{i})_{j})^{2}}}$

# Calculate the loss function

$\ell({y}_{k}^{(i)},\hat{y}_{k}^{(i)})=1-\phi({y}_{k}^{(i)}\hat{y}_{k}^{(i)})$

# Adadelta optimizes to minimize loss, maximizing satisfaction of logical
axioms.

$\textbf{A}({y}_{k}^{(i)},\hat{y}_{k}^{(i)})=\textbf{{Forall}}(\textbf{{ltn.diag}}({y}_{k}^{(i)},\hat{y}_{k}^{(i)}),\textbf{{eq}}({y}_{k}^{(i)},\hat{y}_{k}^{(i)}))$

Each local $\mathrm{vBS}$ $k$ sends $\mathbf{W}_{k}^{(t)}$ to the aggregation
server.

end parallel for

# FL Server Aggregation

return
_$\mathbf{W}^{(t+1)}=\sum_{k\in\\{k_{1},\ldots,k_{m}\\}}\frac{D_{k}}{D}\mathbf{W}_{k}^{(t)}$_

Broadcasts $\mathbf{W}^{(t+1)}$ to all $K$ $\mathrm{vBS}$s.

end for

Algorithm 1 Federated Machine Reasoning & Learning Table II: Settings Parameter | Description | Value
---|---|---
$MLP$ | Multilayer Perception (ANN) | 2 hidden layers
$K$ | # vBSs | $50$
$D_{k,n}$ | Local dataset batch size |  $500$ samples
$T$ | # Max FL rounds | $50$
$\eta_{\lambda}$ | Learning rate (AdaDelta) | $0.85$
$\alpha$ | Positive constant ("eq" predicate) | $0.5$

##  V Results

This section analyses the proposed closed loop FLMR framework in detail. To
build the logical reasoning based predictive model, we include logical
reasoning and learning into local FL model training.

###  V-A Parameter Settings and Baseline

  * •

Settings: For simulations, we utilized Python in an Ubuntu 20.04 environment
on a laptop. The datasets, sourced from vBSs, are detailed in Table I. The
hyperparameters of our FLMR model are listed in Table II, with AdaDelta as the
optimizer.

  * •

Baseline: As a baseline, we employ the _DeepCog_ framework [3], which
introduces a novel data analytics tool for the cognitive management of
resources in 5G systems. It introduces a customized loss function designed for
capacity forecasting, enabling operators to balance overprovisioning and
demand violations effectively. This function considers the costs associated
with underprovisioning and overprovisioning, ensuring optimal resource
allocation.

In the following section, we conduct a comparative analysis between our
proposed FLMR model and the established deepCog model, evaluating aspects such
as convergence and satisfiability. Additionally, we explore the trade-off
between over and under resource provisioning in our analysis.

Figure 4: Comparative performance analysis between the proposed FLMR and
baseline DeepCog FL Figure 5: Comparative Analysis of CPU Usage Load
Prediction: Original vs. Predicted CPU Load

(a)  FLMR

(b)  DeepCog FL

Figure 6: Trade-off between the resource overprovisioning (Red level) and
underprovisioning (Blue level)

###  V-B Results Analysis

####  V-B1 Performance Analysis:

Fig. 4a shows that FLMR achieves faster convergence than the DeepCog FL
baseline model.

The satisfaction levels over FL rounds of both models are shown in Fig. 4b.
The satisfaction level serves as a metric to gauge the model’s effectiveness
and user contentment during the FL training process. Remarkably, the FLMR
model attains a highly desirable satisfaction level, hovering around 98$\%$.
This outcome indicates better performance and user acceptance. By comparison,
the baseline DeepCog FL model exhibits a comparatively lower satisfaction
level. FLMR’s superiority can be attributed to its integration of logical
reasoning through LTNs, which effectively handle logical constraints while
optimizing the trade off between resource allocation and SLA violations. In
contrast, DeepCog FL primarily focuses on capacity forecasting and may not
explicitly consider logical constraints, potentially leading to less accurate
predictions.

####  V-B2 Validation of Predictive Models: CPU Usage Evaluation

Fig. 5 compares the proposed model’s accuracy in predicting CPU load usage to
the baseline model. The proposed model, as shown in Fig. 5a, closely aligns
with actual CPU load, showcasing its effectiveness. In contrast, the baseline
model, as exhibits in Fig. 5b, struggles to make accurate predictions,
highlighting its limitations. The superior performance of the FLMR model is
attributed to its unique design choices, including FL for collaborative
learning, NSAI for interpretability, and adaptive resource management for
efficient responses to CPU demand fluctuations.

####  V-B3 Overprovisioning and Underprovisioning trade-off analysis

In Fig. 6 analyzes the trade-off between resource overprovisioning and
underprovisioning in the proposed FLMR model compared to the baseline. The
figure displays prediction errors for 100 samples at the start and end of the
FL round. The prediction error is the difference between the predicted output
$\hat{y}_{k}^{(i)}$ and measured ${y}_{k}^{(i)}$. We can define it by,

$P_{err}(x,y)=\hat{y}_{k}^{(i)}-{y}_{k}^{(i)}$ (6)

The red part indicates overprovisioning, where predicted resources exceed
actual needs, while the blue part represents underprovisioning, risking SLA
violations due to fewer allocated resources.

The presented model in Fig. 6a effectively balances the trade-off, minimizing
both over-provisioning and under-provisioning during FL convergence. This
surpasses the DeepCog baseline model shown in Fig. 6b by a factor of 6.

##  VI Conclusion

In conclusion, we present a unique neurosymbolic-based FLMR framework that
overcomes the difficulty of comprehending AI/ML decision processes in vBS
instances while maintaining a careful balance between overprovisioning and
underprovisioning resources. By optimizing CPU demand for virtual base
stations in the dynamic 6G O-RAN landscape, FLMR stands out as a transparent
and optimized solution. The comparative analysis underscores FLMR’s
superiority in achieving a more effective equilibrium between resource
overprovisioning and underprovisioning. FLMR presents itself as a state-of-
the-art option for transparent and effective AI/ML decision-making in the
dynamic O-RAN ecosystem as the telecom sector develops.

##  VII Acknowledgment

## References

  * [1] M. Polese, L. Bonati, S. D’oro, S. Basagni, and T. Melodia, “Understanding o-ran: Architecture, interfaces, algorithms, security, and research challenges,” _IEEE Communications Surveys & Tutorials_, vol. 25, pp. 1376–1411, 2022.
  * [2] M. Kalntis, G. Iosifidis, and F. A. Kuipers, “Adaptive resource allocation for virtualized base stations in o-ran with online learning,” 2023.
  * [3] D. Bega, M. Gramaglia, M. Fiore, A. Banchs, and X. Costa-Perez, “Deepcog: Cognitive network management in sliced 5g networks with deep learning,” in _IEEE Conference on Computer Communications_ , 2019, pp. 280–288.
  * [4] M. Hervás-Gutiérrez, E. Baena, C. Baena, J. Villegas, R. Barco, and S. Fortes, “Impact of cpu resource allocation on vran performance in o-cloud,” 2023.
  * [5] H. Zhang, H. Zhou, and M. Erol-Kantarci, “Federated deep reinforcement learning for resource allocation in o-ran slicing,” _IEEE Global Communications Conference_ , pp. 958–963, 2022.
  * [6] N. Duan, D. Tang, and M. Zhou, “Machine reasoning: Technology, dilemma and future,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts_ , 2020, pp. 1–6.
  * [7] M. Orlić, M. Daoutis, and S. Singh, “An introduction to machine reasoning in networks,” _Ericsson Blog_ , 2019.
  * [8] S. Badreddine, A. d’Avila Garcez, L. Serafini, and M. Spranger, “Logic tensor networks,” _Artificial Intelligence_ , vol. 303, p. 103649, Feb. 2022.
  * [9] D. Yu, B. Yang, D. Liu, H. Wang, and S. Pan, “A survey on neural-symbolic learning systems,” 2023.
  * [10] S. D’Oro, M. Polese, L. Bonati, H. Cheng, and T. Melodia, “dapps: Distributed applications for real-time inference and control in o-ran,” _IEEE Communications Magazine_ , vol. 60, no. 11, pp. 52–58, 2022.
  * [11] J. X. Salvat, J. A. Ayala-Romero, L. Zanzi, A. Garcia-Saavedra, and X. Costa-Perez, “Open radio access networks (o-ran) experimentation platform: Design and datasets,” _IEEE Communications Magazine_ , vol. 61, no. 9, pp. 138–144, 2023.
  * [12] J. X. Salvat Lozano, J. A. Ayala-Romero, L. Zanzi, A. Garcia-Saavedra, and X. Costa-Perez, “O-ran experimental evaluation datasets,” 2022. [Online]. Available: https://dx.doi.org/10.21227/64s5-q431


# A Convolutional-Attentional Neural Framework for Structure-Aware
Performance-Score Synchronization

Ruchit Agrawal, †, Daniel Wolff ⋆, and Simon Dixon †
† Centre for Digital Music, Queen Mary University of London, UK
⋆ Institute for Research and Coordination in Acoustics/Music, Paris, France
This manuscript was accepted to IEEE SPL on 18-11-2021. This research is
supported by the European Union’s Horizon 2020 research and innovation
programme under the Marie Skłodowska-Curie grant agreement No. 765068.Ruchit
Agrawal (e-mail<EMAIL_ADDRESS>and Simon Dixon (e-mail:
<EMAIL_ADDRESS>are with the Centre for Digital Music, Queen Mary
University of London, Mile End Road, London E1 4NS, United Kingdom.Daniel
Wolff (e-mail<EMAIL_ADDRESS>is with the Institute for Research and
Coordination in Acoustics/Music, 75004 Paris, France.

###### Abstract

Performance-score synchronization is an integral task in signal processing,
which entails generating an accurate mapping between an audio recording of a
performance and the corresponding musical score. Traditional synchronization
methods compute alignment using knowledge-driven and stochastic approaches,
and are typically unable to generalize well to different domains and
modalities. We present a novel data-driven method for structure-aware
performance-score synchronization. We propose a convolutional-attentional
architecture trained with a custom loss based on time-series divergence. We
conduct experiments for the audio-to-MIDI and audio-to-image alignment tasks
pertained to different score modalities. We validate the effectiveness of our
method via ablation studies and comparisons with state-of-the-art alignment
approaches. We demonstrate that our approach outperforms previous
synchronization methods for a variety of test settings across score modalities
and acoustic conditions. Our method is also robust to structural differences
between the performance and score sequences, which is a common limitation of
standard alignment approaches.

###### Index Terms:

Performance-score synchronization, audio-to-score alignment, convolutional
neural networks, multimodal data, time-series alignment, stand-alone self-
attention

##  I Introduction

The alignment of time-series data corresponding to different sources of
information is an important task in signal processing, with applications in a
variety of scenarios such as subtitle synchronization, performance analysis
and speech recognition. Performance-score synchronization is one such
alignment task that is aimed at computing the optimal mapping between a
performance and the score for a given piece of music. Depending upon the task,
the alignment computation is either carried out online, known as _score
following_ , or offline, known as _performance-score synchronization_ or
_audio-to-score alignment_. In this paper, we focus on the offline alignment
task, i.e. _performance-score synchronization_. Traditional alignment methods
are generally based on Dynamic Time Warping (DTW) [1] or Hidden Markov Models
(HMMs) [2]. A particular limitation of DTW-based methods is the inability to
capture structural differences between the two input sequences, since the
alignments computed using DTW are constrained to progress monotonically
through both the sequences. Similarly, the Markov assumption in HMMs limits
contextual modelling, which has proven to be useful in various ML tasks [3, 4,
5]. Data-driven approaches have shown promise for various signal processing
tasks, including alignment [6], [7]. Recent methods have proposed the use of
neural networks for the similarity computation [8, 9], coupled with DTW for
the alignment computation. This paper furthers the development of data-driven
alignment approaches and proposes a neural architecture for learnt alignment
computation, thereby eschewing the limitations of DTW-based alignment.

We present a novel neural method for performance-score synchronization, which
is also robust to structural differences between the performance and the
score. We propose a convolutional-attentional encoder-decoder architecture,
with the encoder based on a convolutional stem and the decoder based on a
stand-alone self-attention block [10]. Our motivation behind employing the
stand-alone self-attention block, as opposed to the more commonly used
approach of an attention computation on top of the convolution operation, is
that the stand-alone self-attention layers have proven to be effective at
capturing global relations in vision tasks when employed in later stages of a
convolutional network [10]. In addition to proposing a novel architecture, we
employ a customized divergence loss function based on the differentiable soft-
DTW computation [11] to train our models. We conduct experiments for two
synchronization tasks involving different score modalities, namely audio-to-
MIDI alignment, i.e. aligning audio to symbolic music representations, and
audio-to-image alignment, i.e. aligning audio to scanned images of sheet
music. Our results demonstrate that the proposed method generates robust
alignments in both settings. The primary contributions of this paper are
summarized below:

  * •

We present a novel neural architecture for learning structure-aware
performance-score synchronization.

  * •

We demonstrate that combining stand-alone self-attention layers with a
convolutional stem outperforms multiple alignment methods across different
test settings.

  * •

We demonstrate that our method effectively handles performances containing
structural deviations from the score.

  * •

We demonstrate that the custom soft-DTW based divergence is an effective loss
function for training performance-score synchronization models.

Figure 1: Schematic diagram illustrating our model architecture.
The output alignment path is plotted against the distance matrix for a simple
example to aid visualization.

##  II Related Work

Traditional performance-score synchronization approaches relied mainly on DTW
[12, 1, 13, 14] or HMMs [15, 16, 17, 18]. Methods were subsequently proposed
that modified DTW for optimizing alignment performance across various settings
[19, 20, 21, 22]. With the advent of data-driven approaches, recent methods
have demonstrated the efficacy of learnt representations coupled with DTW-
based alignment computation for performance synchronization [8, 23, 24, 9]. A
step further are fully learnt methods, which have been explored for generic
multiple sequence alignment. These include Deep Canonical Time Warping [25],
the first deep temporal alignment method for simultaneous feature selection
and alignment; NeuMATCH [26], an alignment method based on LSTM blocks, and
more recently, Neural Time Warping [27] that models multiple sequence
alignment as a continuous optimization problem. While neural methods have
recently been explored for score following [28, 29], their application to
performance-score synchronization remains relatively unexplored.

An approach related to our method is the audio-conditioned U-net for position
estimation in sheet images [30], later adapted for score following by
addressing the temporal aspects of the task [29]. While they employ a U-net
architecture and focus on score following, we present a hybrid convolutional-
attentional architecture and focus on performance-score synchronization. We
also employ a custom loss to train our models. Additionally, our method is
able to handle structural deviations from the score, which is a limitation of
their model. Approaches to specifically tackle structural differences in
audio-to-score alignment include $\mathit{JumpDTW}$ [31], Needleman-Wunsch
Time Warping [32] ($\mathit{NWTW}$) and progressively dilated convolutional
neural networks [7]. Our method differs from these in that we do not
explicitly model the structural deviations, and our architecture learns to
model them inherently while predicting the alignment. Additionally, the
alignment computation is performed by our network itself, rather than relying
on a dynamic programming framework.

##  III Proposed Method

We model the performance-score synchronization task as a sequence prediction
task, given the two input sequences corresponding to the performance and score
respectively. However, rather than relying on recurrent neural networks or
Transformers [33] and predicting the output sequence one token at a time, we
propose a convolutional-attentional architecture that predicts the entire
alignment path in a one-shot fashion. This allows the model to capture long-
term dependencies and also handle structural differences between the
performance and score sequences. The architecture of our model is depicted in
Figure 1. Our network operates on the cross-similarity matrix between the
performance and score feature sequences and predicts the ($x$, $y$) co-
ordinates corresponding to the frame indices that make up the optimal
alignment path. Since the $X$-axis of the matrix corresponds to the
performance, it progresses linearly in the alignment and the goal is
essentially to predict the sequence of $y$ co-ordinates (i.e. frame indices in
score axis) that determine the alignment path. Formally, let $X$=
$(x_{1},x_{2},...,x_{p})$ and $Y$ = $(y_{1},y_{2},...,y_{q})$ be the feature
sequences corresponding to the performance and score respectively. The network
is trained to predict the sequence of frame indices $\hat{Y}_{p}$=
$(\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{p})$, denoting the path taken by the
performance $X$ through the score $Y$.

Our network has an encoder-decoder architecture, with the encoder comprising
four convolutional and downsampling blocks, and the decoder comprising an
upsampling block, a stand-alone self-attention (hereafter abbreviated as
_SASA_) block and a fully connected block. It must be noted that the _SASA_
block employed by our decoder is different from the commonly explored
combination of an attention computation applied on top of a convolution
operation [34], or the self-attention layer from the sequence to sequence
Transformer architecture[33]. The _SASA_ block borrows ideas from both
convolution and self-attention, and is able to replace spatial convolutions
completely and effectively integrate global information, especially when
deployed in the later stage of a convolutional neural network [10]. As part of
our upsampling strategy, we employ the max-unpooling operation [35] as opposed
to the transposed convolution, which has been shown to result in artifacts
[36]. We store the indices of the highest activations during pooling and pass
these recorded locations to the upsampling block, where the max-unpooling
places each element in the unpooled map according to the mask, instead of
assigning it to the upper-left pixel.

The upsampled output is passed on to the attentional block, comprising two
_SASA_ layers. For each pixel $(i,j)$ in the upsampled output, we compute the
self-attention relative to the memory block $M_{k}(i,j)$, which is a
neighbourhood with spatial extent $k$ centered around $(i,j)$, as follows:

$y_{ij}=\sum\limits_{a,b\in
M_{k}(i,j)}\mathit{softmax_{ab}}(q_{ij}^{\intercal}k_{ab}+q_{ij}^{\intercal}r_{a-i,b-j})\hskip
2.84544ptv_{ab}$ (1)

where $q_{ij}=W_{q}x_{ij}$ are the queries, $k_{ab}=W_{k}x_{ab}$ the keys and
$v_{ab}=W_{v}x_{ab}$ the values computed as linear transformations from the
activations at the $(i,j)^{th}$ pixel and its memory block. The displacements
from the current position $(i,j)$ to the neighborhood pixel $(a,b)$ are
encoded by row and column offsets, given by $r_{a-i}$ and $r_{b-j}$
respectively. We employ four attention heads and split the pixel features
depthwise into four groups of the same size. The attention is then computed on
each group individually with different matrices $W$ and the results are
concatenated to yield the pixel-wise attention values $y_{ij}$. This
computation is repeated twice and the output is passed through a fully
connected block with two dense layers to predict the alignment path. A graphic
elaboration can be found in Figure 2 in the supplementary material.

We employ a time-series divergence loss function to train our models, as
opposed to using a cross entropy loss. The primary motivation behind this loss
is that it allows us to minimize the overall cost of aligning the performance
and score feature sequences by comparing the paths rather than the feature
sequences using a positive definite divergence. Our loss function captures the
divergence between the predicted and ground truth alignment sequences, based
on the soft-DTW [11] distance. We employ soft-DTW since it offers a
differentiable measure of the discrepancy between the two sequences. Given the
predicted alignment sequence $\hat{Y}$=
$(\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{p})$ and the ground truth alignment
sequence $Y$ = $(y_{1},y_{2},...,y_{r})$, we compute the soft-DTW distance
$D_{\lambda}(p,r)$ as follows:

$D_{\lambda}(p,r)=e(p,r)+min_{\lambda}\begin{cases}D_{\lambda}(p,r-1)\\\
D_{\lambda}(p-1,r)\\\ D_{\lambda}(p-1,r-1)\\\ \end{cases}$ (2)

where $e(p,r)$ is the Euclidean distance between points $\hat{y}_{p}$ and
$y_{r}$, and $min_{\lambda}$ is the soft-min operator parametrized by a
smoothing factor $\lambda$, as follows:

$min_{\lambda}\\{m_{1},m_{2},...,m_{n}\\}=\begin{cases}min\\{m_{1},m_{2},...,m_{n}\\}\hskip
14.22636pt\lambda=0\\\ -\lambda\log\sum_{i=1}^{i=n}e^{-m_{i}/\lambda}\\\
\end{cases}$ (3)

We then normalize $D_{\lambda}(p,r)$ in order to make it a positive definite
divergence [37], as follows:

$SD_{\lambda}(\hat{Y},Y)=D_{\lambda}(\hat{Y},Y)-1/2(D_{\lambda}(\hat{Y},\hat{Y})+D_{\lambda}(Y,Y))$
(4)

This ensures that $SD_{\lambda}(\hat{Y},Y)>0$ for $\hat{Y}\neq Y$ and
$SD_{\lambda}(\hat{Y},Y)=0$ for $\hat{Y}=Y$, yielding a completely learnable
framework since $SD_{\lambda}(\hat{Y},Y)$ is non-negative and differentiable
at all points. Note that we only employ the distance metric $D_{\lambda}(p,r)$
from the soft-DTW computation in Equation 2, and not the alignment path
itself. The alignment computation is carried out by our neural framework, by
minimizing the custom divergence loss $SD_{\lambda}(\hat{Y},Y)$.

##  IV Experiments and Results

We conduct experiments for two alignment tasks pertinent to different score
modalities, namely audio-to-MIDI alignment and audio-to-image alignment. For
each performance-score pair, the cross-similarity matrix is computed using the
Euclidean distance between the chromagrams for audio-to-MIDI alignment, and
the Euclidean distance between learnt cross-modal embeddings [8] for audio-to-
image alignment. We employ librosa [38] for computing the chromagrams and the
cross-similarity matrices. We employ a sampling rate of 22050 Hz, a frame
length of 2048 samples and a hop length of 512 samples for the chromagram
computation. On the encoder side, the output of each 2D convolution is batch
normalized and passed through a Rectified Linear Unit (ReLU) non-linearity,
before being passed on to max-pooling. We employ a dropout of 0.4 for the
fully connected layers to avoid overfitting. The output of the final layer is
a vector of length 2048, encoding the y-indices making up the alignment path.
During training and testing, each performance feature sequence is scaled to
length 2048, and the score feature sequence is padded accordingly. These are
then rescaled back to the original dimensions for comparing the predicted
alignment with the ground truth. It must be noted that the output vector is
sufficient to capture the length of all pieces in the data, since the audio-
to-MIDI task has beat-level annotations (less than 2048 per piece), and the
audio-to-image task has notehead-level annotations (also less than 2048 per
piece). Since the entire vector is predicted in a one-shot manner, we do not
encounter instability issues and hence do not explore smoothing for refining
the predicted alignment path.

We employ two publicly available datasets for our experiments on the two
alignment tasks. For the audio-to-MIDI alignment task, we employ the Mazurka-
BL dataset [39]. This dataset comprises 2000 recordings with annotated
alignments at the beat level. The recordings correspond to performances of
Chopin’s Mazurkas dating from 1902 to the early 2000s, and span various
acoustic settings. We randomly divide this set into sets of 1500, 250 and 250
recordings respectively, forming the training, validation and testing sets. We
compare the results obtained by our models with MATCH [1], JumpDTW [31],
SiameseDTW [9], DilatedCNN [7], and the Deep Canonical Time Warping
_(DeepCTW)_ method [25]. We also conduct ablative studies to analyze the
specific improvements obtained by employing _SASA_ and the custom loss
function in our architecture. To this end, we replace the _SASA_ layers in our
model with convolutional layers keeping the input and output dimensionalities
constant. The resulting Conv-Deconv models are abbreviated as _CD_ x, with $x$
denoting the loss function employed, i.e. _CE_ for the cross-entropy loss and
_custom_ for the custom loss. Our convolutional-attentional models are
similarly abbreviated as _CA_ x. We compute the percentage of beats aligned
correctly within error margins of 50, 100 and 200 ms respectively for each
piece, and report the alignment accuracy [40] obtained by each model averaged
over the entire test set in Table I. We also conduct significance testing
using the Diebold-Mariano test [41] and perform pairwise comparisons of all
model predictions with the _CA_ ${}_{\textit{custom}}$ predictions for each
error margin (Table I).

For the audio-to-image alignment task, we employ the Multimodal Sheet Music
Dataset [28], a standard dataset for sheet image alignment analysis. It
comprises polyphonic piano music for 495 classical pieces, with notehead-level
annotations linking the audio files to the sheet images. We divide this set
randomly into sets of 400, 50 and 45 recordings respectively, forming the
training, validation and testing sets. We compare the results obtained by our
model with contemporary audio-to-image alignment methods Dorfer et al. [8],
[6], Henkel et al. [29], and the DilatedCNN model [7]. For comparison with
Henkel et al. [29], we interpolate their predicted sheet image co-ordinates to
the time domain from the ground truth alignment between the note onsets and
the corresponding notehead co-ordinates in the sheet images. We then compute
the alignment accuracy obtained by each model by calculating the percentage of
onsets aligned correctly within the error margins of 500 ms, 1 s and 2 s
respectively, and report the results averaged over the test set along with the
significance tests in Table II. Note that we employ the same feature
representation [8] for all audio-to-image methods.

In addition to the overall accuracy on the test sets, we also report alignment
results for structurally different performance-score pairs for both datasets.
In order to specifically test the model performance on structure-aware
alignment for both the tasks, we generate 20% additional samples that contain
structural differences between the score and the performance. These are
generated using a randomized split-join operation using the audio from the
respective datasets. We append 50% of these samples to our training sets and
employ the other 50% as our testing sets. The ground truth alignments are
extrapolated from the original alignments using the split-join locations. The
two best performing models for each evaluation setup are highlighted in Tables
I and II.

Model | Overall | Structure
---|---|---
$<$50 ms |  $<$100 ms |  $<$200 ms |  $<$100 ms
MATCH[1] | 74.6* | 79.5* | 85.2* | 67.4*
JumpDTW[31] | 75.2* | 80.4* | 86.7* | 76.2*
_SiameseDTW_ [9] | 77.9 | 83.3* | 89.5* | 72.8*
DeepCTW[25] | 76.1* | 81.6* | 88.9* | 75.6*
DilatedCNN[7] | 77.5* | 82.4* | 90.4* | 80.3
_CD ${}_{\textit{CE}}$_ | 72.8* | 80.1* | 85.3* | 71.9*
_CD ${}_{\textit{custom}}$_ | 74.1* | 81.7* | 87.5* | 74.2*
_CA ${}_{\textit{CE}}$_ | 76.4* |  84.1* |  90.9* | 76.8*
_CA ${}_{\textit{custom}}$_ | 78.7 | 85.2 | 92.6 | 79.5
TABLE I: Audio-to-MIDI alignment accuracy in % on the _Mazurka-BL_ dataset.
Best in bold, second best underlined.
$*$: significant differences from _CA ${}_{\textit{custom}}$_, $p<0.05$

##  V Discussion and Conclusion

The results obtained by our method for audio-to-MIDI alignment are given in
Table I. Overall accuracy on the Mazurka-BL dataset suggests that our best
model outperforms the DTW-based frameworks _MATCH_ , _JumpDTW_ and
_SiameseDTW_ by up to 7% (Table I, rows 1-3) as well as the neural frameworks
$\mathit{DeepCTW}$ and $\mathit{Dilated}\mathit{CNN}$) by up to 4% (Table I,
rows 4-5) for all error margins. The comparison with contemporary approaches
reveals that our method yields higher improvement over the state-of-the art
for coarse alignment (Error margins $>$ 50ms) than for fine-grained alignment
(Error margin $<$ 50ms). Moreover, the ablative studies suggest that the
convolutional-attentional architecture (_CA_) outperforms the conv-deconv
architecture (_CD_) by 3-5%, with higher improvement (4-6%) observed on
structurally different pieces (Table I, column 4). Additionally, the custom
loss yields an improvement of 1-3% over the cross-entropy loss, for both the
_CD_ and _CA_ architectures (Table I, rows 6-9), with the _CA_
${}_{\textit{custom}}$ model yielding the best overall performance. The
experimentation for audio-to-image alignment similarly reveals that _CA_
${}_{\textit{custom}}$ outperforms Dorfer et al. [8, 6] and DilatedCNN [7] in
overall alignment accuracy by 2-10% and Henkel et al. [29] by 1-4% for all
error margins (Table II, columns 1-3). A further advantage of our method over
Henkel et al. [29] is the ability to work with pieces containing several pages
of sheet music, as opposed to only one. Our model is also able to handle
structural deviations from the score, which is a limitation of the majority of
alignment approaches, including Henkel et al. [29] (Table II, column 4).

Model | Overall | Structure
---|---|---
$<$0.5 s |  $<$1 s |  $<$2 s |  $<$1 s
Dorfer et al. 2017[8] | 73.5* | 81.2* | 84.7* | 67.8*
Dorfer et al. 2018 [6] | 76.4* | 84.5* | 89.3* | 70.3*
DilatedCNN[7] | 82.8* | 87.6* |  90.8* | 78.5
Henkel et al. 2020 [29] | 84.6 |  88.4* | 90.1* | 72.1*
_CA ${}_{\textit{custom}}$_ | 85.2 | 91.5 | 92.9 | 77.4
TABLE II: Audio-to-Image alignment accuracy in % on the _MSMD_ dataset. Best
in bold, second best underlined.
$*$: significant differences from _CA ${}_{\textit{custom}}$_, $p<0.05$

Experiments specifically on structure-aware alignment demonstrate that our
method outperforms all approaches except DilatedCNN [7] for both the tasks by
3-12% (Tables 1, 2, column 4). Our model _CA_ ${}_{\textit{custom}}$
demonstrates comparable results to DilatedCNN [7], without explicitly modeling
structure, and while being trained on limited structure-aware data. The
ablative analysis demonstrates that the _CA_ x models yield better structure-
aware alignment than the _CD_ x models (Table I, rows 6-9, column 4),
confirming that the stand-alone self-attention layers in the decoder
facilitate long-term contextual incorporation. Manual inspection of the
alignment plots corroborated that _CA_ ${}_{\textit{custom}}$ was able to
capture structural deviations such as jumps and repeats. The reader can find
such examples in Figure 3 in the supplementary material.

To conclude, we demonstrate that the proposed convolutional-attentional
architecture trained with a custom time-series divergence loss is a promising
framework for performance-score synchronization. Our approach is compatible
with both multi-modal and uni-modal data, since the similarity and alignment
computations are done separately. Our method is also robust to structural
differences between the performance and score sequences without explicit
structure modeling. In the future, we would like to explore multi-modal
methods that work directly with raw data, and dynamic neural methods that can
adjust to the alignment granularity needed for the task at hand.

## References

  * [1] Simon Dixon,  “An on-line time warping algorithm for tracking musical performances,”  in International Joint Conference on Artificial Intelligence (IJCAI), 2005, pp. 1727–1728.
  * [2] Meinard Müller,  Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications,  Springer, 2015.
  * [3] Tebbifakhr Amirhossein, Ruchit Rajeshkumar Agrawal, Rajen Chatterjee, Matteo Negri, and Marco Turchi,  “Multi-source transformer with combined losses for automatic post editing,”  in Third Conference on Machine Translation (WMT). The Association for Computational Linguistics, 2018, pp. 859–865.
  * [4] Ruchit Agrawal and Turchi Marco,  “Contextual handling in neural machine translation: Look behind,”  Ahead and on Both Sides, 2018.
  * [5] Ruchit Agrawal, Vighnesh Chenthil Kumar, Vigneshwaran Muralidharan, and Dipti Misra Sharma,  “No more beating about the bush: A step towards idiom handling for indian language NLP,”  in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
  * [6] Matthias Dorfer, Florian Henkel, and Gerhard Widmer,  “Learning to listen, read, and follow: Score following as a reinforcement learning game,”  in International Society for Music Information Retrieval Conference (ISMIR), 2018.
  * [7] Ruchit Agrawal, Daniel Wolff, and Simon Dixon,  “Structure-aware audio-to-score alignment using progressively dilated convolutional neural networks,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 571–575.
  * [8] Matthias Dorfer, Andreas Arzt, and Gerhard Widmer,  “Learning audio-sheet music correspondences for score identification and offline alignment,”  Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR), 2017.
  * [9] Ruchit Agrawal and Simon Dixon,  “Learning frame similarity using Siamese networks for audio-to-score alignment,”  in 28th European Signal Processing Conference (EUSIPCO). IEEE, 2020, pp. 141–145.
  * [10] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens,  “Stand-alone self-attention in vision models,”  33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
  * [11] Marco Cuturi and Mathieu Blondel,  “Soft-DTW: A differentiable loss function for time-series,”  in International Conference on Machine Learning. PMLR, 2017, pp. 894–903.
  * [12] Meinard Müller, Frank Kurth, and Tido Röder,  “Towards an efficient algorithm for automatic score-to-audio synchronization,”  in International Conference on Music Information Retrieval (ISMIR), 2004.
  * [13] Meinard Müller, Henning Mattes, and Frank Kurth,  “An efficient multiscale approach to audio synchronization,”  in International Conference on Music Information Retrieval (ISMIR), 2006, pp. 192–197.
  * [14] Andreas Arzt, Gerhard Widmer, and Simon Dixon,  “Automatic page turning for musicians via real-time machine listening.,”  in 18th European Conference on Artificial Intelligence (ECAI), 2008, pp. 241–245.
  * [15] Arshia Cont, Diemo Schwarz, and Norbert Schnell,  “Training IRCAM’s score follower (audio to musical score alignment system),”  in International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 2005, vol. 3, pp. iii–253.
  * [16] Akira Maezawa, Hiroshi G Okuno, Tetsuya Ogata, and Masataka Goto,  “Polyphonic audio-to-score alignment based on Bayesian latent harmonic allocation hidden Markov model,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2011, pp. 185–188.
  * [17] Philippe Cuvillier and Arshia Cont,  “Coherent time modeling of semi-Markov models with application to real-time audio-to-score alignment,”  in International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2014, pp. 1–6.
  * [18] Rong Gong, Philippe Cuvillier, Nicolas Obin, and Arshia Cont,  “Real-time audio-to-score alignment of singing voice based on melody and lyric information,”  in Sixteenth Annual Conference of the International Speech Communication Association, 2015.
  * [19] Feng Zhou and Fernando Torre,  “Canonical time warping for alignment of human behavior,”  Advances in Neural Information Processing Systems, vol. 22, pp. 2286–2294, 2009.
  * [20] Andreas Arzt and Gerhard Widmer,  “Towards effective ‘any-time’ music tracking,”  in Proceedings of the Fifth Starting AI Researchers’ Symposium. IOS Press, 2010, pp. 24–36.
  * [21] Julio José Carabias-Orti, Francisco J Rodríguez-Serrano, Pedro Vera-Candeas, Nicolás Ruiz-Reyes, and Francisco J Cañadas-Quesada,  “An audio to score alignment framework using spectral factorization and dynamic time warping,”  in International Society for Music Information Retrieval Conference (ISMIR), 2015, pp. 742–748.
  * [22] Siying Wang, Sebastian Ewert, and Simon Dixon,  “Robust and efficient joint alignment of multiple musical performances,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 11, pp. 2132–2145, 2016.
  * [23] Ruchit Agrawal and Simon Dixon,  “A hybrid approach to audio-to-score alignment,”  Machine Learning for Media Discovery workshop at International Conference on Machine Learning (ICML), 2019.
  * [24] Thitaree Tanprasert, Teerapat Jenrungrot, Meinard Müller, and TJ Tsai,  “MIDI-sheet music alignment using bootleg score synthesis,”  in Proceedings of the International Society for Music Information Retrieval (ISMIR), Delft, The Netherlands, 2019.
  * [25] George Trigeorgis, Mihalis A Nicolaou, Björn W Schuller, and Stefanos Zafeiriou,  “Deep canonical time warping for simultaneous alignment and representation learning of sequences,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 5, pp. 1128–1138, 2017.
  * [26] Pelin Dogan, Boyang Li, Leonid Sigal, and Markus Gross,  “A neural multi-sequence alignment technique (neumatch),”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [27] Keisuke Kawano, Takuro Kutsuna, and Satoshi Koide,  “Neural time warping for multiple sequence alignment,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 3837–3841.
  * [28] Matthias Dorfer, Jan Hajič Jr, Andreas Arzt, Harald Frostel, and Gerhard Widmer,  “Learning audio–sheet music correspondences for cross-modal retrieval and piece identification,”  Transactions of the International Society for Music Information Retrieval, vol. 1, no. 1, 2018.
  * [29] Florian Henkel, Rainer Kelz, and Gerhard Widmer,  “Learning to read and follow music in complete score sheet images,”  Proceedings of the 21th International Society for Music Information Retrieval Conference (ISMIR), 2020.
  * [30] Florian Henkel, Rainer Kelz, and Gerhard Widmer,  “Audio-conditioned U-net for position estimation in full sheet images,”  International Workshop on Reading Music Systems (WoRMS), 2019.
  * [31] Christian Fremerey, Meinard Müller, and Michael Clausen,  “Handling repeats and jumps in score-performance synchronization,”  in International Society for Music Information Retrieval Conference (ISMIR), 2010, pp. 243–248.
  * [32] Maarten Grachten, Martin Gasser, Andreas Arzt, and Gerhard Widmer,  “Automatic alignment of music performances with structural differences,”  in Proceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR), 2013.
  * [33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in Neural Information Processing Systems, 2017, pp. 6000–6010.
  * [34] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.,  “Attention U-net: Learning where to look for the pancreas,”  Medical Imaging with Deep Learning, 2018.
  * [35] Matthew D Zeiler and Rob Fergus,  “Visualizing and understanding convolutional networks,”  in European Conference on Computer Vision. Springer, 2014, pp. 818–833.
  * [36] Augustus Odena, Vincent Dumoulin, and Chris Olah,  “Deconvolution and checkerboard artifacts,”  Distill, vol. 1, no. 10, 2016.
  * [37] Mathieu Blondel, Arthur Mensch, and Jean-Philippe Vert,  “Differentiable divergences between time series,”  in International Conference on Artificial Intelligence and Statistics. PMLR, 2021, pp. 3853–3861.
  * [38] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto,  “librosa: Audio and music signal analysis in Python,”  in Proceedings of the 14th Python in Science Conference, 2015, pp. 18–25.
  * [39] Katerina Kosta, Oscar F Bandtlow, and Elaine Chew,  “Mazurka-BL: Score-aligned loudness, beat, expressive markings data for 2000 Chopin mazurka recordings,”  in Proceedings of the Fourth International Conference on Technologies for Music Notation and Representation (TENOR), 2018, pp. 85–94.
  * [40] Arshia Cont, Diemo Schwarz, Norbert Schnell, and Christopher Raphael,  “Evaluation of real-time audio-to-score alignment,”  in International Conference on Music Information Retrieval (ISMIR), 2007.
  * [41] David Harvey, Stephen Leybourne, and Paul Newbold,  “Testing the equality of prediction mean squared errors,”  International Journal of Forecasting, vol. 13, no. 2, pp. 281–291, 1997.