# Classifying Nearly Complete Intersection Ideals Generated in Degree Two

Charlie Miller<EMAIL_ADDRESS>and  Branden Stone
<EMAIL_ADDRESS>
###### Abstract.

Nearly complete intersection ideals were introduced in [BS18] and defines a
special class of monomial ideals in a polynomial ring. These ideals were used
to give a lower bound of the total sum of betti numbers that appear a minimal
free resolution of a monomial ideal. In this note we give a graph theoretic
classification of nearly complete intersection ideals generated in degree two.
In doing so, we define a novel graph operation (the inversion) that is
motivated by the definition of this new class of ideals.

The first author was supported by McGowan Family Fund, a summer research award
for students at Hamilton College, Clinton, NY

##  1\. Introduction

Let $I$ be a homogeneous ideal in a polynomial ring $R$ over a field $\Bbbk$.
We denote the rank of the $i$-th free module in a minimal free resolution of
$R/I$ as $\beta_{i}(R/I)$. The long-standing conjecture of Buchsbaum-Eisenbud
[BE77] and Horrocks [Har79] states that if $I$ has height $c$, then

$\beta_{i}(R/I)\geqslant\binom{c}{i}.$

While the case when $c\geqslant 5$ is still open, the weaker statement

$\sum\beta_{i}(R/I)\geqslant 2^{c},$

known as the Total Rank Conjecture, has been completely solved for arbitrary
ideals (with char $k\not=2$) by M. Walker [Wal17]. At the same time, a special
case of this conjecture was independently shown for monomial ideals by A.
Boocher and J. Seiner [BS18]. In particular, they show that if $I$ is not a
complete intersection, then

$\sum\beta_{i}(R/I)\geqslant 2^{c}+2^{c-1}.$

In order to achieve this lower bound, the authors reduce to a special class of
ideals they define as nearly complete intersections (NCI) (see Definition
2.2). Our main theorem, Theorem 3.7, gives a complete characterization of NCI
ideals generated in degree 2 by examining the associated graph $G$. For
example, a squarefree monomial ideal $I$ generated in degree 2 is _not_ a
nearly complete intersection if $P_{5}$ is an induced subgraph of $G$. Section
2 gives the necessary background information. The main classification theorem
is proved in Section 3 as well as a new graph operation, the inversion of a
vertex (Definition 3.1), motivated by the definition of this new class of
ideals.

### Acknowledgments

This project is the result of undergraduate summer research supported by
Hamilton College. We are thankful to the faculty in the Math department for
their encouragement throughout the process. In particular, we are grateful to
Courtney Gibbons for her continual guidance and feedback. The first author
would also like to thank the organizers and speakers at the Thematic Program
in Commutative Algebra and its Interaction with Algebraic Geometry held at
Notre Dame in the summer of 2019. This workshop introduced the foundational
material relevant to this work.

##  2\. Preliminaries

Unless otherwise noted, we let $R=\Bbbk[x_{1},x_{2},\dots,x_{n}]$ be a
standard graded polynomial ring over a field $\Bbbk$ in $n$ variables. Given a
monomial ideal $I\subseteq R$, the support of $I$ (denoted $\text{Supp}(I)$)
will refer to the set of variables appearing in at least one minimal monomial
generator. The following fact about the support is helpful throughout the note
and we state it without proof.

######  Lemma 2.1.

Let $R=\Bbbk[x_{1},x_{2},\dots,x_{n}]$ and
$I=(m_{1},m_{2},\dots,m_{t})\subseteq R$ be a monomial ideal. If $m\in R$ is a
monomial such that

$\text{Supp}(m)\cap\left[\bigcup^{t}_{i=1}\text{Supp}(m_{i})\right]=\varnothing,$

then $\overline{m}\in R/I$ is a non-zero divisor.

Using notation defined in [BS18], $I(x=1)$ is the ideal defined by setting
$x=1$ for some variable $x$ in the support of $I$. As such, $I\subseteq
I(x=1)$. E.g., if $I=(ab,bc,ac)\subseteq\Bbbk[a,b,c]$, then $I(a=1)=(b,c)$ and
$I\subseteq I(a=1)$. The following was defined in [BS18] and is the main
object of study in this note.

######  Definition 2.2 ([BS18]).

A squarefree monomial ideal $I\subseteq R$ is a nearly complete intersection
if

  1. (1)

it is generated in degree at least two,

  2. (2)

is not a complete intersection, and

  3. (3)

for each variable $x$ in the support of $I$, $I(x=1)$ is a complete
intersection.

For example, let $I=(ab,ac,bc)\subseteq\Bbbk[a,b,c]$. We see that $I$ is
generated in degree $2$ and is not a complete intersection. Further, for each
element of $\text{Supp}(I)$, $I(a=1)=(b,c)$, $I(b=1)=(a,c)$, and
$I(c=1)=(a,b)$ are complete intersections. Thus $I$ is a nearly complete
intersection.

The main result, Theorem 3.7, completely classifies the NCIs generated in
degree two via their associated graphs. Throughout, a finite graph $G$ is a
pair $G=(V(G),E(G))$ where $V(G)=\\{x_{1},x_{2},\dots,x_{n}\\}$ is the set of
vertices of $G$, and $E(G)$ is a collection edges of $G$ consisting of two
element subsets of $V(G)$. We will further assume all graphs are simple, i.e.
not allowing loops and multiple edges between vertices.

There exists a one-to-one correspondence between finite simple graphs and
monomial ideals generated degree two. In particular, given a graph $G$, the
edge ideal $I(G)$ is typically defined by

$I(G)=\left(x_{i}x_{j}\;|\;\\{x_{i},x_{j}\\}\in
E(G)\right)\subseteq\Bbbk[x_{1},x_{2},\dots,x_{n}],$

where $V(G)=\\{x_{1},x_{2},\dots,x_{n}\\}$. For bookkeeping reasons, we
slightly modify the standard definition of edge ideal to allow for singletons,
while at the same time preserving the one-to-one correspondence. In
particular, in this note the edge ideal of a graph $G$ is defined as

$I(G)=\left(x_{i}x_{j},x_{k}\;|\;\\{x_{i},x_{j}\\}\in E(G),x_{k}\in V(G)\text{
is a singleton}\right)\subseteq\Bbbk[x_{1},x_{2},\dots,x_{n}].$

For example, the graph $G$ below corresponds to the edge ideal
$I(G)=(ab,ac,bc,d)\subseteq\Bbbk[a,b,c,d]$.

$a$$b$$c$$d$$G$:

Abusing notation we will often refer to an element $u_{k}\in I(G)$ both as
$u_{k}=x_{i_{k}}x_{j_{k}}\in I(G)$ and $u_{k}=\\{x_{i_{k}},x_{j_{k}}\\}\in
E(G)$. Using this correspondence, we say a graph $G$ is a nearly complete
intersection (NCI) if the edge ideal $I(G)$ is a nearly complete intersection.
As such classifying the NCI graphs will in turn classify the NCI ideals
generated in degree two.

We end this section with a standard fact about graphs associated to complete
intersections.

######  Lemma 2.3.

Let $G$ be a simple graph and $I(G)\subseteq R$ be its edge ideal. Then $R/I$
is a complete intersection if and only if $G$ is a disjoint union of edges and
singletons.

###### Proof.

Assume that $E(G)=\\{u_{1},u_{2},\dots,u_{n}\\}$ and that $R/I(G)$ is a
complete intersection. Suppose to the contrary that there exists a vertex $v$
of degree 2 in $V(G)$. This implies that there exist edges $u_{i},u_{j}\in
E(G)$ such that $u_{i}\cap u_{j}=\\{v\\}$. Assuming $i=1$ and $j=2$, we have
that $\overline{u_{1}}\in R/(u_{2},\ldots,u_{n})$ is a zero-divisor. As such
there does not exist a vertex of degree two and $G$ must be a disjoint union
of edges and singletons.

Assume $G$ is a disjoint union of edges as well as singletons. With out loss
of generality, we can reduce to the case that $G$ does not contain any
singletons. Thus, the edge ideal of $G$ is
$I(G)=(x_{i}y_{i}\;|\;i=1,\ldots,n)$. Notice that

$\text{Supp}(x_{i}y_{i})\cap\left[\bigcup^{i-1}_{j=1}\text{Supp}(x_{j}y_{j})\right]=\varnothing$

for all $i=1,\ldots,n$. Therefore, $R/I(G)$ is a complete intersection by
Lemma 2.1. ∎

##  3\. Classifying NCIs

As mentioned in the previous section, a graph $G$ is an NCI if the edge ideal
$I(G)$ is a nearly complete intersection as defined in Definition 2.2. The
main result, Theorem 3.7, gives a complete classification of NCI ideals
generated in degree at most 2 using the above graph correspondence. Before we
can prove the result, we define a new graph operation necessary for the proof.
We denote the neighbors of a vertex $v$ in $V(G)$, $\operatorname{N}(v)$, and
the induced subgraph on a subset $V^{\prime}\subseteq V(G)$ as
$G[V^{\prime}]$.

######  Definition 3.1.

The inversion of a vertex $v$ in a graph $G$ is the graph defined by

$\mathscr{I}(v,G)=(V^{\prime},E^{\prime}),$

where $V^{\prime}=V\setminus\\{v\\}$ and
$E^{\prime}=E\left(G\left[V^{\prime}\setminus\mathrm{N}(v)\right]\right)$.

This operation is a direct translation of the operation $I(x=1)$ used in
Definition 2.2 (3) and is the main tool used in the classification of NCI
graphs. With it, we can further formalize NCI graphs with the following lemma
whose proof is a direct translation of definitions.

######  Lemma 3.2.

A graph $G$ is an NCI if and only if

  1. (1)

$G$ is not a complete intersection, and

  2. (2)

for each vertex $v\in V(G)$, $\mathscr{I}(v,G)$ is a complete intersection.

From this we have an immediate corollary.

######  Corollary 3.3.

NCI graphs are connected.

This corollary highlights the observations in Section 4 of [BS18]. In the next
example we can use Lemma 3.2 to determine if graphs are NCI or not.

######  Example 3.4.

Here we have a graph $G$ and two inversions at the vertices $c$ and $f$.
Notice that after the inversions we do not have a complete intersection (Lemma
2.3), hence $G$ is not an NCI since every inversion must be a complete
intersection (Lemma 3.2).

$G$$a$$b$$c$$d$$e$$f$$g$$\mathscr{I}(c,G)$$a$$b$$d$$e$$f$$g$$\mathscr{I}(f,G)$$a$$b$$c$$d$$e$$g$

Above shows that not all graphs are NCI. In fact the NCI property seems to be
quite rare. Below are examples of graphs that are NCI. Notice that any
inversion of a vertex will create a disjoint union of edges and singletons,
i.e. a complete intersection. Applying Lemma 3.2 shows they are NCI.

$C_{4}$$K_{5}$$C_{4}$$P_{3}$$S_{7}$$S_{7}$$S_{7}$$S_{7}$$S_{7}$$S_{7}$$S_{7}$

It’s natural to look at the families these graphs belong to. For example, the
family of paths are not all NCI. Indeed, if $n>4$, then the path $P_{n}$ is
not an NCI. To see this one only needs to invert a leaf of the graph and
notice the resulting graph is not a complete intersection, but another path
connecting at least three vertices. A similar result/argument holds for
cycles, i.e. if $n>5$, then a cycle $C_{n}$ is not an NCI. However, this is
not the case for complete graphs.

######  Proposition 3.5.

Any complete graph with more than 2 vertices is an NCI.

###### Proof.

Let $G=K_{n}$ be a complete graph on $n\geq 3$ vertices. If $v\in V=V(G)$, the
inversion of $v$ is given by

$\mathscr{I}(v,G)=\left(V^{\prime},E\left(G\left[V^{\prime}\setminus\operatorname{N}(v)\right]\right)\right),$

where $V^{\prime}=V\setminus\\{v\\}$. As $G$ is complete, we have that
$N(G)=V\setminus\\{v\\}=V^{\prime}$, and hence

$E\left(G\left[V^{\prime}\setminus\operatorname{N}(v)\right]\right)=\emptyset.$

Thus $\mathscr{I}(v,G)$ is a collection of singletons and hence a complete
intersection by Lemma 2.3. ∎

In the above path and cycle examples, we saw that the threshold for a graph to
be NCI was having $|V(G)|\leqslant 4$ and 5 respectively. It turns out that we
can explicitly state the NCI property for connected graphs with at most 4
vertices.

######  Proposition 3.6.

Let $G$ be a connected graph.

  1. (1)

If $|V(G)|\leqslant 2$, then $G$ is not an NCI.

  2. (2)

If $|V(G)|=3\text{ or }4$, then $G$ is an NCI.

###### Proof.

When $|V(G)|\leqslant 2$ the graph is a complete intersection by Lemma 2.3 and
hence cannot be an NCI by Lemma 3.2. When $|V(G)|=3\text{ or }4$, $G$ cannot
be a complete intersection due to the connected assumption, i.e. any vertex
$v\in V(G)$ must be connected to at least one other vertex. Thus
$\mathscr{I}(v,G)$ has at most one edge and is a complete intersection. This
forces $G$ to be an NCI. ∎

We are now ready to prove the main classification theorem. In the theorem, we
define the graph $T$ as the following.

$T$$v_{1}$

This graph, along with $P_{5}$, become the major obstructions to the NCI
property.

######  Theorem 3.7.

Let $G$ be a connected graph with $|V(G)|\geqslant 5$. The graph $G$ is not an
NCI if and only if there exist vertices $v_{1},v_{2},v_{3},v_{4},v_{5}\in
V(G)$ such that the following conditions hold:

  1. (1)

the vertex $v_{1}$ is a leaf in $G[v_{1},v_{2},v_{3},v_{4},v_{5}]$;

  2. (2)

the path $P_{5}$ or $T$ is a spanning tree of
$G[v_{1},v_{2},v_{3},v_{4},v_{5}]$ where the neighbors of $v_{1}$ all have
degree 2 in the spanning tree.

###### Proof.

Assume $G$ is not an NCI. As such, there exists $v\in V(G)$ such that
$\mathscr{I}(v,G)$ is not a complete intersection. In particular,
$\mathscr{I}(v,G)$ has a vertex $w\in V^{\prime}=V\setminus\\{v\\}$ of degree
two. As $G$ is connected, there must exist a path from $v$ to $w$ that passes
through the neighbors of $v$ in $G$. So there exists
$v_{2}\in\operatorname{N}_{G}(v)$ such that the path

(1) $v\longrightarrow v_{2}\longrightarrow
v_{3}\longrightarrow\cdots\longrightarrow w$

exists in $G$. Without losing generality, we can assume the vertices in the
path from $v_{3}$ to $w$ (inclusive) avoid $\operatorname{N}_{G}(v)$. Indeed
if there was a vertex $u\in\operatorname{N}_{G}(v)$ between $v_{3}$ and $w$,
we could replace $v_{2}$ with $u$, shortening the path. As such, we may assume
the path from $v_{3}$ to $w$ is completely contained in the subgraph
$G[V^{\prime}\setminus\operatorname{N}_{G}(v)]\subset G$. We now consider two
cases, $v_{3}=w$ and $v_{3}\not=w$, which can be visualized in the following
abstract representation of $G$.

$v$$v_{2}$$v_{3}$$w^{\prime}$$w$$w^{\prime\prime}$$\operatorname{N}_{G}(v)$III$G[V^{\prime}-\operatorname{N}_{G}(v)]$

### Case I

Assume $v_{3}=w$. Since $w$ is a degree two vertex in $\mathscr{I}(v,G)$,
there exist $w^{\prime},w^{\prime\prime}\in V^{\prime}$ such that
$ww^{\prime},ww^{\prime\prime}\in
E^{\prime}=E\left(G\left[V^{\prime}\setminus\mathrm{\operatorname{N}}_{G}(v)\right]\right)$.
In particular $w^{\prime},w^{\prime\prime}\notin\operatorname{N}_{G}(v)$. As
such, $v$ is a leaf in the induced subgraph
$H=G[v,v_{2},w,w^{\prime},w^{\prime\prime}]$, and by construction, $T$ is a
spanning tree of $H$ where $v_{2}$ is the only neighbor of $v$. Further, the
degree of $v_{2}$ is two in the spanning tree $T$, thus both of the desired
conditions are satisfied.

### Case II

Assume $v_{3}\not=w$. As $v_{3}$ and $w$ are distinct, we can reduce to the
case where there is a single vertex between them on the path (1), say
$w^{\prime}$. As $v_{3},w^{\prime},w\notin\operatorname{N}_{G}(v)$, we have
that $v$ is a leaf in the induced subgraph $H=G[v,v_{2},v_{3},w^{\prime},w]$.
In this case, by construction, $P_{5}$ is a spanning tree of $H$ where $v_{2}$
is the only neighbor of $v$. As the degree of $v_{2}$ is two in the spanning
tree $P_{5}$, we have our desired result.

Conversely, assume the conditions hold for a graph $G$ that is NCI. In this
scenario, there exists vertices $v_{1},v_{2},v_{3},v_{4},v_{5}\in V(G)$ such
that $v_{1}$ is a leaf in the induced subgraph
$G[v_{1},v_{2},v_{3},v_{4},v_{5}]$. In the situation where $P_{5}$ is a
spanning tree of $G[v_{1},v_{2},v_{3},v_{4},v_{5}]$, we can assume the vertex
labels of the path are as follows.

$P_{5}$:$v_{1}$$v_{2}$$v_{3}$$v_{4}$$v_{5}$

Since $v_{1}$ is a leaf in the induced subgraph, we know that
$v_{3},v_{4},v_{5}\notin\operatorname{N}_{G}(v)$. Hence the degree of $v_{4}$
is at least two in $\mathscr{I}(v_{1},G)$. This shows that
$\mathscr{I}(v_{1},G)$ is not a complete intersection, a contradiction of
Lemma 3.2. A similar argument holds for when $T$ is a spanning tree of
$G[v_{1},v_{2},v_{3},v_{4},v_{5}]$. ∎

Theorem 3.7, together with Proposition 3.6 give a complete classification of
NCI graphs. As a result, we have a graph theoretic classification of NCI
ideals generated in degree two. A natural desire is to extend this result to
NCIs with generators in higher degrees. One direction to consider is
classifying these ideals with hypergraphs. A Hypergraph is a pair $G=(V,E)$
where $V$ is the set of vertices of $G$ and the set of edges $E$ is a set of
nonempty subsets of $V$. In this scenario, more than two vertices can be
incident to a single edge. As with graphs, a similar correspondence exits
between hypergraphs and ideals and can be seen in the following example.

######  Example 3.8.

The left image below is an example of an NCI hypergraph $G$ on a vertex set
$V(G)=\\{a,b,c,d,e,f,g\\}$. Notice this hypergraph has three edges,
$\\{a,b,c\\}$, $\\{g\\}$, and $\\{d,e,f\\}$. Further, there is a natural
correspondence between these hypergraphs and monomial ideals in
$\Bbbk[a,b,c,d,e,f,g]$. In particular $I(G)$ is listed below $G$.

$a$$b$$c$$g$$d$$e$$f$$I(G)=(abc,def,ag,bg,cg,dg,eg,fg)$$b$$c$$g$$d$$e$$f$$\mathscr{I}(a,G)$
is CI

Lemma 3.2 can also be extended to this scenario as well as the definition of
inversion. Notice that inverting $a$ (or any vertex) will produce the complete
intersection on the right. It is worth noting that all the examples of NCI
hypergraphs we were able to construct were related to the above example. This
hints at the possibility that all higher degree NCI ideals are related to the
above hypergraph.

We end this section with an observation relating to the original result of
[BS18]. Let $I$ be a height $c$ monomial ideal in a polynomial ring $S$ that
is not a complete intersection. A. Boocher and J. Seiner show that
$\sum\beta_{i}(S/I)\geqslant 2^{c}+2^{c-1}$. In particular, equality holds if
and only if the generating function for $\beta_{i}(S/I)$ is either

$(1+3t+2t^{2})(1+t)^{c-2}\text{ or }(1+5t+5t^{2}+t^{3})(1+t)^{c-3}.$

When $c=2$ or $3$, respectively, the generating functions are defined by
ideals with the betti sequence $\\{1,3,2\\}$ and $\\{1,5,5,1\\}$,
respectively. We are able to retrieve these sequences from the obstructions
noted in Theorem 3.7. Indeed, the edge ideal $I(P_{5})$ and $I(T)$ both have
the betti sequence $\\{1,4,4,1\\}$. However, if we connect the end points of
the path $P_{5}$ to create a 5-cycle, the betti sequence becomes
$\\{1,5,5,1\\}$. Similarly, removing a leaf of either $P_{5}$ or $T$ can
create the path $P_{4}$, obtaining the betti sequence $\\{1,3,2\\}$.

## References

  * [BE77] David A Buchsbaum and David Eisenbud.  Algebra structures for finite free resolutions, and some structure theorems for ideals of codimension 3.  American Journal of Mathematics, 99(3):447–485, 1977.
  * [BS18] Adam Boocher and James Seiner.  Lower bounds for Betti numbers of monomial ideals.  J. Algebra, 508:445–460, 2018.
  * [Har79] Robin Hartshorne.  Algebraic vector bundles on projective spaces: a problem list.  Topology, 18(2):117–128, 1979.
  * [Wal17] Mark E. Walker.  Total Betti numbers of modules of finite projective dimension.  Ann. of Math. (2), 186(2):641–646, 2017.


# Self-Similarity-Based and Novelty-based loss for music structure analysis

###### Abstract

Music Structure Analysis (MSA) is the task aiming at identifying musical
segments that compose a music track and possibly label them based on their
similarity. In this paper we propose a supervised approach for the task of
music boundary detection. In our approach we simultaneously learn features and
convolution kernels. For this we jointly optimize \- a loss based on the Self-
Similarity-Matrix (SSM) obtained with the learned features, denoted by SSM-
loss, and \- a loss based on the novelty score obtained applying the learned
kernels to the estimated SSM, denoted by novelty-loss. We also demonstrate
that relative feature learning, through self-attention, is beneficial for the
task of MSA. Finally, we compare the performances of our approach to
previously proposed approaches on the standard RWC-Pop, and various subsets of
SALAMI.

##  1 Introduction

Music Structure Analysis (MSA) is the task aiming at identifying musical
segments that compose a music track (a.k.a. segment boundary estimation) and
possibly label them based on their similarity (a.k.a. segment labeling). We
deal here with MSA from audio. MSA is one of the oldest task in Music
Information Retrieval111Foote’s paper [1] on SSM was published in 1999. but
still one of the most challenging. This is due to the difficulty to exactly
define what music structure is and hence be able to create annotated datasets
to measure progress or train systems. People agree that the structure can be
considered from multiple viewpoints222musical role, acoustic similarity,
instrument role, perceptual tests [2][3], is hierarchical [4] and is partly
subjective [5]. Probably because of this complexity, the number of
contributions in MSA has remained low despite its large number of
applications: audio summarization [6], interactive browsing [7, 8, 9], musical
analysis [10], tools for researcher (to help chord recognition [11], source
separation [12] or downbeat estimation [13]).

To solve the two MSA tasks (boundary detection and segment labeling), three
assumptions [14] are commonly used: (1) novelty (we assume that segments are
defined by large —novel— changes of the musical content over time), (2)
homogeneity (the musical content is homogeneous within a given segment) and
(3) repetition (the musical content —homogeneous or not— can be repeated over
time). This has been extended by [15] to a fourth regularity assumption (the
segment’s durations are regular over time). Combining those allows to
construct MSA systems.

###  1.1 Related works

Over time, a large palette of approaches has been proposed for MSA. We only
review the ones related to our work and refer the reader to Nieto et al. [16]
for a good overview. We consider three periods according to the nature of the
audio features – hand-crafted (HC) or learned by deep learning (DL) –, and the
nature of the detection system which uses the audio features – HC or trained
by DL –.

First period: HC detection system applied to HC audio features. In these
systems HC audio features (such as MFCC or Chroma) were given as input to HC
detection system (such as the checkerboard kernel, novelty-score [17]),
unsupervised training (such as HMM [6], NMF [18]), supervised (such as OLDA
[19]) or pattern matching algorithms (such as DTW [20] or variants [21]) .

Second period: DL detection system applied to HC audio features. Over time,
more and larger annotated datasets for MSA have been developed; which
concomitantly with the development of DL has allowed to re-formulate the MSA
task in terms of supervised learning. The detection system developed here
mainly target the task of boundary detection. For example, [22] [23] [24]
propose to train in a supervised way a Convolutional Networks (ConvNet)
$\hat{y}=f^{\theta}(\mathbf{X})$ to estimate if the center of a patch of HC
audio features $\mathbf{X}$ is a boundary ($y$=1). Various HC audio features
(or combinations of) are used here: Log-Mel-Spectrogram, Pich-Class-Profile
through SSM expressed in (time,time) or (time,lag).

Third period: HC detection system applied to DL audio features.  To deal with
the endless debate about the choice of HC audio features, McCallum et al. [25]
propose to learn them. For this, they train an encoder $f^{\theta}$ by
minimizing a Triplet Loss (TL) [26] between patches of beat-synchronous
Constant-Q-Transform (CQT). For the TL, they propose a Self-Supervised-
Learning (SSL) paradigm333which does not require any annotated segments and
labels to define the anchor $A$ patch, positive $P$ patch and negative $N$
patch. Using the homogeneity assumption, neighboring times are supposed to be
more similar to each other (therefore used to define $A$ and $P$) than to
distant ones (used to define $N$). For training they use a very large
unlabeled dataset of 28345 songs. This method however does not consider the
repetition assumption444$N$ could potentially be in a segment which is a
repetition of the segment containing $A$.

Wang et al. [27] revised McCallum approach in a supervised setting. In this,
the patches $P$ (resp. $N$) are now explicitly chosen so as to have the same
(resp. different) annotated segment label than the patches $A$. This
supervised method now consider both the homogeneity and repetition assumption.
In another work [28], they propose a spectral-temporal Transformer-based model
(SpecTNT) trained with a connectionist temporal localization (CTL) loss to
jointly estimate music segments ad their labels.

McCallum approach has also been extended by Buisson et al. [29] to take
benefit from the hierarchy of structure in music. They show that the obtained
deep embeddings can improve segmentation at various levels of granularity.

Rather than learning features for MSA, Salamon et al. [30] proposed to re-use
pretrained ones. Those are obtained using encoders previously trained on
different tasks (Few-Shot Learning sound event and music auto-tagging). Those
are then used as input to a Laplacian Structural Decompositon algorithm for
MSA.

###  1.2 Proposal and paper organization

Figure 1: Proposed architecture and training paradigm minimizing a SSM loss
$\mathcal{L}^{\theta}_{SSM}$ and a novety loss $\mathcal{L}^{\theta}_{nov}$ .

Following the previous taxonomy, our proposal would belong to the category “DL
detection system applied to DL audio features” . Unlike previous feature
learning approaches (that rely on a Triplet Loss paradigm), we utilize a more
straightforward paradigm (illustrated in Figure 1) which is a succession of
two steps, each with its own objective. The two objectives are jointly
optimized.

In the first step, we learn the parameters $\theta$ of an encoder $f^{\theta}$
such that when applied to the sequence of inputs
$\\{\mathbf{X}_{i}\\}_{i\in\\{1\ldots T\\}}$ that represent a given track
(where $T$ is the length of temporal sequence), the encoded features allows
the estimation of a SSM, $\mathbf{\hat{S}}_{ij}^{\theta}$, which attempts to
reproduce a ground-truth SSM, $\mathbf{S}_{ij}$. For training $f^{\theta}$ we
use an approach similar to the SSM-Net approach proposed in [31], i.e.
defining a loss which directly compare the obtained SSM
$\mathbf{\hat{S}}_{ij}^{\theta}$ to a ground-truth SSM $\mathbf{S}_{ij}$.

In the second step, we learn a set of kernels $\mathbf{K}^{\theta}$ such that
when convolved over the main diagonal of the estimated SSM
$\mathbf{\hat{S}}_{ij}^{\theta}$ it allows the estimation of a novelty score
$\mathbf{\hat{n}}_{i}^{\theta}$, which attempts to reproduce a ground-truth
novelty score, $\mathbf{n}_{i}$. This novelty score is usually obtained using
a fixed checkerboard kernel [32]. The resulting function is named novelty
score since high values in it indicate times where the content change (it is
homogeneous before and after). It has been shown that better kernels can be
used (for example using multi-scale kernels [33]) and that it is possible to
train such kernels $\mathbf{K}^{\theta}$ considered as the kernels of a
ConvNet (for example [22] and [23] in the case of a (time,lag) SSM or [24] in
the case of a (time,time) SSM, which is our case). This is the approach we
follow here.

Another proposal we make in this paper, is to consider the learning of
relative features, i.e. features which are relative to the given track.

Paper organization. We provide an overview of our system in part 2, describe
the inputs to our system (part 2.1), detail the two losses (parts 2.2 and
2.3), motivate relative feature learning (part 2.4), detail the architecture
of our encoder $f^{\theta}$ (part 2.5) and the training process (part 2.6). In
part 3, we provide a large-scale evaluation of our proposal. It should be
noted that although we only evaluate our method for the task of segment
boundary detection, it can also be used for segment labeling given the
clearness of the obtained SSM.

##  2 Proposal

###  2.1 Input data $\\{\mathbf{X}_{i}\\}$

The inputs $\\{\mathbf{X}_{i}\\}$ to our system are simple patches555 We
utilized patches as input (rather than frames) because we believe that
homogeneity exists at the pattern level rather than the frame level. of Log-
Mel-Spectrogram. We didn’t consider beat-synchronous features as in [25] given
the non-reliability of beat estimation outside popular music. Using librosa
[34], we first computed Mel-spectrogram with 80 mel-bands, using a 92ms window
length and 23ms hop size. Those are then converted to log-amplitude using
$\log(1\\!+\\!100\cdot mel)$. We then aggregate them (mean operator) over time
to lead to a 0.1s hop size. The final $\\{\mathbf{X}_{i}\\}$ are then patches
of 40 successive frames (corresponding to 4s.) with a hop size of 5 frames
(corresponding to 0.5s.).

###  2.2 SSM-loss

Given a sequence of inputs $\\{\mathbf{X}_{i}\\}_{i\in\\{1\ldots T\\}}$ , we
apply the same encoder $f^{\theta}$ individually to each $\mathbf{X}_{i}$ to
obtain the corresponding sequence of embeddings
$\\{\mathbf{e}^{\theta}_{i}\\}_{i\in\\{1\ldots T\\}}$. Those are then
L2-normalized. We can then easily construct an estimated SSM,
$\mathbf{\hat{S}}_{ij}^{\theta}$, using a distance/similarity/divergence $g$
between all pairs of projections:

$\mathbf{\hat{S}}_{ij}^{\theta}=g(\mathbf{e}^{\theta}_{i}=f^{\theta}(\mathbf{X}_{i}),\mathbf{e}^{\theta}_{j}=f^{\theta}(\mathbf{X}_{j})),\;\;\;\forall
i,j$ (1)

We use here a “scaled” cosine-similarity for $g$ which, because the embeddings
are L2-normalized, reduces to

$\mathbf{\hat{S}}_{ij}^{\theta}=1-\frac{1}{4}\lVert\mathbf{e}^{\theta}_{i}-\mathbf{e}^{\theta}_{j}\rVert_{2}^{2}\;\;\;\in[0,1]$
(2)

It is then possible to compare $\mathbf{\hat{S}}_{ij}^{\theta}$ to a ground-
truth binary SSM, $\mathbf{S}_{ij}$, derived from annotations. We consider
this as a multi-class (a set of $T^{2}$ binary classifications) problem and
hence minimize the sum of Binary-Cross-Entropy (BCE) losses. However, given
the unbalancing between the two classes in $\mathbf{S}_{ij}$ (which contains
much more 0 than 1), we used a weighting factor $\lambda$ computed as the
percentage of positive values in $\mathbf{S}_{ij}$. The lower $\lambda$ is,
the more we put emphasis on positive ($\mathbf{S}_{ij}$=1) examples:

$\mathcal{L}^{\theta}_{SSM}\\!=\\!-\\!\frac{1}{T^{2}}\\!\sum_{i,j=1}^{T}\\!(1\\!-\lambda)\\!\left[\mathbf{S}_{ij}\\!\log(\\!\mathbf{\hat{S}}_{ij}^{\theta}\\!)\\!\right]\\!+\\!\lambda\\!\left[\\!(\\!1\\!-\\!\mathbf{S}_{ij}\\!)\\!\log\\!(\\!1\\!-\\!\mathbf{\hat{S}}_{ij}^{\theta}\\!)\\!\right]$
(3)

Since the computation of the SSM $\mathbf{\hat{S}}_{ij}^{\theta}$ is
differentiable w.r.t. to the embeddings $\\{\mathbf{e}^{\theta}_{i}\\}$, we
can compute $\frac{\partial\mathcal{L}^{\theta}_{SSM}}{\partial\theta}$:

$\frac{\partial\mathcal{L}^{\theta}_{SSM}}{\partial\theta}=\sum_{i,j=1}^{T}\frac{\partial\mathcal{L}^{\theta}_{SSM}}{\partial\mathbf{\hat{S}}_{ij}^{\theta}}\left(\frac{\partial\mathbf{\hat{S}}_{ij}^{\theta}}{\partial\mathbf{e}^{\theta}_{i}}\frac{\partial\mathbf{e}^{\theta}_{i}}{\partial\theta}+\frac{\partial\mathbf{\hat{S}}_{ij}^{\theta}}{\partial\mathbf{e}^{\theta}_{j}}\frac{\partial\mathbf{e}^{\theta}_{j}}{\partial\theta}\right)$
(4)

We can then use standard gradient-descent algorithms to optimize $\theta$
which will jointly optimize $f^{\theta}$ for all the $\mathbf{X}_{i}$.

Optimizing directly $\mathbf{\hat{S}}_{ij}^{\theta}$ has relationship with
Metric Learning / Contrastive Learning in which the $A$,$P$,$N$ are chosen
based on their similarity (such as in Wang et al. [27]). In comparison, we
consider here simultaneously all possible pairs of time as $A$,$P$,$N$. This
is actually in line with the fact that we aim at learning features relative to
a track (see part 2.4) and we therefore need to consider simultaneously the
interaction between all projections $\\{\mathbf{e}^{\theta}_{i}\\}$.

###  2.3 Novelty-loss

We propose to learn the kernels $\mathbf{K}^{\theta}$ such that when convolved
with the estimated SSM $\mathbf{\hat{S}}_{ij}^{\theta}$ (see eq.(2)) along its
main diagonal the resulting estimated novelty score
$\mathbf{\hat{n}}_{i}^{\theta}$ approximate a ground-truth novelty score
$\mathbf{n}_{i}$. This kernel convolution can be simply implemented as an
extra convolution layer (without bias) on top of the estimated SSM
$\mathbf{\hat{S}}_{ij}^{\theta}$ with a sigmoid output activation. We then
define the novelty-loss as

$\mathcal{L}^{\theta}_{nov}=\frac{1}{T}\sum_{i=1}^{T}BCE(\mathbf{\hat{n}}_{i}^{\theta},\mathbf{n}_{i})$
(5)

###  2.4 Relative feature learning

In previous works dealing with feature learning for MSA it is assumed that,
once trained, the network $f^{\theta}$ always projects a given segment
$\mathbf{X}_{i}$ in the same way whatever its surrounding context.

We advocate here that for the task of MSA the projection of $\mathbf{X}_{i}$
should depend on its context. The motivation for doing so is that the features
that highlight the temporal structure of a music track usually depend on the
track itself. For example, if the instrumentation or the timbre remains
constant over the track, the structure may arise from variation of the
harmonic content; in other cases, it will be the opposite. Therefore, feature
learning for MSA should be made relative-to-a-track.

To let each feature $\mathbf{X}_{i}$ “know” about surrounding times features
$\\{\mathbf{X}_{1}\ldots\mathbf{X}_{i-1},\mathbf{X}_{i+1}\ldots\mathbf{X}_{T}\\}$
we introduce layers of Self-Attention (SA) [35] in our encoder666Note that the
use of the SSM-loss alone does not allows $f^{\theta}$ to encode relative
features; this is the task of the SA..

###  2.5 Network architecture $f^{\theta}$

The architecture of the encoder $f^{\theta}$ is given in Figure 1. It is made
of a succession of 5 consecutive convolution blocks followed by $N$ blocks of
Transformer-Encoder.

Each convolution block is made of a 2D convolution followed by a PReLU [36]
activation and a 2D max-pooling. The kernel size $(k_{f},k_{t})$, the number
of channels $n_{c}$ and pooling size $(p_{f},p_{t})$) of each layer are the
following: layer-1: $(k_{f},k_{t})$=(5,5) $n_{c}$=32 $(p_{f},p_{t})$=(2,2),
layer-2: (5,5) 32 (2,2), layer-3: (5,5) 64 (2,2), layer-4: (5,5) 64 (2,2),
layer-5: (5,2) 128 (5,2). The output of the last convolutional blocks has
dimension (1,1) with $n_{c}$=128 channels and is flattened to a 128-dim
vector.

Each input $\mathbf{X}_{i}$ is independently projected using the convolutional
blocks. These outputs are then considered as a temporal sequence which is fed
to $N$ blocks of Transformer Encoder (each made up of a SA layer with 8 heads,
skip-connection, a normalization layer and two fully-connected layers with an
internal dimension of 128). The outputs are then passed to a tanh and
L2-normalized. They form a sequence of embeddings
$\\{\mathbf{e}^{\theta}_{i}\\}_{i\in\\{1\ldots T\\}}$ with
$\mathbf{e}^{\theta}_{i}\in R^{128}$ which are used to compute
$\mathbf{\hat{S}}_{ij}^{\theta}$.

The size of the kernels $\mathbf{K}^{\theta}$ is fixed to (41,41) which
roughly corresponds to 20s. The kernels $\mathbf{K}^{\theta}$ are either
initialized randomly or initialized with checkerboard kernels similar to the
ones of [32]. In this case, checkerboard kernels have the same size (41,41)
but are damped with Gaussian function with different $\sigma$ (randomly chosen
in the range $[3s,5s]$). We used 3 different kernels $\mathbf{K}^{\theta}$
which are then combined using (1x1) convolution. The diagonal of the resulting
feature-map then goes to a sigmoid activation and is considered as the
estimated novelty $\mathbf{\hat{n}}_{i}^{\theta}$.

Our architecture remains lightweight with a number of parameters ranging from
268K to 567K depending on the number of Transformer Encoder blocks (from $N$=0
to 3).

###  2.6 Training.

We train our network by minimizing jointly the two losses defined by eq. (3)
and eq. (5):

$\mathcal{L}^{\theta}=\alpha\mathcal{L}^{\theta}_{SSM}+(1-\alpha)\mathcal{L}^{\theta}_{nov}$
(6)

We used the ADAM optimizer with a learning rate of 0.001, used early-stopping
monitoring $\mathcal{L}^{\theta}$ on the validation data with a patience of 50
and a maximum of 500 epochs.

Considering that we need the whole sequence of embeddings
$\\{\mathbf{e}^{\theta}_{i}\\}$ of a track to compute
$\mathbf{\hat{S}}_{ij}^{\theta}$ and get the gradients
$\frac{\partial\mathcal{L}^{\theta}}{\partial\theta}$, the mini-batch-size $m$
is here defined as the number of tracks. We used a value of $m$=10 tracks.

####  2.6.1 Generating ground-truth for training

Ground-truth SSM $\mathbf{S}_{ij}$. The ground-truth SSM, $\mathbf{S}_{ij}$,
is constructed using annotated segments (start and end time) and their
associated labels. We rely on the homogeneity assumption, i.e. we suppose that
all times $t_{i}$ that fall within a segment are identical since they share
the same label. If we denote by $\text{seg}(t_{i})$ the segment $t_{i}$
belongs to and by $\text{label}(\text{seg}(t_{i}))$ its label, we assign the
value $\mathbf{S}_{ij}\\!=\\!1$ if
$\text{label}(\text{seg}(t_{i}))=\text{label}(\text{seg}(t_{j}))$ and $0$
otherwise. Note that we could relax this identity constraint to allow
representing similarity between labels (for example using an edit distance
between labels). This is for example important for RWC-POP dataset, where
labels denotes some proximities (verse A and verse B) but are here considered
as different. Also, it could be important to consider the case of non-
homogeneity of the repetitions and create a ground-truth $\mathbf{S}_{ij}$
made of “sub-diagonals” rather than “blocks”.

Ground-truth novelty score $\mathbf{n}_{i}$. The ground-truth novelty score,
$\mathbf{n}_{i}$, is also constructed using the annotated segments (start and
end time). We set $\mathbf{n}_{i}$ to 1 when segment changes at time $i$, 0
otherwise. As proposed by [37] we smooth over time the boundary annotations by
applying a low-pass filter with a triangular-shape
$\\{0.25,0.5,1,0.5,0.25\\}$.

##  3 Evaluation

We assess here the performance of our proposal using various test sets,
compare it to previously published results, conduct an ablation study, and
illustrate its results.

###  3.1 Datasets

For training we used a subset of 693 tracks from the Harmonix dataset
[38]777Given the non-accessibility of Harmonix audio, those have been
downloaded from YouTube and re-annotation has been necessary because of non-
synchronicity of the original annotations. and the 298 tracks of the
Isophonics dataset [39]. For testing we used

  * •

RWC-Pop-AIST the 100 tracks of the RWC-Pop [40] with AIST annotations [41] and
the following three subsets of the SALAMI [3] dataset:

  * •

SA-Pop is the subset of SALAMI tracks with CLASS equal to Popular,

  * •

SA-IA is the subset of SALAMI tracks with SOURCE equal to IA (Internet
Archive),

  * •

SA-Two is the subset of SALAMI tracks with at least two annotations.

For each SALAMI subset we considered the two annotations (An1, An2) and the
two levels of flat annotations (Upper, Lower); those correspond to the files
`textfile{1,2}_{upper,lowercase}.txt`.

Datasets | T | S | L | S | L
---|---|---|---|---|---
Harmonix | 693 | 13 | 17.15 |  |
Isophonics | 298 | 11 | 15.98 |  |
RWC-Pop-AIST | 100 | 17 | 14.31 |  |
|  | Upper |  | Lower |
SA-Pop (An1) | 276 | 12 | 15.49 | 30 | 5.73
SA-Pop (An2) | 175 | 12 | 14.64 | 31 | 5.67
SA-IA (An1) | 444 | 14 | 18.32 | 50 | 4.43
SA-IA (An2) | 244 | 12.5 | 18.67 | 37 | 7.00
SA-Two (An1) | 882 | 11 | 18.25 | 30 | 6.89
SA-Two (An2) | 882 | 11 | 17.76 | 31 | 6.39
Table 1: Description of the datasets used in our evaluation: number of tracks
$T$, median value of the number of segments per track $S$ , median value of
segment duration $L$ in seconds (note that [29] indicate $L$ in number of
beats).

In Table 1 we describe these datasets. According to the values of $L$ our
training-sets better match the Upper annotations than the Lower ones of
SALAMI. Also, the size of our kernels $\mathbf{K}^{\theta}$ (roughly 20s., see
part 2.5) assumes homogeneous segments of roughly 10s. and are therefore
closest to the $L$ of Upper annotations.

| RWC-Pop-AIST | SA-Pop | SA-IA | SA-Two |
---|---|---|---|---|---
| HR.5F | HR3F | HR.5F | HR3F | HR.5F | HR3F | HR.5F | HR3F | Annotation
Grill [23, 42] GS1 | .506 | .715 | - | - | - | - | .541 | .623 | Up./An-*
McCallum [25] Unsynch. | - | - | - | - | - | .497 | - | - |
Beat-synch. | - | - | - | - | - | .535 | - | - |
Salamon [30] $\text{DEF}_{0\\!.5\\!,0\\!.\\!5}$/$*_{\mu\\!^{H},\\!\gamma\\!^{H}}$ | - | - | - | - | - | - | .337 | .563 | Up./An-*
Wang [27] scluster/D/eu/mul | .438 | .653 | .447 | .623 | - | - | .356 | .553 | Up./An-*
Buisson [29] $\text{HE}_{0}$/$\text{HE}_{1}$ | - | .681 | - | - | - | - | - | .597 / .595 | Up./An-1/2
|  |  | - | - | - | - | - | .611 / .600 | Low./An-1/2
Ours (best conf.) | .399 | .713 | .298 / .295 | .631/ .624 | .250 / .261 | .520 / .511 | .231 / .237 | .521 / .530 | Up./An-1/2
|  |  | .296 / .318 | .570 / .610 | .302 / .336 | .547 / .612 | .287 / .287 | .589 / .589 | Low./An-1/2
Ablation study $N$ |  |  |  |  |  |  |  |  |
N=3/$\alpha$=0.5/K:train-Init:chck |  | .713 |  | .532 |  | .472 |  | .448 | Up./An-1
N=2/$\alpha$=0.5/K:train-Init:chck |  | .701 |  | .535 |  | .474 |  | .449 | Up./An-1
N=1/$\alpha$=0.5/K:train-Init:chck |  | .677 |  | .631 |  | .520 |  | .521 | Up./An1
N=0/$\alpha$=0.5/K:train-Init:chck |  | .696 |  | .535 |  | .459 |  | .443 | Up./An-1
Ablation study $\alpha$ |  |  |  |  |  |  |  |  |
N=3/$\alpha$=1/K:train-Init:chck |  | .154 |  | .121 |  | .102 |  | .111 | Up./An-1
N=3/$\alpha$=0/K:train-Init:chck |  | .007 |  | .120 |  | .026 |  | .095 | Up./An-1
Ablation study $\mathbf{K}^{\theta}$ |  |  |  |  |  |  |  |  |
N=3/$\alpha$=0.5/K:train-Init:randn |  | .713 |  | .543 |  | .470 |  | .457 | Up./An-1
N=1/$\alpha$=0.5/K:train-Init:randn |  | .709 |  | .547 |  | .470 |  | .457 | Up./An-1
N=3/$\alpha$=0.5/K:fix-Init:chck |  | .330 |  | .250 |  | .199 |  | .196 | Up./An-1
Table 2: Results of segment boundary detection using various test-sets and
configurations

###  3.2 Segment detection from novelty score

To get the estimated segment boundaries from the estimated novelty score
$\mathbf{\hat{n}}_{i}^{\theta}$ we used a simple peak-to-mean ratio algorithm
similar to [25]. Using the same notations as [25] eq. (5), we compute the mean
with a window of duration $2T$=20s, and then detect local peaks with a
threshold $\tau$=1.35 and a minimum inter-distance of 7s.

###  3.3 Performance metrics

We evaluate the performance of segment boundary detection using the common
Hit-Rate metrics using precision-windows of 3s and 0.5s. We only display here
the Hit-Rate F-measures denoted by HR3F  and HR0.5F. We used `mir_eval` [43]
with `mir_eval.segment.detection` ignoring track start and end annotations
(`Trim=True`). We point out that without “trimming” (the start and end time)
we would gain +3% on average (from .713 to .743 for RWC-Pop).

###  3.4 Comparison with previous works

In the following we will compare our results with the ones previously
published by Grill and Schlüter in [23, 42], McCallum et al. in [25], Salamon
et al. in [30], Wang et al. in [27] and Buisson et al. in [29]. We first check
if their test-sets match ours.

For SA-Pop, Wang [27] used “a subset with 445 annotated songs (from 274 unique
songs) in the “popular” genre”. This roughly matches our SA-Pop (An1)+(An2)
which provides 276+175=451 annotations. They used the Upper-case annotations
(personal communication).

For SA-IA, McCallum [25] used “the internet archive portion of the SALAMI
dataset (SALAMI-IA) consisting of 375 hand annotated recordings”. This is much
lower than our SA-IA (An1)+(An2) which provides 444+244=688 annotations.
Moreover, it is not clear whether they used the Upper, Lower or Functional
annotations.

Finally, for SA-Two, Salamon [30] Table 3 used the Upper-case annotations of
tracks with at least 2 annotations (884 tracks); Wang et al. [27] “we treat
each annotation of a song separately, yielding 2243 annotated songs in total”
and Buisson et al. [29] used the Upper and Lower-case annotations of tracks
with at least 2 annotations (884 tracks). This roughly corresponds to our SA-
Pop (An1)+(An2) which has 882 tracks.

###  3.5 Results and discussions

Results are given in Table 2. The upper part shows previously published
results, although not all systems were evaluated on all test sets. The middle
part shows the results achieved with the best configuration of our system.

For RWC-Pop-AIST, we obtained a HR3F= .713888The Precision and Recall at
3seconds are P3F=.735, R3F=0.715 which is comparable to those of Grill and
Schlüter (.715). However, for HR.5F our results are below (.399 < .506). This
can be explained by the fact that the hop-size of our
features$\\{\mathbf{X}_{i}\\}$ was chosen large (0.5s) and does not allow to
have a precise boundary estimation. We have chosen a large hop size to reduce
the size of $\mathbf{\hat{S}}_{ij}^{\theta}$ (hence the computation time and
memory footprints); it also allows to keep the size of the
$\mathbf{K}^{\theta}$ manageable. Because of this, all our results with HR.5F
are actually low. Therefore, we only discuss the results for HR3F in the
following.

For SA-Pop, we obtained a HR3F of .631/.624999P3F=.581, R3F=0.760/ P3F=.566,
R3F=0.771 $\rightarrow$ over-segmentation for the two Upper annotations
(Up./An-1/2) which is slightly above those of Wang et al. (.623). For the two
lower annotations (Low./An-1/2) we get a HR3F of .570/.610101010P3F=.860,
R3F=0.468/ P3F=.877, R3F=0.497 $\rightarrow$ under-segment.. Wang et al. does
not provide these results.

For SA-IA, we obtained a HR3F of .520/.511111111P3F=.435, R3F=0.718/ P3F=.411,
R3F=0.751 $\rightarrow$ over-segment. for the two Upper annotations and
.547/.612121212P3F=.811, R3F=0.451/ P3F=.756, R3F=0.546 $\rightarrow$ under-
segment. for the two Lower annotations. This has to be compared to the .497
(unsynchronized) and .535 (beat-synchronized) obtained by McCallum et al., but
as explained, it is not clear whether they used Upper, Lower or Functional
annotations.

For SA-Two, we obtained a HR3F of .521/.530131313P3F=.433, R3F=0.749/
P3F=.442, R3F=0.754$\rightarrow$ over-segment. for the two Upper annotations.
This is slightly lower than the results of Wang et al. (.553), Salamon et al.
(.563), Buisson et al. (.597) and largely below the ones of Grill and Schlüter
(.623). For the Lower annotations, we obtained a HR3F of
.589/.589141414P3F=.768, R3F=0.523/ P3F=.768, R3F=0.523$\rightarrow$ under-
segment. which is slightly below the ones of Buisson et al. (.611). It should
be noted however that in our work we didn’t used any data from SALAMI, neither
for training or validation (such as early stopping).

For SA-IA and SA-two, our results are higher for the Lower annotations than
the Upper ones. This is surprising since according to Table 1 the
characteristics ($L$ value) of our training sets are closer to the Upper case.
Also (see footnotes 8–14), our algorithm tends to over-segment when
considering the Upper annotation and under-segment when considering the Lower
ones. Our kernel size is actually between the $L$ values of the Upper and
Lower annotations.

###  3.6 Ablation study

In the lower part of Table 2 we perform an ablation study of our system. For
the SA-{Pop,IA,Two} test-sets, we only perform the study using the Upper/An1
annotations

We first check the optimal number $N\in\\{0,1,2,3\\}$ of layers of Transformer
Encoder. We see that for all test-sets the use of Transformer Encoder ($N>0$)
is beneficial. For RWC-Pop-AIST, the optimal number is $N$=3 while for all
three SA-{Pop,IA,Two} test-sets it is always $N$=1.

We then check whether jointly optimizing the two losses $L^{\theta}_{SSM}$ and
$\mathcal{L}^{\theta}_{nov}$ of eq. (6) is necessary. We considered three
cases: $\alpha$=1 (only optimizing $L^{\theta}_{SSM}$), $\alpha$=0.5
(optimizing both), $\alpha$=0 (only optimizing $\mathcal{L}^{\theta}_{nov}$).
For all test-sets, we see that optimizing jointly the two losses is highly
beneficial: for example, for RWC-Pop-AIST, HR3F=.713 with $\alpha$=0.5, .154
with $\alpha$=1 and .007 for $\alpha$=0.

Finally, we check various configurations of the kernels $\mathbf{K}^{\theta}$.
$\mathbf{K}^{\theta}$ is either [K:train-Init:chck]: trained starting from
checkerboard kernels initialisation, [K:train-Init:randn]: trained starting
from random initialisations, [K:fix-Init:chck]: fixed (not trained) to
checkerboard kernels (we still train the 1x1 convolution). We see that for all
test-sets it is beneficial to train $\mathbf{K}^{\theta}$ (the worst results
are obtained with [K:fix-Init:chck]). For RWC-Pop-AIST, the results are the
same whether kernels are initialized randomly or with checkerboard kernels.
For SA-{Pop,IA,Two} the checkerboard kernels initialization is beneficial.

###  3.7 Examples

Figure 2 illustrates the three kernels $\mathbf{K}^{\theta}$ learned using the
[N=3/$\alpha$=0.5/K:train-Init:chck] configuration. As one can see, while the
middle one looks close to the classical checkerboard kernel of Foote [32] (but
with an emphasis on the diagonal), the first seems to highlight the transition
from a non-homogeneous to an homogeneous part; while the third seems a re-
scaled version of the second (homogeneity at a larger scale). Figure 3
illustrates the $\mathbf{\hat{S}}_{ij}^{\theta}$ and
$\mathbf{\hat{n}}_{i}^{\theta}$ obtained by our system on track-01 from RWC-
Pop-AIST (chosen as the first item of our test-set). We compare the results
when trained in the [N=3 / $\alpha$=0.5 / K:train-Init:chck] configuration and
with the untrained system using [K:fix-Init:chck] for the kernels. For
comparison we indicate the ground-truth $\mathbf{S}_{ij}$ and
$\mathbf{n}_{i}$. In this figure, the benefits of training both
$L^{\theta}_{SSM}$ and$\mathcal{L}^{\theta}_{nov}$ appears clearly.

##### Reproducibility.

The code and the datasets used in this work are available at:
https://github.com/geoffroypeeters/ssmnet_ISMIR2023

Figure 2: The three kernels $\mathbf{K}^{\theta}$ learned using the [N=3 /
$\alpha$=0.5 / K:train-Init:chck] configuration.

Figure 3: [Top] $\mathbf{\hat{S}}_{ij}^{\theta}$ and
$\mathbf{\hat{n}}_{i}^{\theta}$ obtained with untrained system using [K:fix-
Init:chck] for the kernels, [Middle] same with [N=3 / $\alpha$=0.5 / K:train-
Init:chck], [Bottom] ground-truth $\mathbf{S}_{ij}$ and $\mathbf{n}_{i}$.

##  4 Conclusion

In this work, we proposed a simple approach for deep learning-based Music
Structure Analysis algorithm: we learn an encoder $f^{\theta}$ such that the
resulting learned features allow to best approximate a ground-truth SSM; we
jointly learn segmentation kernels that when applied to the estimated SSM we
best approximate a ground-truth novelty score. We also propose to learn
relative features, i.e. features relative to a track, by introducing Self-
Attention layers in our encoder. According to HR3F, our results are either
better than previous state-of-the-art (SA-Pop, SA-IA unsynchronous), similar
(RWC-Pop-AIST) or worst (SA-Two). Our approach has the advantage to be
lightweight (around 500K parameters) and based on criteria which are
semantically linked to the task of MSA. Future works will concentrate on
making our approach applicable to finer temporal resolutions, therefore
allowing improving our performances at HR.5F.

## References

  * [1] J. Foote, “Visualizing music and audio using self-similarity,” in _Proc. of ACM Multimedia_ , Orlando, Florida, USA, 1999, pp. 77–80.
  * [2] G. Peeters and E. Deruty, “Is music structure annotation multi-dimensional ? a proposal for robust local music annotation,” in _Proc. of LSAS (International Workshop on Learning the Semantics of Audio Signals)_ , Graz, Austria, 2009.
  * [3] J. B. L. Smith, J. Burgoyne, I. Fujinaga, D. Roure, and J. S. Downie, “Design and creation of a large-scale database of structural annotations,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Miami, Florida, USA, 2011.
  * [4] B. McFee, O. Nieto, M. M. Farbood, and J. P. Bello, “Evaluating hierarchical structure in music annotations,” _Frontiers in Psychology_ , vol. 8, p. 1337, 2017.
  * [5] M. Bruderer, M. McKinney, and A. Kohlrausch, “Structural boundary perception in popular music,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Victoria, Canada, 2006.
  * [6] G. Peeters, A. Laburthe, and X. Rodet, “Toward automatic music audio summary generation from signal analysis,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Paris, France, 2002, pp. 94–100.
  * [7] G. Peeters, D. Fenech, and X. Rodet, “MCIpa: A music content information player and annotator for discovering music,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Philadelphia, PA, USA, 2008.
  * [8] G. Peeters, F. Cornu, D. Tardieu, C. Charbuillet, J. J. Burred, M. Ramona, M. Vian, V. Botherel, J.-B. Rault, and J.-P. Cabanal, “A multimedia search and navigation prototype, including music and video-clips,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Porto, Portugal, October 2012.
  * [9] M. Goto, K. Yoshii, H. Fujihara, M. Mauch, and T. Nakano, “Songle: A web service for active music listening improved by user contributions.” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Miami, Florida, USA, 2011.
  * [10] M. Mueller and N. Jiang, “A scape plot representation for visualizing repetitive structures of music recordings,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Porto, Portugal, 2012\.
  * [11] M. Mauch, K. Noland, and S. Dixon, “Using musical structure to enhance automatic chord transcription,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Kobe, Japan, 2009.
  * [12] Z. Rafii and B. Pardo, “Repeating pattern extraction technique (repet): A simple method for music/voice separation,” _Audio, Speech and Language Processing, IEEE Transactions on_ , vol. 21, no. 1, pp. 73–84, January 2013.
  * [13] M. Fuentes, B. McFee, H. C. Crayencour, S. Essid, and J. P. Bello, “A music structure informed downbeat tracking system using skip-chain conditional random fields and deep learning,” in _Proc. of IEEE ICASSP (International Conference on Acoustics, Speech, and Signal Processing)_ , Brighton, UK, 2019, pp. 481–485.
  * [14] J. Paulus, M. Müller, and A. Klapuri, “Audio-based music structure analysis,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Utrecht, The Netherlands, 2010.
  * [15] G. Sargent, F. Bimbot, and E. Vincent, “A regularity-constrained viterbi algorithm and its application to the structural segmentation of songs,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Miami, Florida, USA, 2011.
  * [16] O. Nieto, G. J. Mysore, C.-i. Wang, J. B. Smith, J. Schlüter, T. Grill, and B. McFee, “Audio-based music structure analysis: Current trends, open challenges, and applications,” _Transactions of the International Society for Music Information Retrieval_ , vol. 3, no. 1, 2020.
  * [17] J. Foote, “Automatic audio segmentation using a measure of audio novelty,” in _Proc. of IEEE ICME (International Conference on Multimedia and Expo)_ , New York City, NY, USA, 2000.
  * [18] F. Kaiser and T. Sikora, “Music structure discovery in popular music using non-negative matrix,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Utrecht, The Netherlands, 2010.
  * [19] B. McFee and D. P. W. Ellis, “Learning to segment songs with ordinal linear discriminant analysis,” in _Proc. of IEEE ICASSP (International Conference on Acoustics, Speech, and Signal Processing)_ , Florence, Italy, 2014\.
  * [20] M. Müller, N. Jiang, and P. Grosche, “A robust fitness measure for capturing repetitions in music recordings with applications to audio thumbnailing,” _Audio, Speech and Language Processing, IEEE Transactions on_ , vol. 21, no. 3, pp. 531–543, 2013.
  * [21] J. Serrà, M. Müller, P. Grosche, and J. L. Arcos, “Unsupervised Music Structure Annotation by Time Series Structure Features and Segment Similarity,” _IEEE Transactions on Multimedia_ , vol. 16, no. 5, pp. 1229–1240, Aug. 2014.
  * [22] K. Ullrich, J. Schlüter, and T. Grill, “Boundary Detection in Music Structure Analysis using Convolutional Neural Networks,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Taipei, Taiwan, 2014.
  * [23] T. Grill and J. Schlüter, “Music Boundary Detection Using Neural Networks on Combined Features and Two-Level Annotations,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Malaga, Spain, 2015.
  * [24] A. Cohen-Hadria and G. Peeters, “Music Structure Boundaries Estimation Using Multiple Self-Similarity Matrices as Input Depth of Convolutional Neural Networks,” in _AES International Conference on Semantic Audio_ , Erlangen, Germany, June, 22–24, 2017.
  * [25] M. C. McCallum, “Unsupervised Learning of Deep Features for Music Segmentation,” in _Proc. of IEEE ICASSP (International Conference on Acoustics, Speech, and Signal Processing)_ , Brighton, UK, May 2019.
  * [26] F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2015, pp. 815–823, iSSN: 1063-6919.
  * [27] J.-C. Wang, J. B. L. Smith, W.-T. Lu, and X. Song, “Supervised metric learning for music structure features,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Online, November, 8–12 2021.
  * [28] J.-C. Wang, Y.-N. Hung, and J. B. L. Smith, “To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functions.” in _Proc. of IEEE ICASSP (International Conference on Acoustics, Speech, and Signal Processing)_ , Virtual and Singapore, May 2022.
  * [29] M. Buisson, B. McFee, S. Essid, and H.-C. Crayencour, “Learning multi-level representations for hierarchical music structure analysis,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , 2022.
  * [30] J. Salamon, O. Nieto, and N. J. Bryan, “Deep embeddings and section fusion improve music segmentation,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Online, November, 8–12 2021.
  * [31] G. Peeters and F. Angulo, “Ssm-net: Feature learning for music structure analysis using a self-similarity-matrix based loss,” in _Late-Breaking/Demo Session of ISMIR (International Society for Music Information Retrieval)_ , 2022.
  * [32] J. Foote, “Automatic audio segmentation using a measure of audio novelty,” in _Proc. of IEEE ICME (International Conference on Multimedia and Expo)_ , New York City, NY, USA, 2000, pp. 452–455.
  * [33] F. Kaiser and G. Peeters, “Multiple hypotheses at multiple scales for audio novelty computation within music,” in _Proc. of IEEE ICASSP (International Conference on Acoustics, Speech, and Signal Processing)_ , Vancouver, British Columbia, Canada, May 2013.
  * [34] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” in _Proceedings of the 14th python in science conference_ , vol. 8, 2015.
  * [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [36] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-normalizing neural networks,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, pp. 972–981.
  * [37] J. Schlüter and S. Böck, “Improved musical onset detection with Convolutional Neural Networks,” in _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2014, pp. 6979–6983, iSSN: 2379-190X.
  * [38] O. Nieto, M. McCallum, M. E. P. Davies, A. Robertson, A. Stark, and E. Egozy, “The Harmonix Set: Beats, Downbeats, and Functional Segment Annotations of Western Popular Music,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Delft, The Netherlands, 2019.
  * [39] M. Mauch, C. Cannam, M. Davies, S. Dixon, C. Harte, S. Klozali, D. Tidhar, and M. Sandler, “Omras2 metadata project 2009,” in _Late-Breaking/Demo Session of ISMIR (International Society for Music Information Retrieval)_ , Kobe, Japan, 2009.
  * [40] M. Goto, “Development of the RWC Music Database,” _Proc. of ICA (18th International Congress on Acoustics)_ , 2004.
  * [41] ——, “Aist annotation for the rwc music database,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Victoria, BC, Canada, 2006, pp. pp.359–360.
  * [42] T. Grill and J. Schlüter, “Structural segmentation with convolutional neural networks MIREX submission,” 2015.
  * [43] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis, “mir_eval: A transparent implementation of common mir metrics,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Taipei, Taiwan, 2014.

  *[MSA]: Music Structure Analysis
  *[SSM]: Self-Similarity-Matrix
  *[HC]: hand-crafted
  *[DL]: deep learning
  *[ConvNet]: Convolutional Networks
  *[TL]: Triplet Loss
  *[CQT]: Constant-Q-Transform
  *[SSL]: Self-Supervised-Learning
  *[BCE]: Binary-Cross-Entropy
  *[SA]: Self-Attention


# Remixing Music for Hearing Aids Using Ensemble of Fine-Tuned Source
Separators

###### Abstract

This paper introduces our system submission for the Cadenza ICASSP 2024 Grand
Challenge, which presents the problem of remixing and enhancing music for
hearing aid users. Our system placed first in the challenge, achieving the
best average Hearing-Aid Audio Quality Index (HAAQI) score on the evaluation
data set. We describe the system, which uses an ensemble of deep learning
music source separators that are fine tuned on the challenge data. We
demonstrate the effectiveness of our system through the challenge results and
analyze the importance of different system aspects through ablation studies.

© IEEE 2024. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.
Index Terms—  Cadenza ICASSP 2024 Grand Challenge, hearing aids, music
enhancement, music source separation, remixing

##  1 Introduction

The Cadenza ICASSP 2024 Grand Challenge [1] presents a machine learning
challenge to enhance music signals for hearing aid users. The goal of the
challenge is to improve music processing for hearing aids, since hearing loss
and hearing aid processing can negatively affect the music listening
experience.

The task issued by the challenge is to design a system to remix a music signal
for a listener according to their specific preferences and hearing loss
characteristics. The inputs to the system include an audio signal containing
music and the listener-specified gains indicating their preference on how to
remix the four component music tracks: vocals, drums, bass, and other (VDBO).
The “other” track refers to any component of the signal not belonging to the
other three tracks. The system is also provided with the listener’s audiogram,
which specifies their characteristic hearing loss at different frequencies.

The system is evaluated by comparing the enhanced music signal with a ground-
truth enhanced signal constructed using the true separate music tracks. This
comparison uses the Hearing-Aid Audio Quality Index (HAAQI) [2], which
objectively measures the perceived audio quality of the enhanced signal given
the ground-truth reference and the listener’s specific hearing loss as
characterized by their audiogram.

The Cadenza challenge provides a data set for training, a validation data set,
and a test data set for the final evaluation. The data includes songs, sets of
gains for the VDBO components, and listener audiograms. An important aspect of
the data is that the audio signals are convolved with head-related transfer
functions (HRTFs) to simulate a person listening to stereo speakers in a free
field with different head positions. This introduces crosstalk to the stereo
channels, meaning each received stereo channel contains a combination of the
separate transmitted stereo channels, depending on the head position relative
to the speakers.

The provided baseline system for the challenge uses the Hybrid Demucs
(HDemucs) model [3], trained on the MUSDB18-HQ data set [4], to separate the
VDBO tracks. The baseline remixes the music by adjusting the tracks to the
desired gains and summing them together. The baseline normalizes the signal to
match the loudness of the input and then performs NAL-R hearing aid
amplification [5] given the listener’s audiogram.

We propose a music enhancement system for the challenge that follows the
baseline’s basic structure, while focusing on improving the music source
separation (MSS) aspect. A variety of deep neural network architectures have
demonstrated strong results on the MSS problem [3, 6, 7, 8], and we use an
ensemble of three different architectures, each fine tuned on the Cadenza
challenge data with crosstalk.

Our proposed ensemble system placed first in the Cadenza ICASSP 2024 Grand
Challenge, with the best average HAAQI score on the evaluation data set. This
paper describes the system and presents evaluation results demonstrating its
effectiveness in the challenge. We analyze the results and use ablation
studies to measure the importance of different system components.

##  2 System Design

Our system follows the baseline’s structure by using an MSS model to split the
music into VDBO tracks, adjusting the tracks to the desired gains, summing the
tracks together, performing loudness normalization, and applying NAL-R
amplification. We mainly focus on improving the MSS step by using an ensemble
of three different deep neural network architectures that have achieved state-
of-the-art results on the MUSDB18-HQ data set. The model architectures are
HDemucs [3], KUIELab-MDX-Net [7], and DTTNet [6]. We simply average together
the VDBO signals from each of the models in the ensemble.

The KUIELab-MDX-Net and DTTNet architectures are both based on a Time-
Frequency Convolutions with Time-Distributed Fully-connected networks UNet
(TFC-TDF UNet) architecture [9]. Both of these architectures require a
separate model trained for each VDBO track. Each KUIELab-MDX-Net model
contains 7.4 million parameters, and each DTTNet model contains 5 million
parameters. The HDemucs model uses a convolutional encoder-decoder
architecture that operates on both frequency-domain and time-domain
representations. The HDemucs architecture requires a single model to separate
the four VDBO tracks, and it contains 83.6 million parameters.

For the models in the MSS ensemble, we started with pretrained checkpoints
from the official implementations, which were all trained on MUSDB18-HQ. We
then fine tuned the checkpoints on the Cadenza data, where the key difference
from MUSDB18-HQ is the inclusion of HRTFs to introduce crosstalk between the
stereo channels. We trained for a maximum of 10 epochs using the same
hyperparameters as the original implementations, except for reduced learning
rates. We performed early stopping by selecting model checkpoints with the
highest average signal-to-distortion ratio (SDR) on the validation set.

For the DTTNet and KUIELab-MDX-Net models we selected salient segments from
the training set using the source activity detection algorithm mentioned in
[8].

Based on our experiments with the validation set, we found it useful to
calculate the residual signal by subtracting the predicted vocals, drums, and
bass tracks from the original music signal. We then incorporate this residual
into the final “other” track prediction at inference time by averaging it
together with the “other” track predicted by the model.

We also try to reduce distortion to the signals introduced by clipping the
NAL-R amplified signals by applying an audio compressor to reduce the dynamic
range of the signal. We use a rough heuristic to apply the compressor only if
we detect 25 thousand samples to be clipped in one of the stereo channels.

##  3 Results

Table 1 displays the average HAAQI score of our system on the evaluation set,
alongside the scores of the individual components of the ensemble, the HDemucs
baseline system, and the systems of the next two highest scoring teams in the
challenge, T003 and T011. Our ensemble system obtained the best score out of
all systems submitted to the challenge.

System | HAAQI Score
---|---
Baseline (HDemucs) | 0.5697
Team T011 submission | 0.5857
Team T003 submission | 0.5929
Fine-tuned HDemucs | 0.6027
Fine-tuned KUIELab-MDX-Net | 0.6001
Fine-tuned DTTNet | 0.6166
Ensemble | 0.6317
Table 1: HAAQI scores of systems on evaluation set.

###  3.1 Ablation Studies

Table 2 shows the results of our ablation studies, where we tested the effect
of incorporating the residual into the prediction of the “other” track. We
also tested the effect of applying the audio compressor. All other aspects of
the ensemble system were kept constant while removing these components for the
ablation studies. We can see both of these ablations reduced performance.

Ablation | HAAQI Score
---|---
Ensemble (no ablation) | 0.6317
Remove use of residual | 0.6081
Remove audio compressor | 0.6053
Table 2: HAAQI scores of ablation studies.

##  4 Discussion

The results show how our system was successful in the challenge. We can
compare the fine-tuned HDemucs model system to the baseline system to see that
fine tuning on the data with crosstalk leads to improved performance. We can
also see the benefit of averaging together results in the ensemble. This
suggests the different model architectures are able to complement each other
on the separation task.

The ablation studies also show that the incorporation of the residual signal
and the application of the audio compressor are important factors in the
system’s success.

##  5 Conclusions

We introduced a successful system for the Cadenza ICASSP 2024 Challenge that
uses an ensemble of source separators to remix music and enhance it for
hearing aid users. Our results demonstrate the effectiveness of using an
ensemble, fine tuning pretrained models, incorporating the residual signal,
and using an audio compressor. These components of our system led to it
achieving the highest score and placing first in the challenge.

## References

  * [1] G. Roa Dabike, M. A. Akeroyd, S. Bannister, J. Barker, T. J. Cox, B. Fazenda, J. Firth, S. Graetzer, A. Greasley, R. R. Vos, and W. M. Whitmer, “Overview and results of the ICASSP SP cadenza challenge: Music demixing/remixing for hearing aids,” in _2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2024.
  * [2] J. M. Kates and K. H. Arehart, “The hearing-aid audio quality index (HAAQI),” _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 24, no. 2, pp. 354–365, 2015.
  * [3] A. Défossez, “Hybrid spectrogram and waveform source separation,” in _Proceedings of the ISMIR 2021 Workshop on Music Source Separation_ , 2021.
  * [4] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “MUSDB18-HQ - an uncompressed version of musdb18,” Dec. 2019. [Online]. Available: https://doi.org/10.5281/zenodo.3338373
  * [5] D. Byrne and H. Dillon, “The national acoustic laboratories’(NAL) new procedure for selecting the gain and frequency response of a hearing aid,” _Ear and hearing_ , vol. 7, no. 4, pp. 257–265, 1986.
  * [6] J. Chen, S. Vekkot, and P. Shukla, “Music source separation based on a lightweight deep learning framework (DTTNET: Dual-path TFC-TDF UNet),” 2023. [Online]. Available: https://arxiv.org/abs/2309.08684
  * [7] M. Kim, W. Choi, J. Chung, D. Lee, and S. Jung, “KUIELab-MDX-Net: A two-stream neural network for music demixing,” 2021. [Online]. Available: https://arxiv.org/abs/2111.12203
  * [8] Y. Luo and J. Yu, “Music source separation with band-split rnn,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 1893–1901, 2023.
  * [9] W. Choi, M. Kim, J. Chung, D. Lee, and S. Jung, “Investigating u-nets with various intermediate blocks for spectrogram-based singing voice separation.” in _21th International Society for Music Information Retrieval Conference_ , ISMIR, Ed., OCTOBER 2020.