# Understanding Adverse Biological Effect Predictions Using Knowledge Graphs

Erik B. Myklebust  Ernesto Jiménez-Ruiz  Jiaoyan Chen
Raoul Wolf  Knut Erik Tollefsen  Norwegian Institute for Water Research
(NIVA), Oslo, Norway  NORSAR, Kjeller, Norway  SIRIUS, University of Oslo,
Oslo, Norway  City, University of London, London, United Kingdom  University
of Oxford, Oxford, United Kingdom  Norwegian Geotechnical Institute (NGI),
Oslo, Norway  Norwegian University of Life Sciences (NMBU), Ås, Norway

###### Abstract

Extrapolation of adverse biological (toxic) effects of chemicals is an
important contribution to expand available hazard data in (eco)toxicology
without the use of animals in laboratory experiments. In this work, we
extrapolate effects based on a knowledge graph (KG) consisting of the most
relevant effect data as domain-specific background knowledge. An effect
prediction model, with and without background knowledge, was used to predict
mean adverse biological effect concentration of chemicals as a prototypical
type of stressors. The background knowledge improves the model prediction
performance by up to 40% in terms of $R^{2}$ (i.e., coefficient of
determination). We use the KG and KG embeddings to provide quantitative and
qualitative insights into the predictions. These insights are expected to
improve the confidence in effect prediction. Larger scale implementation of
such extrapolation models should be expected to support hazard and risk
assessment, by simplifying and reducing testing needs.

###### keywords:

Adverse effect prediction , Knowledge graph , Explanation

††journal: Computational Toxicology

##  1 Introduction

Extrapolation using already available adverse biological (toxic) effects to
species and stressors, which are not readily available (e.g., new chemicals
and/or species) or are ethically questionable (e.g., vulnerable or endangered
species), is an important development towards more sustainable testing
approaches in (eco)toxicology. According to Animal and Plant Health Inspection
Service [2], Mukerjee [22], 1 million animals are used in laboratory
experiments each year in the United States alone. Moreover, Busquet et al. [7]
puts this number between 10 and 18 million in the European Union.

We introduce a novel approach for extrapolation using a stressor-effect
informed knowledge graph (KG) and its vector embeddings to provide additional
information to the extrapolation models.

To this end, we have created the Toxicological Effect and Risk Assessment KG
(TERA), which integrates key data concerning chemicals and species (e.g.,
categorization) used in laboratory experiments as well as the experiments
themselves [23, 24].

Knowledge graph embedding (KGE) models such as DistMult and TransE [39, 30]
can be applied to TERA to extract low-dimensional vector representations of
its entities. These vector representations enable the use of standard machine
learning prediction models to extrapolate effect data from existing chemical-
species pairs to new combinations [23, 24].

TERA can also help us understand the prediction based both on the vectors and
the structure of the graph itself, e.g., neighbourhoods. These explanations
are very important when using models in downstream tasks (e.g., ecological
risk assessment) to increase the confidence in predictions.

This paper builds on our previous work, in Myklebust et al. [23, 24], by
moving from a strictly (binary) classifications of effects to a prediction of
chemical concentrations for a given effect in a specific organism, i.e., which
concentration is toxic (lethal) to an organism. In this paper we restrict the
data coverage to organic compounds, in order to yield a more consistent data
set. We have also built on the KGE analysis section in Myklebust et al. [24]
to provide insight and interpretations for the predictions. This works
contributes to a larger effort of making effect prediction models explainable.
The work presents the results of gap-filling tasks, one for chemicals and one
for species. We perform chemical-species pair predictions to fill
(synthetically) missing adverse effect data. We show that using knowledge
graph embeddings in this gap-filling prediction task can improve both the
results and the interpretation of a complex model.

The rest of the paper is organized as follows. Section 2 provides relevant
background information on effect extrapolation and introduces the concepts of
KG, KGE, and Explainable AI (XAI). Section 3 provides an overview of the
methods while the specific details are found in A. Section 4 introduces the
metrics used to evaluate the models followed by the evaluation and the
explanations of the predictions of these models. Section 5 discusses the
implications of the results, while Section 6 concludes the work and indicates
some future research directions.

##  2 Background and State-of-the-Art

###  2.1 Effect Extrapolation

In ecological risk assessment, it is not feasible to conduct experiments on
all possible species-chemical combinations ($S\times C$), therefore,
extrapolation models have been developed, such as Quantitative Structure-
Activity Relationship (QSAR) [6], and read-across111Use data from one sub-
domain as predictor for another. methods have been designed, such as [14] that
uses predicted features for chemical hydrophobicity in a linear regression
model. Most of these models consider one (or a small group treated as one)
species and extrapolates on relatively small groups of chemicals. This is to
remove inter-species variability, however, intra- and inter-assay variability
have been shown to be large [4].

This work goes under the (whole organism) read-across methodology where we try
to mitigate its shortcomings. Standard read-across is done by creating
similarity measures (unique to the method), e.g., chemical fingerprint
similarity, among species or chemical to justify the read-across (e.g., Russo
et al. [31]). Read-across is more difficult in the species domain than in the
chemical one due to the lack of defining features such as molecular weight or
partition coefficient, although, genetic similarity has emerged as a viable
option. One of these tools is the Sequence Alignment to Predict Across Species
Susceptibility (SeqAPASS; LaLone et al. [20], Doering et al. [10]). This tool
uses available data on phylogenetic conservation of proteins across different
taxa to indicate the likelihood of the existence of similar biological targets
in other species. It only provides a sequence similarity of a given molecular
target (e.g., a protein) or the ligand-binding or ligand-protein interacting
site, and thus does not directly infer the toxic potency of the chemical for
the whole organism.

###  2.2 Semantic Web technologies and Knowledge Graphs

The main purpose of using Semantic Web technologies in this work is that it
facilitates the integration and access of disparate sources, e.g., tabular and
graph data. These disparate resources are integrated within the TERA knowledge
graph [24].

We follow the RDF-based notion of knowledge graphs [3] which are composed by
relational facts/triples $\left\langle s,p,o\right\rangle$, where $s$
represents a subject (a class or an instance, e.g., Bluegill), $p$ represents
a predicate (a property, e.g., eol:Habitat) and $o$ represents an object (a
class, freshwater biome, an instance or a data value). RDF entities (i.e.,
classes, properties and instances) are represented by an URI (Uniform Resource
Identifier).

RDF-based KGs like TERA enable the use of the available Semantic Web
infrastructure, including reasoning and the SPARQL query language. SPARQL uses
graph patterns as a query, which is a large benefit for hierarchy-centric data
such as taxonomies. RDF and SPARQL are (Semantic) Web standards defined by the
W3C.222https://www.w3.org/standards/semanticweb/

###  2.3 KG Embeddings

A KG is a symbolic representation and could not be easily or directly used in
machine learning which requires sub-symbolic data representation. Therefore,
models for creating KG embeddings have emerged. An embedding is a low
dimensional representation of an element in the KG, with its semantics kept in
the vector space.

Knowledge graph embedding methods aim at reducing the complexity of a KG into
a smaller (vector) space. This can be done in two main ways. The first is
based on walking the KG in some fashion to create sentences and using
established word embedding models, e.g., RDF2Vec [29] and OWL2Vec* [9] use
Word2Vec [21] on the generated sentences. The other methods model the truth of
a triple (i.e., a fact) in the KG by a probability. In this work, we use the
latter, such that the aim of a model is to find a (scoring) function
$S(t)\mapsto\mathbb{R}$ where $S$ is proportional to the probability that a
triple $t$ is true. An example of such a model is TransE [5], which represents
the predicate in a triple as a transformation from the subject to the object
of a triple. Formally, the model minimizes the overall loss of a given set of
triples (or all the existing triples) of the KG, where each triple contributes
to a loss of $||\mathbf{s}+\mathbf{p}-\mathbf{o}||_{2}$,
$\mathbf{s},\mathbf{p},\mathbf{o}\in\mathbb{R}^{k}$ denote the vector
representations of the subject ($s$), predicate ($p$) and object ($o$) of a
triple ($\left\langle s,p,o\right\rangle$), respectively. $k$ is the embedding
dimension.

###  2.4 Explainable AI

Explainable AI (XAI) has recently attracted wide attention in many research
communities (e.g., Vilone and Longo [36, 37], Tiddi et al. [33], Chen et al.
[8]). Models traditionally used in effect prediction are inherently
explainable as they are mostly based on linear regression or decision tree-
based approaches. However, deep learning is encroaching on this and there is a
larger push in the community for the use of data-centric approaches, due to
its better performance [40].

Certain prediction models can be black boxes but by using a knowledge graph we
can provide an explanation to the user of the model [11]. These explanations
can be presented to experts who can verify if a model prediction is consistent
with their expectations.

In this work, we consider steps toward explanations using the knowledge graph
structure both quantitatively and qualitatively, which are explained in the
next section.

##  3 Method

This section introduces the knowledge graph and embeddings along with the
prediction model and methods for gaining insight. Appendix A provides details.

###  3.1 The TERA Knowledge Graph

In Myklebust et al. [23] we introduced the Toxicological Effect and Risk
Assessment Knowledge Graph (TERA) and it consists of three distinct parts:

  1. 1.

Effects sub-KG. This sub-KG contains knowledge from the ECOTOXicology
Knowledgebase (ECOTOX KB) [26], where the original tabular datasets are
transformed into triples.

  2. 2.

Chemical sub-KG. This part comprises chemically related data, which includes
selected data from ChEMBL [17], Chemical Entities of Biological Interest
(ChEBI) [13], Medical Subject Headings (MeSH) [25], and PubChem [18].

  3. 3.

Species sub-KG. This part includes knowledge transformed from two distinct
taxonomies: the National Center for Biotechnology Information (NCBI) Taxonomy
and the Encyclopedia of Life (EOL) traits [27].

The sources mentioned above are disparate and need to be aligned. We used
ontology alignment systems such as LogMap [16, 15] and external sources such
as Wikidata [38], to construct the alignments among sources. The integration
of new sources in to TERA can vary in effort from simple, where exact mappings
exist (e.g., using Wikidata), to a highly manual process if there are
mismatches between sources or multi-level merging is needed (e.g., ECOTOX
species to NCBI Taxonomy).

###  3.2 Knowledge Graph Embedding Model

Based on our previous work in Myklebust et al. [24], we found that the model
ComplEx [34] is best suited to embed TERA and, therefore, we only use this
model in this work. ComplEx performed consistently well over all data sampling
strategies. ComplEx models the probability of a fact as the inner product of
complex vectors
$\mathbf{e}_{s},\mathbf{e}_{p},\mathbf{e}_{o}\in\mathbb{C}^{k}$, i.e.,

$\displaystyle\begin{split}S_{\text{ComplEx}}(s,p,o)&=\left\langle\mathbf{e}_{s},\mathbf{e}_{p},\mathbf{e}_{o}\right\rangle\\\
&=\left\langle\Re(\mathbf{e}_{s})+i\Im(\mathbf{e}_{s}),\Re(\mathbf{e}_{s})\right\rangle\\\
&+\left\langle
i\Im(\mathbf{e}_{s}),\Re(\mathbf{e}_{p})+i\Im(\mathbf{e}_{o})\right\rangle\\\
&=\left\langle\Re(\mathbf{e}_{s}),\Re(\mathbf{e}_{p}),\Re(\mathbf{e}_{o})\right\rangle\\\
&+\left\langle\Im(\mathbf{e}_{s}),\Re(\mathbf{e}_{p}),\Im(\mathbf{e}_{o})\right\rangle\\\
&+\left\langle\Re(\mathbf{e}_{s}),\Im(\mathbf{e}_{p}),\Im(\mathbf{e}_{o})\right\rangle\\\
&+\left\langle\Im(\mathbf{e}_{s}),\Im(\mathbf{e}_{p}),\Re(\mathbf{e}_{o})\right\rangle\end{split}$
(1)

where $i=\sqrt{-1}$ and, $\Re(x)$ and $\Im(x)$ are the real and complex parts
of $x$, respectively. The optimization of this model is described in A.2.

###  3.3 Adverse Effect Data

We focus on the acute mortality of test organisms in this work. We use the
ECOTOX KB which is the largest publicly available collection of
ecotoxicological effect data. This data is integrated into TERA (see details
in Section 3.1) and can be accessed via SPARQL queries. In short, we gather
results with endpoint (experimental steady state) in
$\\{LC_{50},LD_{50},EC_{50}\\}$333Only $EC_{50}$ where effect is mortality.
and experimental duration of $\geq 24h\ \wedge\leq 96h$. See Query 1 in A.1.
Since the effects are acute, the relevance of experimental duration is assumed
to be mitigated. To limit noise in the data, we only consider compound-species
pairs with three or more results, and we use the median (of results for each
compound-species pair) of these as the ground truth.444There is not _one_
truth in this case, but this is a close approximation.

###  3.4 Grouping Chemicals and Species

We group the species and chemicals that are to be used in the gap-filling
task. This is to test the model’s ability of transferring knowledge from one
group to another. The species are grouped according to the divisions in the
taxonomy In this work, we consider the following species groups as defined in
ECOTOX: Fish, Crustaceans, Insects/Spiders, Amphibians, Worms, Invertebrates,
and Molluscs.

The chemical hierarchy is more complicated and we need to choose a different
method. For example, DEET (_N,N-Diethyl-m-toluamide_ , CAS 134-62-3) is both a
pesticide and an organic compound. We (hierarchically) cluster the chemicals
based on the PubChem fingerprints [18]. These fingerprints are defined using
$881$ binary features of a compound.

We fix the number of clusters to 5. This was chosen to retain a certain size
of the smallest cluster (yellow in Figure 1). The clusters can be seen in
Figure 1. Note that, hierarchical clustering is performed in the original
space ($\\{0,1\\}^{881}$) but have been reduced to $\mathbb{R}^{2}$ in the
figure using t-SNE [35] and, therefore, there is some overlap between clusters
in the figure.

Figure 1: 2D representation of organic chemical clusters. Note that, scale of
this figure is irrelevant.

###  3.5 Prediction

The prediction model used in this work is Support Vector Machine (SVM) [32]
with a Radial basis function (RBF) kernel (additional details provided in
A.3). This is chosen as it is robust to large amounts of noise (uncertainty)
as is the case in this work. We use the scikit-learn package [28] and tune the
most important model parameters ($C$ and $\gamma$ will influence
regularization and kernel scaling, respectably) by performing a grid search
with $C\in\\{10^{k},k\in\\{-4,4\\}\\}$ and
$\gamma\in\\{10^{k},k\in\\{-4,4\\}\\}$.

The input to the model is a vector resulting from the concatenation of the
vectors representing the chemical and the species.555Experimental features,
such as experimental duration, organism details, experimental setup etc. can
be added, however, here we focus on only the embeddings The vector
representations are either generated by the KGE models or a random projection
into a vector space of the same size (as defined by in the KGE, we use $400$
in this work). Note that, we train four different initializations of ComplEx
with the best hyperparameters found in Myklebust et al. [24] and with vectors
of size 100; then these vectors are concatenated to obtain vectors of size 400
for each chemical and species entity. This limits the effect that comes from
the random model initialization as this has been shown to have large influence
on KGE models [41]. The output of the prediction model is $-log_{10}(y)$,
where $y$ is the chemical concentration.

###  3.6 Toward KG-driven Insights

Providing insights into predictions is very important in the verification of
model validity. We explore two different ways of generating prediction
insights:

  1. 1.

Neighbourhood density. i.e., how many graph nodes are within a certain
(Euclidean) radius of the prediction species and chemical. We look at both
entity density666The number of entities in the neighbourhood. and data
density.777The number of (experimental) data points in the neighbourhood. This
is relevant as not all entities have associated data.

  2. 2.

Error prediction. Certain properties of the graph, including neighbourhood can
be a potential predictor of the prediction error. We predict the error based
on the (Euclidean) distance (similarity) among chemicals and species using a
Random Forest model.

  3. 3.

Common facts with close entities. We find the closest entities to the
prediction species and chemical based on their embeddings using euclidean
distance (or cosine similarity) and then find common facts among them. This
will give an indication of whether relevant or generic facts influence the
prediction.

The first two are analogous and provide confidence in the prediction, while
the last will give more insights into which facts affect the prediction.
Algorithms describing the three methods are found in A.4.

###  3.7 Workflow

In Figure 2 we show the steps taken to produce the results and explanations
for the chemical effect prediction. These can be summarized as follows:

  1. 1.

TERA is feed into the KGE model which samples positive and negative triples to
use for training.

  2. 2.

After training, the KGE has produced vector representations of the entities
and relations in TERA.

  3. 3.

Using the biological effect data, we extract the vectors for compound and
species relevant for each sample.

  4. 4.

The samples are divided into five subgroups/folds (equal parts) where there
are no overlap in the relevant group (i.e., chemical group or species division
as described in 3.4.).

  5. 5.

The SVM is trained by leaving one fold out and training on the other four.
This is repeated for all five folds.

  6. 6.

This process produces so-called out-of-fold predictions which are used to
evaluate the model. We produce $R^{2}$ and categorical accuracy as described
in Section 4.3.

  7. 7.

Steps 4 to 6 are repeated 100 times with random folds to produce mean and
standard deviation.

  8. 8.

Explanations are produced using the average predictions from the 100 randomly
initialized runs.

Figure 2: Flowchart of steps taken to produce results. The numbers correspond
to the steps taken in Section 3.7.

##  4 Results

We aim at evaluating the usefulness of using knowledge graph (embeddings) for
effect prediction. We evaluate the problem of gap-filling, where sub-sets of
chemicals or species are unknown and need to be predicted.

###  4.1 Challenges

The results presented in this study heavily relies on the quality of effect
data and the completeness of TERA. However, as shown in Figure 3, the standard
deviation of experiments can be rather large. In addition, we know that
currently TERA does not contain all knowledge related to the domain of effect
prediction. This is a limiting factor, however, increasing the coverage of the
domain without a proper validation might reduce performance by including non-
relevant, inconsistent, or noisy information.

###  4.2 Data sampling strategies

The experimental data in ECOTOX contains ambiguous samples. We demonstrate
this by calculating the standard deviation (of log concentration) of the
experimental results for each chemical-species pair (with $\geq 3$ results).
The distribution of this is shown in Figure 3. We see that the majority of
results are within $0.5$ orders of magnitude. However, up to 10% have a
standard deviation larger than one order of magnitude. These large
discrepancies will make some predictions harder than others.

Figure 3: Distribution of standard deviation (log-concentration) of each
organic chemical-species pair.

The data is sampled as a gap-filling problem, i.e., we remove a proportion of
each group from the training data for testing purposes.

The next subsections presents the results for gap-filling species and
chemicals results. The figures present average metrics (next section) and
standard deviation over 100 random permutations of the unknown chemicals or
species.

###  4.3 Metrics

We use two metrics to evaluate the performance of the models, coefficient of
determination ($R^{2}$) and categorical accuracy (CA). $R^{2}$ is defined as

$\displaystyle
R^{2}=1-\frac{\sum_{i}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i}(y_{i}-\overline{y})^{2}}$
(2)

where $y_{i}$ is the true value, $\hat{y}_{i}$ is the predicted value, and
$\overline{y}$ is the mean true value. Note that, this metric is calculated
based on log-normalized concentrations. This limits the effects of outliers.

Categorical accuracy takes into account regularity requirements as defined in
Agency [1] where categories very toxic ($\leq 1mg/L$), toxic ($>1mg/L\
\wedge\leq 10mg/L$), harmful ($>10mg/L\ \wedge\leq 100mg/L$), and maybe
harmful ($>100mg/L$). We define categorical accuracy as

$\displaystyle CA=\frac{1}{N}\sum_{i}\frac{1}{1+|c_{i}-\hat{c}_{i}|},$ (3)

where $c_{i}$ and $\hat{c}_{i}$ is the true and predicted categories,
respectively. $N$ is the number of data points. This metric does not only
value correct predictions but still gives, e.g., $1/2$ to one category wrong
prediction. Note that, due to this property the lower bound of the metric is
$1/(1+|1-4|))=1/4$ given the four categories defined above.

###  4.4 Gap-filling

In this section, we present the results from gap-filling for missing chemical
and species data.

(a) $R^{2}$

(b) Categorical accuracy.

Figure 4: Average and standard deviation of metrics in the species gap-filling
prediction. w/o embedding are results with random projection while w/
embedding indicate the use of KG embeddings.

(a) $R^{2}$

(b) Categorical accuracy.

Figure 5: Average and standard deviation of $R^{2}$, in Figure 5(a), and
Categorical accuracy ($CA$), in Figure 5(b), in chemical gap-filling
prediction. w/o embedding are results with random projection while w/
embedding indicate the use of KG embeddings. Figure 6: Density map of
predictions (corresponds to Figure 4(a)) as a function of number of neighbours
(with effect data) within a graph depth of prediction chemical and species.
The number of predictions in each plot is show as e.g., $\\#=259$. The
neighbourhood depth is shown as e.g., $depth=9$. Note that, the axis are not
necessary equal in all plots.

Figure 4 show the results for species gap-filling while Figure 5 show the
results for chemical gap-filling.

Figure 4(a) show an improvement of $\sim 0.09$ in $R^{2}$ using the knowledge
graph embeddings. The categorical accuracy in Figure 4(b) is also improved,
albeit, not as large change as for $R^{2}$. The relatively low standard
deviation is down to the choice of the SVM as prediction model.

We show the same plots for chemical gap-filling in Figure 5. For this case,
the prediction results without embeddings are equivalent to predicting prior
mean, while embeddings introduce more noise which leads to negative $R^{2}$
value and $0.03$ lower $CA$. This indicates that the chemical gap-filling task
is more difficult, and either the metadata in TERA does not describe chemicals
adequately or the number of training samples needs to be increased, likely
both.

###  4.5 Explanation

In this section, we present several analysis using KGE to explain predictions.

Figure 7 is generated by considering the categorical error of each individual
prediction (i.e., $0$ \- correctly predicted, $1$ \- one category error, $2$
\- two category error or $3$ \- three category error) and the number of
neighbours the prediction pair (species and chemical) has closer than a
certain radius. 888Note that, the majority of predictions have errors of $0$
or $1$ which makes the other figures less interpretable. The categorical error
is calculated as the difference between the true and the predicted category
(i.e., very toxic, toxic, harmful, and maybe harmful). The density map is
highly dependent on this radius and we have biased the radius to show
interesting patterns. A larger radius would include more entities and _smear_
the density plots, while a smaller radius would include less entities and,
therefore, densities would be _patchy_ and difficult to show. Figure 6 show
the same but defining neighbours as leaf nodes in the graph with data less
than _depth_ (number of hierarchical levels to closest ancestor) away from
prediction pair. This weights the data density more than the KG density in the
calculation. Note that, the axis are not the same across plots.

Comparing category error $0$ and $1$ in Figure 7 we can see (weakly) that the
number of chemical neighbour density is similar, while the distribution of
species distribution is wider where the error is fairly larger. This indicates
that in the case of lack of species data, neighbourhood species density is the
most important. This is not necessary surprising since a majority of the
chemical side in a species-chemical is known and therefore, less emphasis is
put on chemicals by the models.

Figure 6 shows similar patterns for all category errors, albeit a slightly
more dense chemical neighbourhood for category error $1$. Figure 6 also shows
dispersion in both directions when moving from no to a small error
(Categorical error 1). This can indicate the importance of data density for
both chemical and species in this prediction task. The is expected as
prediction methods used always perform better when more data is available. The
larger errors show similar distributions, to each other, albeit at different
scales (as the axis are different). This is a indication that certain
structures are more important than others to good prediction performance.

Figure 7: Density map of predictions (corresponds to Figure 4(a)) as a
function of number of neighbours within a radius to the prediction chemical-
species pair. The number of predictions in each plot is show as e.g.,
$\\#=259$. The neighbourhood radius is shown as e.g., $r=9$. Note that, the
axis are not necessary equal in all plots. Figure 8: Density map of
predictions (corresponds to Figure 5) as a function of number of neighbours
within a radius to the prediction chemical-species pair. The number of
predictions in each plot is show as e.g., $\\#=501$. The neighbourhood radius
is shown as e.g., $r=10$. Note that, the axis are not necessary equal in all
plots.

Figure 8 show the categorical error and the corresponding locations of each
prediction in (chemical and species) neighbourhood space. Comparing bottom
right of each panel, we see that there is a lower error where there is more
diversity in neighbours, i.e., number of neighbours is spread over a larger
area as seen bottom left compared to top right in Figure 8. The difference is
subtle, but is backed up by a small correlation between the number of chemical
neighbours and mean absolute error ($0.08$, $p<0.05$).

Another way to analyse the content of Figure 8 is to train a model to predict
error based on the neighbourhood (density). We take the vectors representing
the distances to all other chemicals and species, and the absolute error of
the pairs prediction and train a random forest model on this data. The model
is able to predict the error with a mean error of $0.59\pm 0.04$ (over 100,
80/20 train/test random runs) which is less than the extremes of the standard
deviation shown in Figure 3. This error model can be used to generate
confidence measures for the effect prediction model.

###  4.6 Common facts

Common facts (triples) among data points can be a good way to analyse if a
prediction is trustworthy. Table 1 shows selected examples of common facts
among the three closest neighbours to test chemical and species. We show two
good, two moderate and one bad prediction. Table 1 can be seen in conjunction
with Table 2, which indicates a (small) correlation between shared facts and
the absolute error of a prediction. The correlation increase with the number
of neighbours up to 11 neighbours and thereafter stagnates. This is down to
the absence of shared facts at such distance away from the chemical-species
pair.

The correlation between number of facts and prediction error is positive for
chemical data, i.e., the more facts added the larger the error, indicating
potential noise or not relevant facts for prediction in the current version of
the chemical sub-KG of TERA. Moreover, the correlation of species data is
negative, i.e., more data999Not necessary more as this is common facts, but
more concentrated data. gives better performance which is expected. Table 1
show certain aspects of this. e.g., Bluegill in the first row shares both geo-
region and habitat with other close entities which leads to good performance.
While Monkey River Prawns present in Australia and P.R. China does not inspire
confidence in the prediction as this does not define any descriptive features
of the species, but rather superficial features of low importance for
toxicity. For the moderate results we can see some noise (missing or wrong
triples) in the KG, e.g., Fathead minnow is present in Mexico (in TERA),
however, in reality it is present in the greater Nearctic region (wrong in EOL
and, therefore, also wrong in TERA).

Overall, we can see that there is little shared data among chemical neighbours
which emphasises that the long hierarchical chains (which are not investigated
here) of the graph might be more important.

Table 1: Table of selected predictions and their common facts to other data (3 closest neighbours in the KG). The columns shown the Error of the prediction where absolute error $AE=log_{10}(\hat{y})-log_{10}(y)$ ($y$ is concentration), categorical accuracy is defined in main text; Entity, Predicate and Entity/Literal are the components of the shared triples. Entity/Literal can contain multiple values if there are multiple objects for the particular subject-predicate pair. Error | Entity | Predicate | Entity/Literals
---|---|---|---
$AE=0.0026$ | Bluegill | eol:Present |  North Atlantic Ocean, North Pacific Ocean
$CA=1$ |  | eol:Habitat | freshwater biome
| DDT1010101,1’-(2,2,2-Trichloroethylidene)bis(4-chlorobenzene) | ex:compoundIsHeavy | False
$AE=0.0211$ | Colorado Squawfish | eol:Habitat |  freshwater biome, freshwater environment
$CA=1$ |  | rdf:type | Vertebrate
| Permethrin | ex:compoundIsHeavy | False
$AE=0.55$ | Fathead Minnow | eol:Habitat |  freshwater biome, large river biome, small river biome
$CA=0.5$ |  | eol:Present | Mexico
|  | rdf:type | Vertebrate
| Hydroquinone | ex:compoundIsHeavy | False
$AE=2.73$ | Western Toad | eol:Habitat |  freshwater biome, woodland, pond
$CA=0.5$ |  | eol:Present | the United States of America
|  | obo:GAZ_00000071111111Biogeographic realm | Nearctic realm
|  | rdf:type | Vertebrate
| Permethrin | ex:compoundIsHeavy | False
$AE=4.87$ | Monkey River Prawn | eol:Present |  Australia, P.R. China,
$CA=0.25$ |  | rdf:type | Invertebrate
| Cypermethrin | ex:compoundIsHeavy | False

$n$ | correlation | p-value
---|---|---
$2$ | $0.08/0.10$ | $0.16/0.8$
$3$ | $0.05/0.05$ | $0.41/0.34$
$5$ | $0.12/0.04$ | $0.03/0.51$
$7$ | $0.11/(-)0.06$ | $0.05/0.32$
$9$ | $0.12/(-)0.14$ | $0.03/0.01$
$11$ | $0.17/(-)0.14$ | $0.00/0.01$
Table 2: Correlation between number of facts (Table 1) with $n$ neighbours and
absolute error. The two numbers represent the correlation and p-value for
experimental chemical and species, respectively.

##  5 Discussion

We have presented results that show the benefit of using knowledge graph
embeddings in predicting the correct toxicity of missing data within similar
chemical-species groups (i.e., read-across). In addition, we have presented
several methods for providing quantitative and qualitative insight of these
predictions.

The results from this study are encouraging when applied in this example case.
We improve results with around 40% in $R^{2}$ ($0.25\to 0.34$) the setting of
gap-filling unknown species with organic chemicals. Along with the results,
three explanations where provided. The first provide an overall view of graph
density in relation to the prediction performance. This can be used to assess
the quality of the knowledge graph, the embeddings, and the trustworthiness of
the adverse effect data itself.

The error prediction can directly be used to assess the confidence of the
individual predictions. This can further provide species or chemical groups
where the data density can be improved, i.e., predicted error can be used as a
ranking of which laboratory experiments will be overall beneficial for the
effect prediction model.

Comparing results of gap-filling unknown species vs. gap-filling unknown
chemicals we can deduce that prediction of unknown chemicals is a harder task
as classifications and functional groups do not describe chemicals as with the
same resolution as hierarchy and traits define species. This is where and
investigation into which parts of TERA contributes and counteract the
predictions is needed. Other information such as the chemical molecules added
as a OWL based sub-graphs (e.g., Hastings et al. [12]) can also be added to
TERA to improve the granularity of the KG.

This lack of complete data is the biggest caveat in this work. Using automated
knowledge extraction and integration gathers available data, but does not
necessary take the specific species or chemical domains into account. This
could be solved with a bottom-up approach seeing where the baseline model
falls short and adding data to the knowledge graph accordingly.

Given the discrepancy in the training data as shown in Figure 3, we are
satisfied with the performance of the models in for initial screening
purposes. Consistent effect data is key for this method to outperform narrow
domain models across multiple application domains, however, these datasets are
not available currently and beyond the scope of this study.

##  6 Conclusion

In this work we have set up a framework for using knowledge graphs in
ecotoxicological effect prediction, especially as a initial screening tool for
unknown and rare chemicals or species.

This framework includes the use of tools to integrate data sources into a
domain specific knowledge graph we call TERA. The knowledge graph entities can
be represented in a vector space using knowledge graph embedding models. This
enables the use of chemicals and species in TERA directly in machine learning
models. Moreover, the prediction results can be improved by using embeddings
over a naive baseline and is the basis of explanations of the predictions.
However, we found that chemical hierarchies are not as descriptive as species
hierarchies when applied in this prediction task.

###  6.1 Future work

Previous [23, 24] and present studies have shown one way knowledge graphs can
be included in effect prediction. And we show how a KG can help gain insight
into the predictions. However, this ground work is just the beginning of
integrating these methodologies into hazard assessment approaches. The next
steps is to get the tools and prediction models integrated into computational
pipelines to support effect predictions and read-across. This would
potentially reduce the application domain further which may require additional
effect data sources to be integrated into TERA. Furthermore, more effect data
will be a benefit for a more extensive knowledge graph and we plan to
integrate yet more resources into TERA. On the other hand, extension of KGE
models to handle hierarchies are expected to increase performance.

This use-case is also well suited to test improvements of new semantic web
tools, e.g., ontology alignment, due to the size of TERA; and KGE models, due
to the unique structure of TERA.

###  6.2 Resources

The resources for this paper are available at https://github.com/NIVA-
Knowledge-Graph/kge_ecotox_regression.

The source code for TERA is available at https://github.com/NIVA-Knowledge-
Graph/TERA.

Acknowledgements. This work is supported by the grant 272414 from the Research
Council of Norway (RCN), the MixRisk project (Research Council of Norway,
project 268294), SIRIUS Centre for Scalable Data Access (Research Council of
Norway, project 237889), Samsung Research UK, Siemens AG, and the EPSRC
projects AnaLOG (EP/P025943/1), OASIS (EP/S032347/1), UK FIRES (EP/S019111/1),
the AIDA project (Alan Turing Institute), by NIVAs Computational Toxicology
Program, NCTP (niva.no/nctp), and donated weekends.

## References

  * Agency [2004] U.S. Environmental Protection Agency.  Chemical Hazard Classification and Labeling: Comparison of OPP Requirements and the GHS, 2004.  https://www.epa.gov/sites/production/files/2015-09/documents/ghscriteria-summary.pdf.
  * Animal and Plant Health Inspection Service [2021] Animal and U.S. Department of Agriculture Plant Health Inspection Service.  Annual report animal usage by fiscal year: Total number of animals research facilities used in regulated activities (column b).  2021\.
  * Arnaout and Elbassuoni [2018] Hiba Arnaout and Shady Elbassuoni.  Effective Searching of RDF Knowledge Graphs.  _Web Semantics: Science, Services and Agents on the World Wide Web_ , 48(0), 2018.  ISSN 1570-8268.
  * Baird and Barata [1998] Donald J. Baird and Carlos Barata.  Variability in the response of daphnia clones to toxic substances: Are safety margins being compromised?  In Jürg P. Seiler, Judith L. Autrup, and Herman Autrup, editors, _Diversification in Toxicology — Man and Environment_ , pages 399–406, Berlin, Heidelberg, 1998. Springer Berlin Heidelberg.  ISBN 978-3-642-46856-8.
  * Bordes et al. [2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multi-relational data.  In _Advances in Neural Information Processing Systems 26_ , pages 2787–2795. Curran Associates, Inc., 2013.
  * Bradbury [1995] Steven P. Bradbury.  Quantitative structure-activity relationships and ecological risk assessment: an overview of predictive aquatic toxicology research.  _Toxicology Letters_ , 79(1):229–237, 1995.  ISSN 0378-4274.  doi: https://doi.org/10.1016/0378-4274(95)03374-T.  URL https://www.sciencedirect.com/science/article/pii/037842749503374T.  Decision Subtances Methodologies for Human Health Risk Assessment of Toxic Substances.
  * Busquet et al. [2020] Francois Busquet, Andre Kleensang, Costanza Rovida, Kathrin Herrmann, Marcel Leist, and Thomas Hartung.  New european union statistics on laboratory animal use – what really counts!  _ALTEX - Alternatives to animal experimentation_ , 37(2):167–186, Mar. 2020.  doi: 10.14573/altex.2003241.  URL https://www.altex.org/index.php/altex/article/view/1755.
  * Chen et al. [2018] Jiaoyan Chen, Freddy Lécué, Jeff Z Pan, Ian Horrocks, and Huajun Chen.  Knowledge-based transfer learning explanation.  In _Sixteenth International Conference on Principles of Knowledge Representation and Reasoning_ , 2018.
  * Chen et al. [2020] Jiaoyan Chen, Pan Hu, Ernesto Jimenez-Ruiz, Ole Magnus Holter, Denvar Antonyrajah, and Ian Horrocks.  OWL2Vec*: Embedding of OWL Ontologies, 2020.
  * Doering et al. [2018] Jon A Doering, Sehan Lee, Kurt Kristiansen, Linn Evenseth, Mace G Barron, Ingebrigt Sylte, and Carlie A LaLone.  In Silico Site-Directed Mutagenesis Informs Species-Specific Predictions of Chemical Susceptibility Derived From the Sequence Alignment to Predict Across Species Susceptibility (SeqAPASS) Tool.  _Toxicological Sciences_ , 166(1):131–145, 07 2018.  ISSN 1096-6080.  doi: 10.1093/toxsci/kfy186.  URL https://doi.org/10.1093/toxsci/kfy186.
  * Futia and Vetrò [2020] Giuseppe Futia and Antonio Vetrò.  On the Integration of Knowledge Graphs into Deep Learning Models for a More Comprehensible AI—Three Challenges for Future Research.  _Information_ , 11(2), 2020.  ISSN 2078-2489.  doi: 10.3390/info11020122.  URL https://www.mdpi.com/2078-2489/11/2/122.
  * Hastings et al. [2010] J. Hastings, M. Dumontier, D. Hull, M. Horridge, C. Steinbeck, U. Sattler, R. Stevens, T. Horne, and K. Britz.  Representing chemicals using OWL, description graphs and rules.  Jun 2010.  URL http://hdl.handle.net/10204/4919.
  * Hastings et al. [2016] Janna Hastings, Gareth Owen, Adriano Dekker, Marcus Ennis, Namrata Kale, Venkatesh Muthukrishnan, Steve Turner, Neil Swainston, Pedro Mendes, and Christoph Steinbeck.  ChEBI in 2016: Improved services and an expanding collection of metabolites.  _Nucleic acids research_ , 44(D1):214—9, 2016\.
  * Jana et al. [2020] Gourhari Jana, Ranita Pal, Shamik Sural, and Pratim Kumar Chattaraj.  Quantitative structure-toxicity relationship: An “in silico study” using electrophilicity and hydrophobicity as descriptors.  _International Journal of Quantum Chemistry_ , 120(6):e26097, 2020.  doi: https://doi.org/10.1002/qua.26097.
  * Jiménez-Ruiz and Cuenca Grau [2011] Ernesto Jiménez-Ruiz and Bernardo Cuenca Grau.  LogMap: Logic-Based and Scalable Ontology Matching.  In _10th International Semantic Web Conference_ , pages 273–288, 2011\.
  * Jiménez-Ruiz et al. [2012] Ernesto Jiménez-Ruiz, Bernardo Cuenca Grau, Yujiao Zhou, and Ian Horrocks.  Large-scale interactive ontology matching: Algorithms and implementation.  In _the 20th European Conference on Artificial Intelligence (ECAI)_ , pages 444–449. IOS Press, 2012.
  * Jupp et al. [2014] Simon Jupp, James Malone, Jerven Bolleman, Marco Brandizi, Mark Davies, Leyla Garcia, Anna Gaulton, Sebastien Gehant, Camille Laibe, Nicole Redaschi, Sarala M. Wimalaratne, Maria Martin, Nicolas Le Novère, Helen Parkinson, Ewan Birney, and Andrew M. Jenkinson.  The EBI RDF platform: linked open data for the life sciences.  _Bioinformatics_ , 30(9):1338–1339, 01 2014.  ISSN 1367-4803.  doi: 10.1093/bioinformatics/btt765.  URL https://doi.org/10.1093/bioinformatics/btt765.
  * Kim et al. [2018] Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E Bolton.  PubChem 2019 update: improved access to chemical data.  _Nucleic Acids Research_ , 47(D1):D1102–D1109, 10 2018.  ISSN 0305-1048.
  * Kingma and Ba [2014] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_ , 12 2014.
  * LaLone et al. [2014] C. LaLone, D. Villeneuve, H. Helgen, and G. Ankley.  Sequence alignment to predict across-species susceptibility.  _SETAC Europe, Basel, SWITZERLAND, May 11 - 15,_ , 2014.
  * Mikolov et al. [2013] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient Estimation of Word Representations in Vector Space.  In Yoshua Bengio and Yann LeCun, editors, _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_ , 2013.  URL http://arxiv.org/abs/1301.3781.
  * Mukerjee [2004] Madhusree Mukerjee.  Speaking for the animals: A veterinarian analyzes the turf battles that have transformed the animal laboratory.  _Scientific American_ , 2004.
  * Myklebust et al. [2019] Erik B. Myklebust, Ernesto Jiménez-Ruiz, Jiaoyan Chen, Raoul Wolf, and Knut Erik Tollefsen.  Knowledge Graph Embedding for Ecotoxicological Effect Prediction.  In _Int’l Sem. Web Conf. (ISWC)_ , 2019.  Best Student Paper in the In-Use track.
  * Myklebust et al. [2022] Erik B. Myklebust, Ernesto Jiménez-Ruiz, Jiaoyan Chen, Raoul Wolf, and Knut Erik Tollefsen.  Prediction of adverse biological effects of chemicals using knowledge graph embeddings.  _Semantic Web_ , 13:299–338, 2022.  ISSN 2210-4968.  doi: 10.3233/SW-222804.  URL https://doi.org/10.3233/SW-222804.  3\.
  * NLM [2020] NLM.  Medical Subject Headings (MeSH) RDF, 2020.  https://id.nlm.nih.gov/mesh/.
  * Olker et al. [2022] Jennifer H. Olker, Colleen M. Elonen, Anne Pilli, Arne Anderson, Brian Kinziger, Stephen Erickson, Michael Skopinski, Anita Pomplun, Carlie A. LaLone, Christine L. Russom, and Dale Hoff.  The ecotoxicology knowledgebase: A curated database of ecologically relevant toxicity tests to support environmental research and risk assessment.  _Environmental Toxicology and Chemistry_ , 41(6):1520–1539, 2022.  doi: https://doi.org/10.1002/etc.5324.  URL https://setac.onlinelibrary.wiley.com/doi/abs/10.1002/etc.5324.
  * Parr et al. [2014] Cynthia S. Parr, Nathan Wilson, Patrick Leary, Katja S. Schulz, Kristen Lans, Lisa Walley, Jennifer A. Hammock, Anthony Goddard, Jeremy Rice, Marie Studer, Jeffrey T. G. Holmes, and Jr. Robert J. Corrigan.  The Encyclopedia of Life v2: Providing Global Access to Knowledge About Life on Earth.  _Biodiversity Data Journal_ , 2:e1079, 2014.  ISSN 1314-2836.  doi: 10.3897/BDJ.2.e1079.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Ristoski et al. [2019] Petar Ristoski, Jessica Rosati, Tommaso Di Noia, Renato De Leone, and Heiko Paulheim.  RDF2Vec: RDF graph embeddings and their applications.  _Semantic Web_ , 10(4):721–752, 2019.  doi: 10.3233/SW-180317.  URL https://doi.org/10.3233/SW-180317.
  * Rossi et al. [2021] Andrea Rossi, Denilson Barbosa, Donatella Firmani, Antonio Matinata, and Paolo Merialdo.  Knowledge graph embedding for link prediction: A comparative analysis.  _ACM Trans. Knowl. Discov. Data_ , 15(2):14:1–14:49, 2021.  doi: 10.1145/3424672.  URL https://doi.org/10.1145/3424672.
  * Russo et al. [2017] D. P. Russo, M. T. Kim, W. Wang, D. Pinolini, S. Shende, J. Strickland, T. Hartung, and H. Zhu.  CIIPro: a new read-across portal to fill data gaps using public large-scale chemical and biological data.  _Bioinformatics_ , 33(3):464–466, 02 2017.
  * Scholkopf et al. [1999] Bernhard Scholkopf, Alexander Smola, and Klaus-Robert Müller.  Kernel principal component analysis.  In _Advances in Kernel Methods - Support Vector Learning_ , pages 327–352. MIT Press, 1999.
  * Tiddi et al. [2020] Ilaria Tiddi, Freddy Lécué, and Pascal Hitzler, editors.  _Knowledge Graphs for eXplainable Artificial Intelligence: Foundations, Applications and Challenges_ , volume 47 of _Studies on the Semantic Web_.  IOS Press, 2020.  ISBN 978-1-64368-080-4.
  * Trouillon et al. [2016] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard.  Complex Embeddings for Simple Link Prediction.  _CoRR_ , abs/1606.06357, 2016.
  * van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing Data using t-SNE.  _Journal of Machine Learning Research_ , 9(86):2579–2605, 2008.  URL http://jmlr.org/papers/v9/vandermaaten08a.html.
  * Vilone and Longo [2020] Giulia Vilone and Luca Longo.  Explainable Artificial Intelligence: a Systematic Review, 2020.
  * Vilone and Longo [2021] Giulia Vilone and Luca Longo.  Notions of explainability and evaluation approaches for explainable artificial intelligence.  _Inf. Fusion_ , 76:89–106, 2021.  doi: 10.1016/j.inffus.2021.05.009.  URL https://doi.org/10.1016/j.inffus.2021.05.009.
  * Vrandecic and Krötzsch [2014] Denny Vrandecic and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85, 2014.
  * Wang et al. [2017] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo.  Knowledge graph embedding: A survey of approaches and applications.  _IEEE Trans. Knowl. Data Eng._ , 29(12):2724–2743, 2017.
  * Wittwehr et al. [2019] Clemens Wittwehr, Paul Blomstedt, John Paul Gosling, Tomi Peltola, Barbara Raffael, Andrea-Nicole Richarz, Marta Sienkiewicz, Paul Whaley, Andrew Worth, and Maurice Whelan.  Artificial intelligence for chemical risk assessment.  _Computational Toxicology_ , page 100114, 2019.  ISSN 2468-1113.  doi: https://doi.org/10.1016/j.comtox.2019.100114.  URL http://www.sciencedirect.com/science/article/pii/S2468111319300349.
  * Xu et al. [2021] Chengjin Xu, Mojtaba Nayyeri, Sahar Vahdati, and Jens Lehmann.  Multiple run ensemble learning with low-dimensional knowledge graph embeddings, 2021.

##  Appendix A Method Details

This appendix provides details for Section 3.

###  A.1 Effect data query

Query 1: Querying experimental data with SPARQL

⬇

SELECT ?species ?chem ?conc ?unit WHERE {

?test ns:hasResult [ns:endpoint ?end ;

ns:effect effect:MOR ;

ns:concentration [

rdf:value ?conc ;

unit:units ?unit

]] .

?test ns:species ?species .

?test ns:chemical ?chem .

?test ns:studyDuration [rdf:value ?sd ;

unit:units ?cu] ] .

FILTER (?unit in (unit:MicrogramPerLitre,

unit:MilligramPerLitre))

FILTER (?end in (endpoint:LC50,

endpoint:LD50,

endpoint:EC50) )

FILTER (?cu = unit:Hour)

FILTER (?sd >= 24 || ?sd <= 96)

}

Query 1 looks for tests (i.e., experiments) in ECOTOX with results which has
mortality as effect and a endpoint in $\\{LC_{50},LD_{50},EC_{50}\\}$.
121212We consider $LC_{50}\land MOR\equiv LD_{50}\land MOR\equiv EC_{50}\land
MOR$. The query also filters on the units used for the chemical concentration,
this is a cheap trick to filter on only results where the chemical is added to
the aquatic environment and not given orally. Finally, we filter test with a
duration between 24 and 96 hours.

###  A.2 Optimizing ComplEx

The KGE model is optimized by minimizing the pointwize logistic loss

$\displaystyle L=\sum_{t\in
KG}\text{log}(1+\text{exp}(-y_{\mathbf{t}}S(\mathbf{t}))$ (4)

where $y_{\mathbf{t}}\in\\{-1,1\\}$ is the label of the triple, $1$ for true,
$-1$ for false. The false triples are created by replacing the subject and/or
object of triples in the KG with random entities from the KG.

Figure 9 show the stepwise optimization of ComplEx. A false fact is generated
randomly from the true fact by permuting subject and/or object. Thereafter,
the true fact follow the blue colour while the false fact follows the red
colour. The two scores are evaluated in the loss function and a correction of
the embedding layer is calculated and applied by the optimizer, we use the
Adam optimizer [19].

Figure 9: Optimization of ComplEx. Each triple $t$ in TERA is corrupted to
$t^{\prime}$; vector representations (random at start) are extracted for the
triples; the score for $t$ and $t^{\prime}$ is calculated; and the loss
function is calculated; finally, updated to the embedding layer is calculated
and applied.

###  A.3 Support Vector Machines

A SVM is normally used in classification problems where the goal is to
maximize the distance between the decision boundary (class divide) and the
individual points. Regression can be performed using a SVM by considering the
margin from the decision boundary as a real number and not a strict boundary.

SVMs are strictly linear, however, applying a kernel can be done to make it
non-linear. This projects the data points into a higher dimensional space
where the problem becomes linear. In this work, we use a radial basis function
kernel defined as

$\displaystyle
K(\mathbf{x},\mathbf{x^{\prime}})=\exp(-\gamma||\mathbf{x}-\mathbf{x^{\prime}}||^{2})$

where $\gamma=1/2\sigma$, and $\mathbf{x}$ and $\mathbf{x^{\prime}}$ are two
different datapoints.

###  A.4 Explanation Algorithms

In this section, we will describe the algorithms in plain text. The algorithms
are available in the projects GitHub repository.
131313https://github.com/NIVA-Knowledge-Graph/kge_ecotox_regression

Neighbourhood density. Based on the embeddings we calculate the distance from
prediction chemical and species to other entities in the KG as in (5).
Thereafter, we find how many chemical entities are within $r_{1}$ of test
chemical and species entities are within $r_{2}$ of test species. Figures 7
and 8 are produces by plotting the density of the number of neighbours in each
error category, i.e., 0,1,2,3 error.

Error prediction. Based on the embeddings a similarity measure can be
calculated. We define a similarity matrix as

$\displaystyle S_{i,j}=||\mathbf{e}_{i}-\mathbf{e}_{j}||_{2},$ (5)

i.e., the Euclidean distance between the entity embeddings.

From the prediction model we can gather an (mean absolute) error for each
prediction. We train a random forest model (RF) to predict the error where the
input is the similarity vectors from above. We use the same 5-fold validation
as described in Section 3.

Common facts. Based on (5) we can find the $n$ closest entities to an
prediction entity (chemical or species). This set of $n$ closest entities and
entity we call M. Then the common facts are defined as

$\displaystyle CF=\bigcap_{e\in M}\\{\langle s,p,o\rangle|s\equiv
e\land\langle s,p,o\rangle\in KG\\},$ (6)

i.e., the intersection of facts associated with each entity in $M$.


# Feature-informed Embedding Space Regularization For Audio Classification

Yun-Ning Hung1 1 This work was performed while Y. Hung was a master’s student
at Georgia Institute of Technology. Speech, Audio & Music Intelligence Team
ByteDance
<EMAIL_ADDRESS>Alexander Lerch  Music Informatics Group
Georgia Institute of Technology
<EMAIL_ADDRESS>
###### Abstract

Feature representations derived from models pre-trained on large-scale
datasets have shown their generalizability on a variety of audio analysis
tasks. Despite this generalizability, however, task-specific features can
outperform if sufficient training data is available, as specific task-relevant
properties can be learned. Furthermore, the complex pre-trained models bring
considerable computational burdens during inference. We propose to leverage
both detailed task-specific features from spectrogram input and generic pre-
trained features by introducing two regularization methods that integrate the
information of both feature classes. The workload is kept low during inference
as the pre-trained features are only necessary for training. In experiments
with the pre-trained features VGGish, OpenL3, and a combination of both, we
show that the proposed methods not only outperform baseline methods, but also
can improve state-of-the-art models on several audio classification tasks. The
results also suggest that using the mixture of features performs better than
using individual features.

###### Index Terms:

Music auto-tagging, transfer learning, audio classification, regularization

##  I Introduction

Feature representations learned with deep architectures from large amounts of
training data have shown to be powerful for a variety of tasks beyond the
tasks they were trained for. The pre-trained features from the BERT model [1],
for example, have been successfully used in multiple Natural Language
Processing (NLP) tasks such as question answering, text classification, and
language inference [2]. In the image domain, large-scale pre-trained models
such as ImageNet [3] and VGG [4] have achieved competitive results in image
classification and assist in many visual feature extraction tasks [5].

There exist several pre-trained models in the audio domain, such as L3-Net
[6], VGGish [7], and Jukebox [8]. L3-Net leverages both audio and visual
information provided by video to train an audio feature extraction network,
while the VGGish model is pre-trained on a large-scale audio dataset, AudioSet
[7]. Jukebox adopts a language model similar to NLP and is pre-trained on
music audio only. Those features have been successfully applied to various
Music Information Retrieval (MIR) tasks, including weakly-supervised
instrument recognition [9], cross-modal representation learning [10, 11],
music auto-tagging [12, 13], music emotion recognition [14, 15], and music
genre classification [16, 17]. This successful application to a multitude of
different tasks combined with the comparably compact representation implies
that these features are able to capture many task-agnostic properties of audio
and music signals.

Although these transfer learning approaches are used successfully for such a
wide range of tasks, there are potential drawbacks when using pre-trained
features directly as the input representation. First, the extracted compact
representation might lack task-specific information, which, in contrast, a
task-specific representation learned from spectrogram input will include [18].
While it is possible to update and fine-tune the pre-trained model with task-
specific datasets to catch this relevant information [19], the computational
burden during training is increased due to the high complexity of the feature
extractor. Second, the time resolution of pre-trained features is fixed,
forcing any system utilizing the features to the same time resolution. VGGish
features, for example, have a time resolution of approx. 1 s. Tasks that need
a higher resolution such as beat-tracking [20], might not be able to utilize
pre-trained feature representations. Third, extractors such as VGGish and
L3-Net rely on fixed input representations such as Mel spectrograms to extract
features. Several recent studies, however, have shown that other input
representations such as harmonic representations [21] or multi-rate PCEN [22]
achieve superior performance on several audio-related tasks. Fourth, the deep
architectures of pre-trained feature extractors also increase the
computational workload and the execution time at inference. Small real-time
devices with limited computational power, for example, might not be able to
benefit from the pre-trained representations.

To address these drawbacks, we propose to incorporate the task-agnostic
information of pre-trained representations into a new smaller-scale model
without using them as input to the model. Our approaches show some similarity
to feature-based knowledge distillation approaches used in teacher-student
learning [23], where the pre-trained representations are used to regularize
the embedding space during training. Thus, the pre-trained representations
provide condensed, task-agnostic information to help shape the embedding, but
are not required during inference. The architecture of the original model does
not need to change, so that the input representation and its time resolution
can be chosen freely for the task at hand. Such feature-based knowledge
transfer approaches are rarely explored in the music/audio domain. Moreover,
to extend commonly-used regularization loss functions (e.g.,
$\pazocal{L}_{1}$, $\pazocal{L}_{\mathrm{CE}}$), we propose two loss functions
based on cosine distance to match the embeddings. Cosine distance has been
commonly used in several metric learning scenarios, such as few-shot learning
[24, 25] and recommendation system [26].

To summarize, the main contributions of this paper are:  (i) the introduction
of two regularization methods integrating the information of pre-trained
feature representations during training time without increasing the
computational complexity during inference, and  (ii) an investigation of the
suitability of the two representations VGGish and OpenL3 and their combination
for this regularization, and a detailed study of the proposed regularization
methods for SOTA models.

Figure 1: Pipeline for the training including the learned feature. The lower
branch is discarded during inference.

##  II Methods

We propose two approaches to incorporate pre-trained feature representations
into existing model architectures. In the following sections, we first
introduce the general training pipeline of our regularization methods and the
two pre-trained representations we use in our experiment. Then, details of two
proposed regularization methods will be presented.

###  II-A System Description

Figure 1 shows the general structure of our training pipeline. For the
regularization to be applicable, we assume the deep-learning model to be
composed of a similar pipeline with an encoder ($E$) to extract the
intermediate representation ($l$) and a decoder ($D$) to predict the output
based on this representation. The intermediate representation $l\in
R^{T_{\mathrm{E}}\times B_{\mathrm{E}}}$ is also referred to as embedding,
where $T_{\mathrm{E}}$ represents the number of time frames while
$B_{\mathrm{E}}$ represents the dimensionality of the embedding.

The pre-trained features are a 2-dimensional representation $v\in
R^{T_{\mathrm{F}}\times B_{\mathrm{F}}}$ where $T_{\mathrm{F}}$ represents the
number of time frames while $B_{\mathrm{F}}$ represents the dimensionality of
the learned features. Although we only investigate two widely used
representations extracted with deep architectures, it is conceivable that any
kind of custom feature in this format can be used.

###  II-B Learned Features

We investigate three regularization inputs in this study, VGGish [7], OpenL3
[6], and the combination of both. VGGish and OpenL3 features are chosen due to
their proven success in several audio-related downstream tasks [9, 11, 12,
16]. Both representations are extracted by models trained on large-scale audio
datasets. VGGish features are extracted by the VGGish model [7] pre-trained on
AudioSet [27] to perform audio classification at a time resolution of approx.
1 s (no overlap) with a feature dimensionality of $B_{\mathrm{F}}=128$. The
features are PCA transformed (with whitening) and quantized to 8-bits. The
OpenL3 features [6] are extracted from L3-net pre-trained on a subset of
AudioSet. These features are also 2-dimensional representations with a time
resolution of 0.1 s (no overlap) and $B_{\mathrm{F}}=512$.

The combination of these two features is achieved by a simple concatenation
along the $B_{\mathrm{F}}$ dimension. To match the temporal resolution of
OpenL3, VGGish features are repeated 10 times along the temporal dimension.
The resulting combined representation has a time resolution of 0.1 s and a
dimensionality of $B_{\mathrm{F}}=640$.111More sophisticated interpolation
methods for upsampling to match the time resolutions will be explored in
future work.

| Methods | DCASE 17 (F1) | MTAT (PR-AUC)
---|---|---|---
|  | None | VGGish | OpenL3 | Combined | None | VGGish | OpenL3 | Combined
Baseline | Won et al. [21] | 0.547 | - | - | - | 0.465 | - | - | -
Feature-only | - | 0.496 | 0.477 | 0.501 | - | 0.454 | 0.454 | 0.456
Concat | - | 0.529 | 0.492 | 0.495 | - | 0.457 | 0.464 | 0.458
FiLM | - | 0.548 | 0.511 | 0.542 | - | 0.470 | 0.463 | 0.470
Proposed | Con-Reg | - | 0.568 | 0.557 | 0.576 | - | 0.471 | 0.466 | 0.469
Dis-Reg | - | 0.548 | 0.543 | 0.563 | - | 0.464 | 0.468 | 0.463
TABLE I: Evaluation results for the DCASE 17 and MTAT datasets. Bold numbers
represent the highest score in each category while the underlined numbers
indicate the scores improved over the baseline model.

###  II-C Proposed Feature Integration Methods

Since VGGish, OpenL3, and the mixture features have different time resolution
than the embeddings from most of the existing models, we repeat or average-
pool the features $n$ times with $n$ representing the number of time frames in
1 s or 0.1 s to fit the time resolution. This approach is based on the
assumption that a slight misalignment in time will not influence the result
too much.

The design of the regularization methods is based on two assumptions. First,
pre-trained features might contain information that is useful for various
tasks but cannot be adequately represented in the unregularized embedding
space (e.g., due to insufficient training data). Second, pre-trained features
have strong discriminative power. The proposed regularization methods attempt
to transfer the knowledge from the pre-trained feature space into the
embedding vectors by adding structure to the embedding vectors resulting in a
more separable embedding space.

Both methods add an extra loss term for network training:

$L_{\mathrm{final}}=L+\alpha L_{\mathrm{Reg}},$ (1)

with the hyperparameter $\alpha$ adjusting the contribution on the overall
loss $L_{\mathrm{final}}$. The following methods are proposed to incorporate
learned features into the embeddings:

  * •

Con-Reg: Con-Reg aims at regularizing the embedding space so that its layout
becomes more similar to the feature space of the pre-trained feature $v$. To
do so, we utilize the features extracted from the audio and add an extra loss
term $L_{\mathrm{Con-Reg}}$ based on cosine distance222Pilot experiments using
$\pazocal{L}_{2}$ as the distance function did not lead to competitive
results. to minimize the distance between embeddings and pre-trained features:

$L_{\mathrm{Con-Reg}}=d_{\mathrm{cos}}(l,f(v))$ (2)

where $f$ represents a 1D CNN with kernel size 1 szegedy2015going to transform
the feature dimensionality $B_{\mathrm{F}}$ to match the embedding
dimensionality, allowing us to compute the cosine distance between $l$ and
$v$.

  * •

Dis-Reg: Dis-Reg is a distance-based regularization. Similar to Con-Reg, the
embedding space is regularized with an additional loss. In this case, however,
the additional loss term $L_{\mathrm{Dis-Reg}}$ aims at forcing the distances
between pairs of embedding vectors to be similar to the distance of
corresponding pairs of pre-trained features:

$L_{\mathrm{Dis-
Reg}}=|d_{\mathrm{cos}}(l_{i},l_{j})-d_{\mathrm{cos}}(v_{i},v_{j})|,$ (3)

where $d_{\mathrm{cos}}$ represents the cosine distance between two embeddings
from samples $i$ and $j$ ($i\neq j$) or the distance of two corresponding
learned features, respectively.

###  II-D Model Architecture

The harmonic CNN model proposed by Won et al. [21] is chosen as our
experimental system. This model utilizes learnable harmonic filters to capture
the inherent harmonic structure of the input audio and achieves SOTA results
on music tagging, sound event tagging, and keyword spotting. The model is
composed of seven residual blocks as an encoder to extract embeddings from the
harmonic representation. Two linear layers are then used as a decoder to
predict the output from the embeddings.

##  III Experiment

###  III-A Baseline Feature Integration Methods

To provide a baseline reference as a comparison to the proposed regularization
methods, we also present the result for three baseline methods incorporating
pre-trained features:

  * •

Features-only: The simplest way of incorporating pre-trained features is to
directly use these pre-trained features as the input for the decoder. The
decoder of the original model is adjusted to fit the dimensionality of the
features.

  * •

Concat: Concat simply concatenates embeddings $l$ and pre-trained features $v$
along the $B$ dimension. The pre-trained features, therefore, act as a
supplementary of the embeddings and the embedding space is transformed into a
joint feature space. The decoder then has the flexibility to decide how to
leverage the information from the joint feature space. Since the concatenated
embeddings have a larger dimension along $B$ than the original embeddings, the
first layer of the decoder will be adjusted to fit the dimensionality of the
new intermediate representation.

  * •

FiLM: A Feature-wise Linear Modulation (FiLM) layer is originally proposed as
a general-purpose conditioning method to assist visual reasoning, which is
difficult for standard deep-learning methods [29]. The success of FiLM leads
to its successful usage in several audio-related tasks [30, 31]. The FiLM
layer influences neural network computation via a simple, feature-wise affine
transformation based on conditioning information:

$l_{new}=\gamma\cdot l+\beta$ (4)

where $\gamma=f(v)$ and $\beta=h(v)$ are calculated based on pre-trained
features $v$. We use a simple linear layer for the functions $f$ and $h$ to
compute the transformation parameters $\gamma$ and $\beta$. The resulting
embeddings after the transformation are $l_{new}$.

###  III-B Tasks

We investigate our proposed methods on two different tasks to demonstrate the
influence of the regularization: music tagging and sound event tagging as also
used by Won et al. [21].

####  III-B1 Music Auto-Tagging

Music auto-tagging is a multi-label classification problem aiming to predict
tags for a musical piece. The used dataset is MagnaTagATune (MTAT) [32]. This
dataset has approx. 21k audio clips with each clip around 30 s. The dataset
contains a variety of tags, including genre, mood, and instrumentation. The
top-50 tags are chosen for label prediction. The results are reported using
the Area Under Precision-Recall Curve (PR-AUC). The OpenL3 features are
extracted in the music subset mode.

####  III-B2 Sound Event Tagging

Sound event tagging is an important task that has been included in DCASE
challenges for many years [33]. The goal of this task is to detect audio
events in a sound excerpt. Following Won et al., we choose “Task 4: Large-
scale weakly supervised sound event detection for smart cars” from the
DCASE2017 challenge as our target task. The dataset is a subset of AudioSet
[27] and contains approx. 53k audio excerpts for 17 classes. Each excerpt is
around 10 s. The results are reported using the average of instance-level
F1-scores with a threshold value of 0.1. In this case, the OpenL3 features are
extracted in the environmental subset mode.

###  III-C Experimental Setup

The training setup, including dataset cleaning and split and the selection of
optimizer and learning rate, is mirrored from their work. All methods are
evaluated on each task by using VGGish, OpenL3, and the combined pre-trained
features. The best model is selected based on the validation metric for each
task for testing. After hyperparameter tuning, we choose $\alpha=5$ for
$L_{\mathrm{Con-Reg}}$ and $\alpha=1$ for $L_{\mathrm{Dis-Reg}}$.

Figure 2: Ablation study for the results under different percentage of
training data for SOTA system and our best method.

##  IV Result and Discussion

The experimental results are shown in Table I. In the following, we discuss
different aspects of these results separately

####  IV-1 Learned Features

It can be observed that VGGish features generally outperform the OpenL3
features. The combined features tend to perform on par or better than the
individual features. This result suggests that —although VGGish and OpenL3
were both trained on AudioSet— they encode somewhat complementary information.
This is most likely due to different training strategies and hints at the
possibility of combining them for other downstream tasks.

####  IV-2 Baselines

Among all the baseline methods, _FiLM_ and _Concat_ generally perform better
than using only pre-trained features for prediction, but comparable to or only
slightly better than Won et al.’s original system. This result indicates that
there is some task-specific information that exists in the harmonic
representations but not in the pre-trained features. _FiLM_ , in general,
tends to outperform _Concat_.

####  IV-3 Proposed Methods

Both proposed methods tend to generally give higher results than the baselines
and thus demonstrate the efficiency of our proposed regularization methods.
Since the pre-trained features could not achieve better performance than
learning from harmonic representation, directly incorporating features into
embedding can potentially be seen as noise during inference and drop the
performance. In contrast, indirectly leveraging pre-trained features does not
have this limitation. The model is able to combine the task-specific
information learned through the harmonic representation with the
complementary, task-agnostic but highly meaningful information of the pre-
trained features.

_Con-Reg_ generally outperforms _Dis-Reg_. We discover that $L_{\mathrm{Dis-
Reg}}$ has larger values than $L_{\mathrm{Con-Reg}}$ during training. The
larger loss term indicates that optimizing the model to predict embedding
vectors matching the distance of pre-trained features is harder than directly
minimizing the distance between embedding vectors and pre-trained features. As
a result, this regularization method could not structure embeddings space
better than Con-Reg.

Furthermore, _Con-Reg_ achieves better results than the SOTA model proposed by
Won et al. [21] on both the DCASE 17 dataset and MTAT dataset.

####  IV-4 Amount of Training Data

We can observe that the improvement for the MTAT dataset is very limited
compared to the DCASE 17 dataset. This might be due to the fact that the MTAT
dataset is larger in size; therefore, the model is able to learn good
embeddings directly from the training data and the additional information from
the learned features is unnecessary. To compare the effect of insufficient
training data, we present the results of an ablation study by randomly
choosing $x$ percent of audio files in the training set to form training
subsets with $x\in{10\%,30\%,50\%,70\%,90\%}$. We use the best setting, Con-
Reg with combined features, for the experiment and compare the performance
with the SOTA system. The results are shown in Figure 2. We can observe that
with decreasing training data, the improvement from the proposed
regularization method becomes more obvious, especially lower than 30% of
training data.

Methods | Training Parameters | Inference Parameters
---|---|---
| VGG | L3 | Comb. | VGG | L3 | Comb.
Feature-only… | 38k | 136k | 169k | 72.2M | 4.8M | 77.0M
Concat | 3.7M | 3.8M | 3.8M | 75.8M | 8.4M | 80.6M
FiLM | 3.7M | 3.9M | 4.0M | 75.8M | 8.6M | 80.8M
Con-Reg | 3.7M | 3.8M | 3.8M | 3.6M | 3.6M | 3.6M
Dis-Reg | 3.6M | 3.6M | 3.6M | 3.6M | 3.6M | 3.6M
TABLE II: Number of training and testing parameters for both proposed and
baseline methods.

###  IV-A Model Complexity

Table II shows the number of training and inference parameters for both the
proposed and the baseline methods. Since audio files can be pre-processed to
extract the learned features, the _feature-only_ method only needs the decoder
during training and has, therefore, the smallest number of trainable
parameters. In contrast, the same _feature-only_ method requires a large
number of parameters during inference due to the large-scale VGGish/OpenL3
extractor. A similar imbalance between the number of training and inference
parameters can be observed for the _Concat_ and _FiLM_ methods.

Comparing the baseline results from Table I with the number of baseline
parameters listed in Table II we can conclude that the better the result, the
more parameters the extractor needs. For example, VGGish features achieve
better performance than OpenL3 but its extractor is also more complex. As
mentioned above, the combination of pre-trained features can achieve better
performance, but the parameters for inference increase considerably as they
require two extractors (both VGGish and OpenL3) during inference.

The proposed regularization methods have a similar number of parameters as the
original model proposed by Won et al. [21] (approx. 3.6M), for both training
and inference. This is because our proposed methods only use features in the
additional regularization loss function without modifying the model
architecture.

##  V Conclusion

In this work, we proposed two novel regularization methods to incorporate the
information from pre-trained features during training by adding an additional
loss term restructuring the embedding space. The proposed regularization
methods show improvements compared to baseline feature integration methods,
which either directly use pre-trained features or directly influence the model
through concatenation or a feature-wise linear transform. Furthermore, the
regularized models can outperform SOTA audio classification models, especially
having a more pronounced performance increase in the case of limited training
data.

Future work will include experiments on other features, such as Jukebox [8]
and Musicnn [34], as well as the exploration of different interpolation
methods to deal with the various time resolutions. We also plan to extend our
experiments to non-audio-related fields such as NLP and computer vision.

## References

  * [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019, pp. 4171–4186.
  * [2] T. Falke, L. F. Ribeiro, P. A. Utama, I. Dagan, and I. Gurevych, “Ranking generated summaries by correctness: An interesting but challenging application for natural language inference,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 2214–2220.
  * [3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” _Advances in Neural Information Processing Systems_ , vol. 25, pp. 1097–1105, 2012.
  * [4] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015.
  * [5] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3431–3440.
  * [6] J. Cramer, H.-H. Wu, J. Salamon, and J. P. Bello, “Look, listen, and learn more: Design choices for deep audio embeddings,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2019, pp. 3852–3856.
  * [7] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, C. Moore, M. Plakal, D. Platt, R. Saurous, B. Seybold, M. Slaney, R. A. Weiss, and K. Wilson, “CNN architectures for large-scale audio classification,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2017.
  * [8] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” _arXiv preprint arXiv:2005.00341_ , 2020.
  * [9] S. Gururani, M. Sharma, and A. Lerch, “An attention mechanism for musical instrument recognition,” in _Proceedings of International Society for Music Information Retrieval Conference_ , 2019, pp. 83–90.
  * [10] Y. Yu, S. Tang, F. Raposo, and L. Chen, “Deep cross-modal correlation learning for audio and lyrics in music retrieval,” _ACM Transactions on Multimedia Computing, Communications, and Applications_ , vol. 15, no. 1, pp. 1–16, 2019.
  * [11] D. Zeng, Y. Yu, and K. Oyama, “Audio-visual embedding for cross-modal music video retrieval through supervised deep CCA,” in _International Symposium on Multimedia_ , 2018, pp. 143–150.
  * [12] K. Koutini, H. Eghbal-zadeh, and G. Widmer, “Receptive field regularization techniques for audio classification and tagging with deep convolutional neural networks,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 1987–2000, 2021.
  * [13] R. Castellon, C. Donahue, and P. Liang, “Codified audio language modeling learns useful representations for music information retrieval,” in _Proceedings of International Society for Music Information Retrieval Conference_ , 2021.
  * [14] E. Koh and S. Dubnov, “Comparison and analysis of deep audio embeddings for music emotion recognition,” _AAAI Workshop on Affective Content Analysis_ , 2021.
  * [15] S. Amiriparian, M. Gerczuk, E. Coutinho, A. Baird, S. Ottl, M. Milling, and B. Schuller, “Emotion and themes recognition in music utilising convolutional and recurrent neural networks,” in _CEUR Workshop Proceedings_ , vol. 2670, 2019.
  * [16] J. Ramírez and M. J. Flores, “Machine learning for music genre: multifaceted review and experimentation with audioset,” _Journal of Intelligent Information Systems_ , vol. 55, no. 3, pp. 469–499, 2020.
  * [17] Q. Huang, A. Jansen, L. Zhang, D. P. Ellis, R. A. Saurous, and J. Anderson, “Large-scale weakly-supervised content embeddings for music recommendation and tagging,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 8364–8368.
  * [18] D. Tompkins and E. Nichols, “DCASE 2019 challenge task 5: CNN+ vggish,” _Technical Report, DCASE2019 Challenge_ , 2019.
  * [19] Z. Xu, P. Smit, and M. Kurimo, “The Aalto system based on fine-tuned audioset features for dcase2018 task2 - general purpose audio tagging,” in _Technical Report, DCASE2018 Challenge_ , 2018.
  * [20] M. E. Davies, N. Degara, and M. D. Plumbley, “Evaluation methods for musical audio beat tracking algorithms,” _Queen Mary University of London, Centre for Digital Music, Tech. Rep. C4DM-TR-09-06_ , 2009.
  * [21] M. Won, S. Chun, O. Nieto, and X. Serra, “Data-driven harmonic filters for audio representation learning,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 536–540.
  * [22] C. Ick and B. McFee, “Sound event detection in urban audio with single and multi-rate PCEN,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021, pp. 880–884.
  * [23] J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” _International Journal of Computer Vision_ , vol. 129, no. 6, pp. 1789–1819, 2021.
  * [24] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra _et al._ , “Matching networks for one shot learning,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [25] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4367–4375.
  * [26] C. Hansen, C. Hansen, L. Maystre, R. Mehrotra, B. Brost, F. Tomasi, and M. Lalmas, “Contextual and sequential user embeddings for large-scale music recommendation,” in _Proceedings of ACM Conference on Recommender Systems_ , 2020, pp. 53–62.
  * [27] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2017, pp. 776–780.
  * [28] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 1–9.
  * [29] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in _Proceedings of AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [30] O. Slizovskaia, L. Kim, G. Haro, and E. Gomez, “End-to-end sound source separation conditioned on instrument labels,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2019, pp. 306–310.
  * [31] J. W. Kim, R. Bittner, A. Kumar, and J. P. Bello, “Neural music synthesis for flexible timbre control,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2019, pp. 176–180.
  * [32] E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” in _Proceedings of International Society for Music Information Retrieval Conference_ , 2009, pp. 387–392.
  * [33] A. Mesaros, T. Heittola, A. Diment, B. Elizalde, A. Shah, E. Vincent, B. Raj, and T. Virtanen, “DCASE 2017 challenge setup: Tasks, datasets and baseline system,” in _DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events_ , 2017.
  * [34] J. Pons and X. Serra, “musicnn: Pre-trained convolutional neural networks for music audio tagging,” in _Proceedings of International Society for Music Information Retrieval Conference_ , 2019.