# Word Embeddings for Automatic Equalization in Audio Mixing

Satvik Venkatesh<EMAIL_ADDRESS>Interdisciplinary Centre for
Computer Music Research, University of Plymouth, Plymouth, UK. David Moffat
<EMAIL_ADDRESS>Plymouth Marine Laboratory, Plymouth, UK.  Eduardo Reck Miranda
<EMAIL_ADDRESS>
Research supported by Engineering and Physical Sciences Research Council
(EPSRC) grant EP/S026991/1.
Submitted to Journal of Audio Engineering Interdisciplinary Centre for
Computer Music Research, University of Plymouth, Plymouth, UK.

## Abstract

In recent years, machine learning has been widely adopted to automate the
audio mixing process. Automatic mixing systems have been applied to various
audio effects such as gain-adjustment, stereo panning, equalization, and
reverberation. These systems can be controlled through visual interfaces,
providing audio examples, using knobs, and semantic descriptors. Using
semantic descriptors or textual information to control these systems is an
effective way for artists to communicate their creative goals. Furthermore,
sometimes artists use non-technical words that may not be understood by the
mixing system, or even a mixing engineer. In this paper, we explore the novel
idea of using word embeddings to represent semantic descriptors. Word
embeddings are generally obtained by training neural networks on large corpora
of written text. These embeddings serve as the input layer of the neural
network to create a translation from words to EQ settings. Using this
technique, the machine learning model can also generate EQ settings for
semantic descriptors that it has not seen before. We perform experiments to
demonstrate the feasibility of this idea. In addition, we compare the EQ
settings of humans with the predictions of the neural network to evaluate the
quality of predictions. The results showed that the embedding layer enables
the neural network to understand semantic descriptors. We observed that the
models with embedding layers perform better those without embedding layers,
but not as good as human labels.

##  1 Introduction

The process of audio production involves multiple tasks such as balancing
sound levels and applying audio effects. An audio effect can be defined as a
function that transforms sound based on a set of controlled parameters [1].
Audio production is needed in various domains such as making albums, films,
and theatre works, to name a few. It is generally carried out by a mixing
engineer who understands the goals of their client. The mixing engineer blends
multiple tracks together by modifying acoustic properties such as dynamics and
timbre [2]. A vast body of research has been exploring how this process can be
automated through the use of _intelligent tools_ [3, 4, 5, 6]. Traditional
Artificial Intelligence (AI) approaches such as expert systems have been
adopted to create autonomous mixing tools [3]. These systems are knowledge-
engineered and adopt a set of rules for mixing depending on the scenario.
However, recent research has grown towards using Machine Learning and Deep
Learning for automatic mixing. On one hand, some studies have focused on
specific areas, such as gain balancing [7] and reverberation [8]. On the other
hand, some have explored building autonomous systems where the entire mixing
process is carried out without human intervention [2, 9].

An equalizer (EQ) is an audio effect created by cascading multiple filters in
series [10]. Timbral adjectives often have a correlation with the parameter
setting for the equalizer. Some examples include, _add air_ , _make it warmer_
, and _make is less muddy_ [11]. Kulka et al. [12] associated adjectives such
as _warmth_ , _honk_ , _crunch_ , and _sibilance_ with frequencies of 125,
500, 2000, and 8000 Hz respectively. For example, according to the Kulka rule,
if the mix sounds honky, cut the region around 500 Hz.

When clients such as instrumentalists and musical directors work with mixing
engineers, they often use semantic descriptors to describe their goals. For
example,_“make the violin sound warmer”_ [13]. It is the role of the mixing
engineer to understand these descriptors mentioned by the client. Popular
semantic descriptors such as _warm_ and _bright_ are easily understood by the
mixing engineer [14]. To expand the vocabulary of such descriptors, studies
have also tried to create a thesaurus with synonyms and antonyms. For example,
significant synonyms of _boom_ are _boxy, dull, and fat_ and significant
antonyms of _boom_ are _air, bright, and crisp_ [11]. However, the problem
arises when individuals without training in audio production describe their
creative goals [15]. They may have ideas that cannot be directly translated
into a studio engineer’s vocabulary.

To address this issue of non-technical descriptors, Cartwright et al. [13]
presented a dataset called SocialEQ, which is a web-based project that adopts
crowd-sourcing to learn a vocabulary of audio descriptors. As it is crowd-
sourced, the study focuses on aggregating a vocabulary to enable non-technical
individuals to describe their sonic goals. Crowd-sourcing was also adopted to
build the datasets for other effects like reverberation [16] and dynamic range
compression [15].

There is a growing interest in adopting natural language processing (NLP)
methodologies to develop semantically-controlled audio effects [17, 18, 19].
Stables et al. [20] presented a system called Semantic Audio Feature
Extraction (SAFE), which focused extracting semantic descriptions for
equalization from a digital audio workstation (DAW). Stasis et al. [21]
investigated the idea of mapping the descriptors to a reduced dimensionality
space, to enable users to interact with the system in a more intuitive way.
Chourdakis et al. [22] explored tagging and retrieval of room impulse
responses for reverberation. They adopted word embeddings to assign impulse
responses to tags that match their short descriptions.

In this paper, we explore the novel idea of adopting word embeddings to
automatically predict EQ settings. We present a methodology to translate words
from a semantic vector space to a vector space representing the parameters of
an equalizer. Word embeddings are representations of words that capture
lexical semantics in language [23]. An embedding layer is often used as the
first layer in a neural network that performs NLP tasks such as machine
translation, caption generation, and automatic speech recognition [24]. We
adopt this approach to translate words to predict values of a parametric
equalizer. This way, the neural network has the ability to understand non-
technical words and even descriptors that it has not seen before. This finding
is significant because artists without training in audio production can
express their creative goals directly to the AI-powered mixing engine.

##  2 Methodology

###  2.1 Dataset

We adopted the SocialEQ dataset [13], which crowd-sources semantic descriptors
for EQ settings. In the raw format, each sample in the dataset contains a
semantic descriptor, language of the descriptor, audio id, a consistency
rating, and 40 values for EQ parameters. During the data collection, each
participant was asked to enter a word in their preferred language. For
example, _warm_ in English, _claro_ in Spanish, or _grave_ in Italian.
Subsequently, they pick a sound file which will be modified by the EQ plugin.
There were three sound files — electric guitar, piano, and drums. Each sound
file had a unique audio id.

After selecting a descriptive term and audio file, the participant was
presented with 40 different modifications of the sound file made by different
EQ settings. Suppose the user has selected _warm_ , they are asked to rate
_how warm that sound is_. Out of the 40 modifications, there are 15
repetitions to test for consistency. The system processes the ratings of the
user and develops a relative boost/cut for 40 different frequency bands. Refer
to Cartwright and Pardo [13] for more details on the dataset.

The dataset has 1595 samples in it. For simplicity, we considered only
descriptors in English. The number of examples in English was 918. It is
important to note that the dataset contained examples with different EQ
parameter settings for the same word. Thus, the number of unique descriptors
in English was 388.

###  2.2 Train-Test Split

An important hypothesis we wanted to test in this paper is that a word
embedding layer helps a model predict EQ parameter settings for semantic
descriptors it has not seen before. Therefore, words in the test set should
not appear in the training set. We adopted a four-fold cross-validation setup
[25] and the strategy is explained below.

We aggregated a list of semantic descriptors that are common in the audio
mixing literature. We labelled these as High Quality (HQ) words. In order to
avoid bias and objectively choose these words, we selected those that were
already listed in table 4.8 in Ref. [11]. Additionally, we included semantic
descriptors that fell under the hierarchical ontology presented by Pearce et
al. [26]. The list of HQ words are presented in bold in table 1. There are 32
HQ words present in the SocialEQ dataset.

| Fold 1 | Fold 2 | Fold 3 | Fold 4
---|---|---|---|---
| Semantic
---
Descriptors
|  smooth, muffled, crisp,
---
punch, clean, brittle,
muddy, soothing, clear,
brassy, caring, mellow,
throbbing, cooing, fluffy,
good, excited, squeaking,
punchy, funky, whispered,
disgusting, beautiful,
reserved, serene, thumpy,
pleasurable, whispering,
gentle, energetic, peace
|  crunchy, woody, flat,
---
metallic, dull, tinny,
cold, booming, deep,
energizing,
heart-warming, edgy,
heavy, edge, strong,
enchanting, cheerful,
plodding, quiet,
radiant, biting, brass,
pleasing, light, taco,
gruff, exciting, love,
heat, techno, solemn
|  sweet, warm, airy,
---
full, boxy, bright,
boom, fat, shrill,
calm, velvety, hard,
rich, noisy, down,
rumble, sloppy,
relaxing, peaceful,
romantic, low, hot,
thunderous, frigid,
happy, poor, cool,
tense, jagged,
forceful, aggressive
| sharp, big, dark,
---
hollow, harsh, smooth,
muffled, crisp, punch,
mournful, clarity,
genius, bold, twangy,
soft, splash, slow,
wistful, brash, fancy,
cute, rousing, loud,
breezy, large,
passionate, baseball,
huge, icy, brassy,
caring

Table 1: Four cross-validation folds from the dataset. The test words from
each fold are presented in the table. For each fold, the training set consists
of words that are not in the test set.

We also aggregated a list of words that were Highly-Rated (HR). HR words need
not semantically meaningful, but have a high consistency score in the dataset.
Words that have a consistency score greater than 0.7 were selected as HR
words. As these words have a high consistency score, the user strongly
associated the semantic word with a particular EQ setting. Words in table 1
that are not formatted as bold text are HR words. Totally, 86 HR words were
present in the SocialEQ dataset.

Each test fold contained 9 HQ words and 22 HR words. We ensured that every HQ
and HR word was tested at least once. In the last test fold, there may be a
few repetitions of words from the first test fold. There was no overlap
between the training set and test set. The test set only contained words that
were not present in the training set. Note that the network for each fold is
trained as a separate experiment. In other words, the network is totally
trained four times and tested four times on different folds and we report the
average performance.

As mentioned earlier, each word can have multiple EQ settings. Each setting is
a separate example and can have different consistency scores. In the test set,
we only included examples that had a consistency score of greater than 0.7. In
the training set, we did not exclude any words based on the consistency score.

###  2.3 Word Embeddings

A vocabulary consists of all the possible words that the neural network can
understand. Generally, a word is converted into a one-hot encoded vector
before passing into the neural network. For instance, in the SocialEQ dataset,
there are 388 unique words, which means that the size of the vocabulary is
388. Therefore, the dimensions of the one-hot encoded vector are $1\times
388$. Each position within the vector is assigned to a unique word. Thus, the
respective position of the word is labelled as 1 and the remaining elements
are 0. However, it is important to note that the Euclidean distance between
any pair of words is equal. As each word is equidistant from each other, the
neural network would not develop the capability of handling words that is not
present in the training set. For example, let us consider the semantic
descriptor _bright_ and assume that it is present in the training set. Let us
also assume that _clear_ and _boom_ are words in the test set. According to
Spyridon [11], _clear_ is a synonym of _bright_ and _boom_ is an antonym of
_bright_. Thus, we expect similar EQ settings for _clear_ and _bright_ , but
considerably different EQ settings _boom_ and _bright_. However, the neural
network cannot perceive this understanding unless it has seen all three words
because each word is equidistant from each other. Furthermore, this issue
becomes exaggerated if a non-technical user is adopting a semantic descriptor
that is not common in the audio mixing literature.

The purpose of a word embedding layer is to convert a one-hot encoded
representation into a vector space of reduced dimensionality. They are useful
for NLP tasks such as machine translation. Large vocabularies with millions of
words can be reduced to a 300-dimensional vector representation [27]. The
distances between words in the embedding space is governed by some form of
semantic correlation. Examples include synonyms or two words frequently
occurring together. There are different algorithms to train word embedding
models. Some of them include Word2Vec [28], GloVe [27], ConceptNet [29], and
Dict2Vec [30]. Each of these algorithms present unique methods to train on
large corpora of text such as Wikipedia. Effectively, they try to learn
semantic relationships between words and represent them through an embedding
vector.

For this study, we investigated four different embedding models — GloVe-6B,
GloVe-840B, Tok2Vec, and Dict2Vec. GloVe is an unsupervised learning algorithm
for obtaining vector representations for words [27]. GloVe-6B refers to the
model that was trained on Wikipedia 2014 and Gigaword 5. It includes 6B tokens
and a vocabulary size of 400k (B, M, k stand for Billion, Million, and
thousand respectively). On the other hand, GloVe-840B uses 840B tokens and a
vocabulary size of 2.2M. It trains on the World Wide Web using Common Crawl,
which is a larger corpus of text. Tok2Vec is a word embedding model provided
by a company called spaCy [31]. We did not find the entire details regarding
its implementation, but the model is publicly available and free to use. It is
important to note that word embeddings are used for NLP tasks, which are
designed to accept sentences. In our application, we are considering only one
word, which is the semantic descriptor. As GloVe and Tok2Vec also focus on the
ordering of words in sentences, we thought it is a good idea to consider
another embedding model called Dict2Vec [30]. Dict2Vec is an embedding model
that uses lexical dictionaries. It builds new word pairs from dictionary
entries so that semantically-related words are closer to each other in the
embedding space [30]. Similar to GloVe-6B, it was trained on the Wikipedia
corpus.

###  2.4 Machine Learning Architecture

####  2.4.1 Word Embedding Layer

We evaluated four different pre-trained word embedding models in the study —
GloVe-6B, GloVe-840B, Tok2Vec, and Dict2Vec. All the models represent words
with 300-D semantic vectors. This is convenient because we can adopt the same
neural network architecture to compare different embeddings. Initially, a word
is converted into a one-hot encoded representation. Subsequently, an embedding
matrix converts this one-hot encoded representation into a 300-D semantic
vector. Then, this vector is connected to hidden layers in the network. Note
that the weights of the embedding matrix are frozen and the layer is not
trainable. We did not consider setting this to trainable because of the
limited data we have.

####  2.4.2 Hidden Layers

Layer type | Units | Activation | Output shape
---|---|---|---
Embedding | - | - | 300
Dense | 300 | ReLu | 300
Dense | 200 | ReLu | 200
Dense | 100 | ReLu | 100
Dense | 80 | ReLu | 80
Dense | 60 | ReLu | 60
Dense | 40 | Sigmoid | 40
Table 2: The neural network architecture

The neural network aims to translate a representation of word embeddings to a
prediction of equalizer parameters. Therefore, our network needs to be deep
enough to learn the translation between two domains. Deeper networks apply the
non-linear activation more number of times on the input and therefore have the
advantage of learning more complex translations. However, it is important to
note that our dataset is relatively small for our task.

All the layers in the neural network were fully connected layers. Table 2
shows an overview of the architecture. After the embedding layer, we had a
series of fully connected layers. The number of hidden units in these layers
were 300, 200, 100, 80, and 60 respectively. Finally, it was connected to an
output layer with 40 units. Excluding the final layer, all the hidden layers
were fitted with ReLu activations and a dropout of 0.1. The output layer is
explained in section 2.4.4. The code and trained models associated with this
study can be found in this GitHub repository111https://github.com/satvik-
venkatesh/word-eq.

####  2.4.3 Normalisation

Traditional min-max normalisation by calculating the maximum and minimum in
the training set was not appropriate for our dataset. This is because if there
exists any outliers amongst the values in the test set, specific features may
get magnified or diminished. Furthermore, as we are predicting values for 40
EQ bands, this issue becomes more crucial. Therefore, we fixed the minimum and
maximum value for each EQ parameter to -4 dB and +4 dB respectively. In other
words, the highest cut/boost within each EQ band was 4 dB. The values were
linearly normalised to the range of 0 to 1. Hence, -4 dB would correspond to 0
and +4 dB would correspond to 1 in the output layer.

####  2.4.4 Output Layer and Loss Function

The output layer of the network contained 40 neurons, with each of them
predicting a value for one EQ band. As we normalised the data within the range
of 0 to 1, we used sigmoid activation functions for the output neurons. As we
are working with a regression problem, we adopted the mean absolute error loss
function, which is commonly adopted by many studies. All EQ bands were given
equal importance when averaging the error for the loss function. In future
work, it would be interesting to weight the EQ bands based on perceptual
frequency band weights. However, that is beyond the scope of this study.

##  3 Results

###  3.1 Error

Word Embedding | Error
---|---
Tok2Vec | 0.760 $\pm$ 0.055
Glove-840 |  0.770 $\pm$ 0.032
Dict2Vec |  0.792 $\pm$ 0.058
Glove-6B |  0.798 $\pm$ 0.046
No Embedding |  0.836 $\pm$ 0.016
Table 3: The error calculated across four folds. The smallest error in the
column is indicated in bold.

Table 3 shows the mean absolute error for different embedding models
calculated across four test folds. As we can see, Tok2Vec obtains the lowest
error rate of 0.76, followed by GloVe-840 with an error rate of 0.77.
GloVe-840 obtains an error lower than GloVe-6B, which conveys that it
benefited from training on a larger corpus. Dict2Vec and GloVe-6B were trained
on similar dataset sizes and the former obtained a better error rate. This
suggests that the performance of Dict2Vec can be improved with training on a
larger corpus of text.

The ‘No Embedding’ model in table 3 means that no word embedding layer was
used in the neural network. The input of the network was a direct one-hot
encoded representation. All the neural networks with word embeddings performed
better than the model without word embeddings. However, the difference was not
huge. The best model was Tok2Vec with an error rate of 0.76 vs ‘No Embedding’
with an error rate of 0.836. This is possibly due to two reasons. Firstly,
error may not be the best metric for our task. For example, the semantic word
_warm_ may have a boost of 1.2 dB at 260 Hz. But, the neural network may
predict a boost at the adjacent EQ band, such as 317 Hz. Although, the error
rate in this case is high, the EQ effect applied to the audio may still be
semantically meaningful. Secondly, the test set contains many semantic
descriptors that occur only once. These examples may be highly subjective to
one individual, despite having a high consistency score. Therefore, in the
next subsection, we evaluate the top two performing models using Partial Curve
Mapping (PCM) [32], which is a method to quantify the similarity between two
curves. For instance, this technique is generally adopted to analyse
similarities between hysteresis curves pertaining to a magnetic field.
Although this technique may not be ideal for our task, it would give us a
better understanding of our model’s performance compared to mean absolute
error.

###  3.2 Partial Curve Mapping

Figure 1: Distances obtained by different models calculated by using Partial
Curve Mapping (PCM). An ideal algorithm would have a distance of zero. Figure
2: Plots of human labels alongside EQ parameters predicted by GloVe-840,
Tok2Vec, and no embedding. These are for words in test folds 1 and 2. Note
that that each word in the test set does not occur in the training set. The
first two rows occur in fold 1 and the last two rows occur in fold 2. The
human label plotted for a semantic word was the EQ settings with the highest
consistency score in the dataset.

In this section, we evaluate the models using PCM. PCM was implemented using
this Python package [33]. We also compare our model to human labels. As
mentioned earlier, each semantic descriptor had multiple EQ settings in the
dataset. To calculate the error in human labels, we considered the mean of the
different EQ settings as the ground truth. However, words that occur only once
in the dataset would not have an error associated with it. These words would
artificially reduce the average error. Hence, we only included words that
occur at least twice in the dataset. Figure 1 shows the distances for
different models. An ideal algorithm would obtain a distance of zero. Human
labels obtain the smallest distance of 2.9, which is an expected observation.
GloVe and Tok2Vec obtain similar distances with the former performing slightly
better. The distances were 9.3 and 10.5 respectively. Note that for this
experiment, we only considered words that occur at least twice, which is
different from results presented in section 3. The mean distance of the model
with no embeddings was 35.4, which was considerably higher. In addition, there
was a much larger standard deviation for this model, which suggests that it
was randomly guessing.

###  3.3 Plots of EQ Parameters

In this section, we perform an error analysis of predictions made by the
machine learning models. We look at individual test words to investigate if
the neural network is actually learning semantic meanings of EQ settings. We
predominantly look at HQ words as they are common in the audio mixing
literature and would be more intuitive to evaluate. In figure 2 and 3, we plot
the EQ settings of human labels alongside the predictions of Tok2Vec,
Glove-840B, and ‘no embedding’. As the literature does not comprise an ‘ideal’
metric for our task of predicting EQ parameters, we thought it is a good idea
to plot graphs and actually visualise the predictions of the algorithms.
Figure 2 plots the graphs for words selected from the test folds 1 and 2.
Figure 3 plots the graphs for words selected from the test folds 3 and 4. Note
that for each word in the test folds, the neural network has not encountered
the word in the training set. The human label chosen for each semantic word in
the plots was the EQ setting with the highest consistency score in the
dataset.

In figure 2, human labels for _muffled_ had boosts at 20 Hz and 3.5 kHz. For
Tok2Vec and GloVe, we saw slight boosts in the mid-rage and high-rage
respectively, which may convey that the neural networks did not interpret this
word correctly. The network with ‘no embedding’ was a basically a flat curve
for all the words in the first fold. For _crisp_ , interestingly, the
predictions of Tok2Vec and GloVe did follow a similar pattern as the human
labels. In the human labels, we saw boosts at 2.1k and 9k. For GloVe and
Tok2Vec, we saw a gradual boost at 3k, which lifts the high-range of the
frequency spectrum. Some semantic synonyms of _crisp_ present in the training
set for this respective fold include _bright_ , _harsh_ , _hollow_ , and
_sharp_. This means that the word embedding has delineated a relationship
between the semantic word and EQ predictions. Again, as mentioned earlier, we
did not observe a meaningful pattern in the neural network with ‘no embedding’
because the curves were flat.

_Muddy_ had a gradual boost from 200 to 380 Hz in the human labels. Tok2Vec
follows a very similar pattern in its prediction by boosting the lows and
cutting the highs. GloVe’s prediction has slightly boosted lows and high,
which is not convincing for the semantic word _muddy_. Some semantic synonyms
in the training set include _boom_ , _muddled_ , _dark_ , _dull_ , and _fat_.
The next test word, _brittle_ was well-understood by both Tok2Vec and GloVe.
There was considerable overlap with the human labels. The synonyms for
_brittle_ in the training set would be similar to those listed for _crisp_.
_Punchy_ was understood GloVe, but not by Tok2Vec. _Gentle_ was not understood
by both embedding models222If the reader is interested in more semantic
synonyms present in the training set, please refer to table 1. If fold 1 is
selected as the test set, folds 2, 3, and 4 are included in the training set..

_Crunchy_ had boosts in the low and high frequency range in the human labels.
We observe a boost for GloVe and Tok2Vec in the high range. The ‘no embedding’
model has a boost in the low range. However, if you observe, it has made the
same prediction for all the test words in the second fold. GloVe and Tok2Vec
and correctly understood the semantic descriptor _metallic_ and have
significant overlap with human labels. For _tinny_ , human labels have boosts
at 1.3k and 9k. Whereas, the neural networks with embeddings have a gradual
boost around 3k. We are not certain if these predictions would have a _tinny_
effect. For test words _enchanting_ and _deep_ , we observed a noticeable
overlap with human labels. However, for _cold_ , it seems as though GloVe and
Tok2Vec predicted the antonym.

Figure 3: Plots of human labels alongside EQ parameters predicted by
GloVe-840, Tok2Vec, and no embedding. These are for words in test folds 3 and
4. Note that that each word in the test set does not occur in the training
set. The first two rows occur in fold 3 and the last two rows occur in fold 4.
The human label plotted for a semantic word was the EQ settings with the
highest consistency score in the dataset.

In test fold 3, _sweet_ was not understood by the networks at all. For _warm_
, Tok2Vec has a noticeable overlap with the human labels because both have a
boost of approximately 2 dB in the low frequency range. _Airy_ was partially
convincing because GloVe recognised a boost at 9 kHz. Although, the networks
have boosted the lows for _full_ , it seems like a random guess as the
prediction significantly overlaps with the one made by ‘no embedding’. The
predictions made by the networks for _boxy_ were not convincing. _Bright_
seemed plausible with Tok2Vec and GloVe boosting the high frequency range.

In test fold 4, we saw reasonable overlap for _sharp_ , _dark_ , _hollow_ ,
and _harsh_. We did not observe a reasonable pattern for _breezy_ and
_smooth_. Interestingly, the network with ‘no embedding’ predicted the EQ
settings for _harsh_ correctly. This is a chance occurrence because the ‘no
embedding’ model predicted a standard template of settings for all the other
words.

##  4 Discussion

The results presented in the previous section show that a word embedding layer
is helpful for automatic mixing. We analysed the error of models in section 3.
All the models with an embedding layer obtained lower errors than the one
without an embedding layer. We further analysed the performance of GloVe-840B
and Tok2Vec by using partial curve mapping. The mean distances obtained by
human labels, Tok2Vec, GloVe, and ‘no embedding’ were 2.9, 10.5, 9.3, and 35.4
respectively. This objectively demonstrates that the embedding models perform
better than models without an embedding layer, but definitely not as good as
human labels.

In section 3.3, we conducted an error analysis of predictions made by GloVe
and Tok2Vec. We observed that the machine learning models were able to
understand semantic descriptors that it had not encountered before. This is a
promising step towards understanding semantic descriptors from non-technical
users. It is important to note the word embedding layers used in the networks
were trained on corpora of written text. This concludes that there exists some
common ground for semantic relationships between words in written text and for
those adopted in EQ mixing.

Considering the fact that we have adopted such a small training dataset, this
performance is reasonable. The SocialFX dataset comprises only 388 unique
English words. Additionally, many of the high-quality and highly rated words
were used for testing in each fold. As our study has demonstrated that word
embeddings are helpful for automatic EQ mixing, we hope to encourage
researchers the build larger datasets with semantic descriptors. In the
literature, another dataset called SAFE [20] focused on extracting semantic
descriptions for equalization from a DAW. We were unable to include the
dataset within this study for two reasons. Firstly, as these are extracted
directly from the DAW without post-processing, some labels can be noisy.
Although the dataset contains many examples with meaningful descriptors, some
words are randomly typed letters such as ‘xy’, which have no semantic meaning.
Perhaps, this noise may not matter when training the network with large-scale
data. The second reason is that both datasets use different EQ plugins. The
SocialFX dataset uses a 40 band EQ, whereas the SAFE dataset uses a five band
EQ. We are not certain if additional noise would be induced in mapping one EQ
domain to the other.

In this study, we analysed the performance of the machine learning model using
objective metrics. However, it is important to perform listening tests with
human participants to obtain subjective evaluations of the system. We need to
investigate if users are satisfied with the way the machine learning model
understands their semantic descriptors. After aggregating a larger dataset for
this task, this could be a potential future pathway.

##  5 Conclusion

In this paper, we demonstrated the feasibility of adopting word embeddings for
automatic EQ mixing. We showed that the word embedding layer is capable of
providing relationships between semantic descriptors, which assists in
predicting EQ parameters. Using this technique, the machine learning model can
predict EQ settings for words it has not seen before. This is a step towards
bridging the gap between artists explaining their creative goals and mixing
engineers understanding them.

In this study, we looked at EQ parameters as a separate entity. This may not
be ideal in some scenarios. For example, the EQ settings for a drum track may
differ from a vocal track. In other words, the EQ settings for “make the
vocals sound brighter” maybe different from “make the drums sound brighter”.
Moreover, the number of EQ bands predicted were 40. This number is pretty
large for a network that performs regression. Future research could explore
how the neural network architecture can be optimised and regularised better.
Furthermore, it may be interesting to augment the size of training sets by
adopting well-known synonyms and antonyms in the mixing engineer’s vocabulary.

For some words, Tok2Vec captured relationships, but GloVe did not and vice
versa. For example, GloVe captured the meaning of _punchy_ as shown in figure
2 and Tok2Vec captured the meaning of _warm_ as shown in figure 3. This may be
simply because there is limited data in the training set. Otherwise, different
embedding models may capture different aspects of semantic relationships.
Therefore, an ensemble of different embedding models will improve performance
in this case. Furthermore, in our study, we discarded non-English words for
simplicity. Word embedding models such as ConceptNet [29] use a knowledge
graph to connect words from different languages. This may be an interesting
avenue to explore.

##  6 Acknowledgment

This paper is supported by EPSRC Grant EP/S026991/1 RadioMe: Real-time Radio
Remixing.

## References

  * [1] T. Wilmering, D. Moffat, A. Milo, and M. B. Sandler, “A history of audio effects,” Applied Sciences, vol. 10, no. 3, p. 791, 2020.
  * [2] M. M. Ramirez, D. Stoller, and D. Moffat, “A deep learning approach to intelligent drum mixing with the wave-u-net,” Journal of the Audio Engineering Society, vol. 69, no. 3, pp. 142–151, 2021.
  * [3] B. De Man and J. D. Reiss, “A knowledge-engineered autonomous mixing system,” in Audio Engineering Society Convention 135, Audio Engineering Society, 2013\.
  * [4] B. De Man, R. Stables, and J. D. Reiss, IntelligentMusic Production.  Focal Press, 2019.
  * [5] D. Moffat, F. Thalmann, and M. B. Sandler, “Towards a semantic web representation and application of audio mixing rules,” in Proceedings of the 4th Workshop on Intelligent Music Production, Huddersfield, UK, 2018.
  * [6] E. Perez-Gonzalez and J. Reiss, “Automatic gain and fader control for live mixing,” in 2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 1–4, IEEE, 2009.
  * [7] D. Moffat and M. Sandler, “Machine learning multitrack gain mixing of drums,” in Audio Engineering Society Convention 147, Audio Engineering Society, 2019\.
  * [8] E. T. Chourdakis and J. D. Reiss, “A machine-learning approach to application of intelligent artificial reverberation,” Journal of the Audio Engineering Society, 2017.
  * [9] D. Moffat and M. B. Sandler, “Approaches in intelligent music production,” in Arts, vol. 8, p. 125, Multidisciplinary Digital Publishing Institute, 2019\.
  * [10] E. Tarr, Hack Audio: An Introduction to Computer Programming and Digital Signal Processing in MATLAB.  Routledge, 2018.
  * [11] S. Spyridon, Audio equalisation using natural language.  PhD thesis, Birmingham City University, 2019.
  * [12] L. d. G. Kulka, “Equalization-the highest, most sustained expression of the recordist’s heart,” Recording Engineer/Producer, vol. 3, 1972.
  * [13] M. B. Cartwright and B. Pardo, “Social-eq: Crowdsourcing an equalization descriptor map.,” in ISMIR, pp. 395–400, 2013.
  * [14] G. Bromham, D. Moffat, M. Barthet, A. Danielsen, and G. Fazekas, “The impact of audio effects processing on the perception of brightness and warmth,” in ACM Audio Mostly Conference, (Nottingham, UK), ACM, September 2019.
  * [15] T. Zheng, P. Seetharaman, and B. Pardo, “Socialfx: Studying a crowdsourced folksonomy of audio effects terms,” in Proceedings of the 24th ACM international conference on Multimedia, pp. 182–186, 2016.
  * [16] P. Seetharaman and B. Pardo, “Crowdsourcing a reverberation descriptor map,” in Proceedings of the 22nd ACM international conference on Multimedia, pp. 587–596, 2014.
  * [17] A. Zacharakis, K. Pastiadis, J. D. Reiss, and G. Papadelis, “Analysis of musical timbre semantics through metric and non-metric data reduction techniques,” in Proceedings of the 12th International Conference on Music Perception and Cognition (ICMPC12) and the 8th Triennial Conference of the European Society for the Cognitive Sciences of Music (ESCOM 08), pp. 1177–1182, 2012.
  * [18] D. Williams and T. Brookes, “Perceptually-motivated audio morphing: Brightness,” in Audio Engineering Society Convention 122, Audio Engineering Society, 2007.
  * [19] E. R. Miranda, “An artificial intelligence approach to sound design,” Computer Music Journal, vol. 19, no. 2, pp. 59–75, 1995.
  * [20] R. Stables, S. Enderby, B. De Man, G. Fazekas, and J. D. Reiss, “SAFE: A system for extraction and retrieval of semantic audio descriptors,” in 15th International Society for Music Information Retrieval (ISMIR) Conference, 2014.
  * [21] S. Stasis, R. Stables, and J. Hockman, “Semantically controlled adaptive equalisation in reduced dimensionality parameter space,” Applied Sciences, vol. 6, no. 4, p. 116, 2016.
  * [22] E. T. Chourdakis and J. D. Reiss, “Tagging and retrieval of room impulse responses using semantic word vectors and perceptual measures of reverberation,” in Audio Engineering Society Convention 146, Audio Engineering Society, 2019.
  * [23] A. Bakarov, “A survey of word embeddings evaluation methods,” arXiv preprint arXiv:1801.09536, 2018.
  * [24] Y. Goldberg, “Neural network methods for natural language processing,” Synthesis lectures on human language technologies, vol. 10, no. 1, pp. 1–309, 2017.
  * [25] G. Forman and M. Scholz, “Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement,” ACM SIGKKD Explorations Newsletter, vol. 12, no. 1, pp. 49–57, 2010.
  * [26] A. Pearce, T. Brookes, and R. Mason, “Audio Commons: Hierarchical ontology of timbral semantic descriptors,” tech. rep., 2016.  https://www.audiocommons.org/assets/files/AC-WP5-Surrey-D5.1%20Hierarchical%20ontology%20of%20timbral%20semantic%20descriptors.pdf Accessed on 26-11-21.
  * [27] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014.
  * [28] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013\.
  * [29] R. Speer and J. Lowry-Duda, “Conceptnet at semeval-2017 task 2: Extending word embeddings with multilingual relational knowledge,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 85–89, 2017.
  * [30] J. Tissier, C. Gravier, and A. Habrard, “Dict2vec : Learning word embeddings using lexical dictionaries,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 254–263, Association for Computational Linguistics, 2017.
  * [31] M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength natural language processing in python,” 2020.
  * [32] K. Witowski and N. Stander, “Parameter identification of hysteretic models using partial curve mapping,” in 12th AIAA Aviation Technology, Integration, and Operations (ATIO) Conference and 14th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, p. 5580, 2012.
  * [33] C. F. Jekel, G. Venter, M. P. Venter, N. Stander, and R. T. Haftka, “Similarity measures for identifying material parameters from hysteresis loops using inverse analysis,” International Journal of Material Forming, 2019.