LCANets++: Robust Audio Classification using Multi-layer Neural Networks
  with Lateral Competition
Audio classification aims at recognizing audio signals, including speech
commands or sound events. However, current audio classifiers are susceptible to
perturbations and adversarial attacks. In addition, real-world audio
classification tasks often suffer from limited labeled data. To help bridge
these gaps, previous work developed neuro-inspired convolutional neural
networks (CNNs) with sparse coding via the Locally Competitive Algorithm (LCA)
in the first layer (i.e., LCANets) for computer vision. LCANets learn in a
combination of supervised and unsupervised learning, reducing dependency on
labeled samples. Motivated by the fact that auditory cortex is also sparse, we
extend LCANets to audio recognition tasks and introduce LCANets++, which are
CNNs that perform sparse coding in multiple layers via LCA. We demonstrate that
LCANets++ are more robust than standard CNNs and LCANets against perturbations,
e.g., background noise, as well as black-box and white-box attacks, e.g.,
evasion and fast gradient sign (FGSM) attacks.
Sayanton V. Dibbo, Juston S. Moore, Garrett T. Kenyon, Michael A. Teti
# LCANets++: Robust Audio Classification Using Multi-Layer Neural Networks with Lateral Competition

###### Abstract

Audio classification aims at recognizing audio signals, including speech commands or sound events. However, current audio classifiers are susceptible to perturbations and adversarial attacks. In addition, real-world audio classification tasks often suffer from limited labeled data. To help bridge these gaps, previous work developed neuro-inspired convolutional neural networks (CNNs) with sparse coding via the Locally Competitive Algorithm (LCA) in the first layer (i.e., LCANets) for computer vision. LCANets learn in a combination of supervised and unsupervised learning, reducing dependency on labeled samples. Motivated by the fact that auditory cortex is also sparse, we extend LCANets to audio recognition tasks and introduce LCANets++, which are CNNs that perform sparse coding in multiple layers via LCA. We demonstrate that LCANets++ are more robust than standard CNNs and LCANets against perturbations, e.g., _background noise_, as well as black-box and white-box attacks, e.g., _evasion_ and _fast gradient sign (FGSM)_ attacks.

Sayanton V. Dibbo\({}^{\lx@sectionsign\dagger}\)+, Juston S. Moore\({}^{\lx@sectionsign}\), Garrett T. Kenyon\({}^{\lx@sectionsign}\), Michael A. Teti\({}^{\lx@sectionsign}\)

\({}^{\lx@sectionsign}\) Los Alamos National Laboratory, Los Alamos, NM, USA, \({}^{\dagger}\)Dartmouth College, Hanover, NH, USA Audio Classification, Robustness, Neural Networks, Adversarial Machine Learning

Footnote †: 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Sayanton V. Dibbo\({}^{\lx@sectionsign\dagger}\)+, Juston S. Moore\({}^{\lx@sectionsign}\), Garrett T. Kenyon\({}^{\lx@sectionsign}\), Michael A. Teti\({}^{\lx@sectionsign}\)

\({}^{\lx@sectionsign}\) Los Alamos National Laboratory, Los Alamos, NM, USA, \({}^{\dagger}\)Dartmouth College, Hanover, NH, USA

Footnote †: 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

## 1 Introduction

Audio signal classification for the purposes of sound recognition (SR) or sound event detection (SE) has become an active area of research interest [1, 2]. This includes using the Convolutional Neural Network (CNN) models for understanding human speech words, e.g., 'yes','stop', etc., or classifying sound events like 'baby cries', and 'barking'. However, standard CNNs are notoriously susceptible to perturbations or adversarial attacks [3, 4, 5, 6]. Standard audio classification models depend highly on large labeled datasets for better performances [7, 8], but large labeled datasets can be scarce for many common tasks, such as speaker identification. Generating augmented samples for audio data is one proposed approach to mitigate this challenge, but data augmentation can be time-consuming and expensive [9, 10]. Therefore, it is crucial to develop audio classifiers that can learn robust features with limited labeled samples.

Recent studies have shown that CNNs that are more similar to the primary visual cortex are more robust than standard CNNs [11]. Based on this, previous work developed CNNs in which the first layer performed sparse coding via the Locally Competitive Algorithm (LCA) [12, 13, 14], which is a biologically plausible model of the primate primary visual cortex [15]. These CNNs, which we refer to as LCANets, were shown to be more robust than standard CNNs on standard CV tasks. However, there are two issues with this approach we address here. First, sparse coding models were designed to model the visual cortex, so it is unclear how they will impact the performance of CNNs on audio classification tasks. Second, these LCANets were robust to natural corruptions, but they were susceptible to white-box adversarial attacks [13] unless the exact attack was known before hand [14].

Motivated by this, we introduce multi-layer LCANets, which perform sparse coding in multiple CNN layers. We refer to these multilayer LCANets as LCANets++ and train them on audio classification tasks. To test the robustness of LCANets++ relative to LCANets and standard CNNs, we first conducted experiments with different audio perturbations, e.g., _background noise_. In addition, we show that our proposed LCANets++ are more robust compared to the state-of-the-art (SOTA) models (e.g., ResNet18, standard CNN) and LCANets against white-box attacks, i.e., _fast gradient sign attack (FGSM)_[16] and _projected gradient descent attack (PGD)_[17], as well as black-box attacks, i.e., _evasion attack_.

## 2 Proposed Method

### LCA Layer

As presented in Fig. 1, LCA layer is the basic building block for the LCA frontend and our proposed LCANets++. LCA layer converts the input \(\mathcal{X}\) to a coding \(\mathcal{C}\), i.e., representation of the input \(\mathcal{X}\), leveraging the least number of active neurons (i.e., features). The goal of the reconstruction minimization problem applied here is to find the sparse coding representation \(C\) closest possible to original input \(\mathcal{X}\) as follows:

\[\mathcal{L}_{re}=\min_{C}\frac{1}{2}\|\mathcal{X}-\mathcal{C}\oplus\Phi\|_{2 }^{2}+\lambda\|\mathcal{C}\|_{1} \tag{1}\]

where \(\Phi\) is the \(\Phi\)-norm of the input \(\mathcal{X}\), and \(\Phi\) is the \(\Phi\)-norm of the input \(\mathcal{X}\). The \(\Phi\)-norm of the input \(\mathcal{X}\) is the \(\Phi\)-norm of the input \(\mathcal{X}\).
where \(\mathcal{L}_{re}\) denotes the reconstruction loss, \(\mathcal{X}\) is the original input, \(\mathcal{C}\) is the sparse code, \(\mathfrak{G}\) is transpose convolution, \(\Phi\) is dictionary components learned last iteration, and \(\lambda\) is the trade-off (i.e., regularization) constant. LCA layers perform lateral competitions to fire neurons and a neuron membrane follows the following ordinary differential equation [13]:

\[\hat{\mathcal{M}}(t)=\frac{1}{\gamma}[\mathcal{D}(t)-\mathcal{M}(t)-\mathcal{ C}(t)*\mathcal{S}+\mathcal{C}(t)] \tag{2}\]

where \(\gamma\) is time constant, \(\mathcal{D}(t)\) stands for neuron's input drive obtained by convolution of inputs with dictionary, i.e., \(\mathcal{X}*\Phi\); \(\mathcal{M}(t)\) is neuron's membrane potential, \(\mathcal{S}\)= \(\Phi*\Phi\) is the pairwise feature similarity, and \(\mathcal{C}(t)\) is the neuron's firing rate obtained by applying a soft threshold activation on the membrane potential \(\mathcal{M}(t)\). Coordinate ascent is used to learn dictionary \(\Phi\), which solves for \(\mathcal{C}\), given an input batch using LCA and then updates \(\Phi\) with stochastic gradient descent (SGD).

### LCA Frontend

LCA frontend is the unsupervised pre-training part of the LCANet, as shown in Fig. 0(a). It basically consists of the raw audio waveform converted to MFCCs, as input signal \(\mathcal{X}\) and the LCA layer to compute the sparse representation \(\mathcal{C}\) of the input \(\mathcal{X}\), which can then feed to conventional CNN layers for the classification task.

### LCANets

LCANets for the audio classification consist of the LCA frontend and followed by CNN layers. LCA frontend learns in unsupervised fashion and then passes the computed sparse code \(\mathcal{C}\) to the CNN layers to finally perform the classification task. One major difference is that, the sparse code \(\mathcal{C}\) does not need to recompute back to original input \(\mathcal{X}\) before feeding to CNN layer, as other reconstruction-based models usually do. This makes the LCANets more effective against perturbations, while reducing dependency on labeled audio samples.

### LCANets++

We present the overview of our proposed LCANets++ in Fig.0(b). The basic building block of our proposed LCANets++ is the LCA layers. In this architecture, multiple LCA layers are inserted that learn in unsupervised fashion. The convolutional layers in SOTA CNN networks are replaced by the LCA layers, performing sparse coding in each layer. Similar to [18], in order to reduce over-sparsity, in between two consecutive sparse layers (i.e., LCA layers), a dense layer, i.e., batch normalization layer, is mounted (Fig.0(b)) in our LCANets++.

## 3 Experiments

In this section, details of the experimental setup, including dataset, pre-processing, and models, are described.

### Dataset and Pre-processing

We experiment with Google Speech Commands v2 [19] dataset. This dataset has audio waveforms of 35 classes of human speech commands like "yes," "no," and "left", "right." We perform pre-processing on the raw waveforms of the three influential classes, i.e., "yes," "no," and "stop" to obtain the Mel-frequency cepstral coefficient (MFCC) [8] features and train all the models with the MFCCs of the waveforms.

### Models

We experiment with regular CNN models with 2 convolutional layers. In our LCANets++ on CNN model, we replace both convolutional layers with the LCA layers. We also compare our LCANets++ with the larger SOTA model, i.e., ResNet18. In the ResNet18 model, we replace the alternative convolutional layers in the first block with LCA layers to obtain the ResNet18_LCA++ model. To experiment with the performance of LCANets++ against _white-box_ or _black-box_ attacks, we consider the regularization constant \(\lambda=1.00\) for better sparse representations and hence, improved robustness against perturbations or attacks.

Figure 1: An overview of (a.) LCA frontend and (b.) pipeline of our proposed LCANets++, utilizing sparse coding via multiple LCA layers in the state-of-the-art (SOTA) CNN backbone, enabling lower misclassification on perturbed test sets or attacks.
### Experimental Setup

We run all the experiments on 8 nodes NVIDIA_A100-PCIE-40GB GPUs with 64-128 cores on the cluster. We use the Pytorch framework to develop the LCA class and LCANets++ implementations. We consider a train test split of 70% and 30% for all the models experimented in this work. For the _background noise_ experiment, we train models for 50 epochs, for rest of the experiments models are trained with 20 epochs. We consider 0.0001 learning rate. We use SGD optimizer with 0.9 momentum for optimization. In order to add background noise, we impose background noise on all test set raw waveforms of audio clips, tuning the SNR [db] values to obtain different perturbed test sets. Similarly, we consider perturbing MFCCs with different \(\epsilon\) values.

## 4 Results and Analysis

In this section, we illustrate the key results of our experiments on different perturbations and adversarial attacks.

### Input Perturbations

We test the robustness of standard CNNs without the LCA layer(s), LCANets, and our proposed LCANets++ to perturbations. We experiment with two different cases of input perturbations: i) _background noise_ on the raw audio clips and ii) _gaussian noise_ on MFCCs to compare robustness against both perturbation scenarios.

#### 4.1.1 Background Noise

In Fig. (a)a, we present the performance of regular CNN, LCANet, and our proposed LCANet++, tested on different perturbed test sets with _background noise_, varying SNR [db] values. Observe that the regular CNN model performance drastically goes down as more perturbation is applied to original waveforms (i.e., lower SNRs). Whereas, LCANet goes down slowly with increasing perturbations, and our proposed LCANet++ shows the most robustness compared to LCANet and regular CNN models, as presented in Fig. (a)a. This is attributed to the fact that LCA layers learn in an unsupervised fashion, reducing the numbers of the neurons activated through lateral competitions. These fewer activated neurons represent the most relevant input features, which are less impacted by slight perturbations.

We also test the robustness of LCANets++ on larger models, i.e., ResNet18 model with 18 layers. As presented in Fig. (b)b, we observe that the ResNet18 with multilayer LCAs, i.e., ResNet18_LCANet++ outperforms regular ResNet18 and ResNet18 with LCA in the first layer, i.e., ResNet18_LCANet. From Table 1, we find that for the ResNet18 architecture, LCANets++ slightly improves the robustness on perturbed test sets than regular ResNet18 without LCA layers, as opposed to significantly higher robustness LCANets++ exhibited on regular CNN model. Larger model with more layers and parameters make ResNet18 inherently more robust than regular CNNs, resulting in LCANets++ to boost up only slightly in ResNet18 than regular CNNs.

#### 4.1.2 Gaussian Noise

We impose _Gaussian noise_ on the MFCCs varying \(\epsilon\) values. As presented in Table. 2, with increasing the \(\epsilon\) (more perturbations), performance of the regular CNN model goes down. Also, LCANet and LCANet++ performance slightly goes down, but still, the models with LCA layers show more robustness compared to the model without LCA layers, i.e., regular CNN model. This shows that our LCANets++ are more robust not only against perturbations on raw waveforms, but also against perturbations on the feature space, i.e., MFCCs.

\begin{table}
\begin{tabular}{l c c c c c} \hline Model & \(SNR\) & & & & \\  & = \(15db\) & \(20db\) & \(24db\) & \(25db\) & \(\infty\) \\ \hline CNN & 0.692 & 0.788 & 0.793 & 0.858 & 0.920 \\ LCANet & 0.760 & 0.840 & 0.876 & 0.904 & 0.940 \\
**LCANet++** & **0.768** & **0.847** & **0.903** & **0.914** & **0.962** \\ \hline \hline ResNet18 & 0.847 & 0.920 & 0.942 & 0.955 & 0.970 \\ ResNet18\_LCA & 0.850 & 0.922 & 0.944 & 0.958 & 0.966 \\
**ResNet18\_LCA++** & **0.853** & **0.928** & **0.945** & **0.959** & **0.971** \\ \hline \end{tabular}
\end{table}
Table 1: Performance comparisons against perturbations with Background Noise on waveforms

\begin{table}
\begin{tabular}{l c c c c c c} \hline Model & \(\epsilon\) & & & & \\  & = 0 & 0.01 & 0.02 & 0.03 & 0.04 & 0.05 \\ \hline CNN & 0.866 & 0.864 & 0.863 & 0.863 & 0.858 & 0.856 \\ LCANet & 0.939 & 0.938 & 0.935 & 0.925 & 0.909 & 0.883 \\
**LCANet++** & **0.950** & **0.943** & **0.939** & **0.927** & **0.914** & **0.900** \\ \hline \end{tabular}
\end{table}
Table 2: Performance comparisons against perturbations with Gaussian Noise on MFCCs

Figure 2: Comparisons of our LCANets++ and other SOTA models against perturbations with _background noise_.
### Adversarial Attacks

We experiment with adversarial attacks having different capabilities. For experimental purposes, we consider both the _white-box_ and _black-box_ adversarial attacks.

#### 4.2.1 White-box Attacks

In _white-box_ attacks, an adversary has more capabilities like having access to the model architectures, including model parameters, weights, and gradients. We consider two different types of _white-box_ attacks, i.e., _FGSM_[16] and _PGD_[17]. In both attacks, the adversary utilizes the gradients to perturb the MFCCs of test sets to misclassify them during inference.

We present the performances of the regular CNN model and LCANets, as well as our proposed LCANets++, against the FGSM attack in Fig. (a)a. We find that the regular CNN is not very robust, and it's performance goes down, as perturbations (\(\epsilon\)) go higher. We observe that, the single-layer LCA, i.e., LCANets are not robust against the _white-box_ FGSM attack, which is consistent to findings in [13] for CV tasks. However, our proposed multi-layer LCANets++ outperforms the CNN model and LCANets on audio classification against the FGSM attack. In Fig. (a)a, we observe that LCANets++ performance decreases comparatively slowly as attack becomes stronger with higher perturbations (\(\epsilon\)). We also experiment with another _white-box_ attack, i.e., PGD attack, where LCANets++ consistently show more robustness than SOTA models and LCANets, as shown in Table 3.

#### 4.2.2 Black-box Attacks

We experiment with the _black-box_ evasion attack, where the adversary has no access to the model gradients. In this attack, an adversary only has query access to the model and can get predictions from the model utilizing the query access. In our setup, the adversary is able to make queries to the original target model and get predictions from the model. The adversary utilizes the predictions and input queries to develop a surrogate model. The surrogate model generates the perturbed samples, varying perturbations (\(\epsilon\)), and we tested the performance of the original models on these perturbed test sets. We present the performances of regular CNN, LCANet, and our proposed LCANet++ against the _black-box_ evasion attack in Fig. (b)b. We observe that, in _black-box_ evasion attack, LCANet shows more robustness compared to CNN and LCANet++ outperforms all the models on perturbed test sets (i.e., \(\epsilon>0\)). Note that, models are trained for 20 epochs with three audio classes (i.e., limited samples), which might lead to a significant performance gap among the regular CNN and LCA-based models on unperturbed test sets \(\epsilon=0\) in Table. 4.

## 5 Conclusions

In this work, we developed CNNs with sparse coding in multiple layers, referred to as LCANets++. We showed that LCANets++ can be easily implemented using regular CNNs like ResNet18. Our empirical analysis shows that LCANets++ can be used in audio classifiers to increase robustness to noise and adversarial attacks relative to LCANets and standard CNNs. In addition, we observe how the unsupervised training with LCA and number of LCA layers impacts clean and robust test accuracy. Overall, our work sheds light into future directions in designing privacy-preserving robust audio classifiers.

## 6 Acknowledgements

We gratefully acknowledge support from the Advanced Scientific Computing Research (ASCR) program office in the Department of Energy's (DOE) Office of Science, award #77902, as well as the Center for Nonlinear Studies and the Cyber Summer School at Los Alamos National Laboratory.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Model & \(\epsilon\) & & & & & \\  & = 0 & 0.01 & 0.02 & 0.03 & 0.04 & 0.05 \\ \hline CNN & 0.866 & 0.865 & 0.865 & 0.864 & 0.862 & 0.860 \\ LCANet & 0.939 & 0.939 & 0.935 & 0.926 & 0.908 & 0.880 \\
**LCANet++** & **0.950** & **0.948** & **0.939** & **0.928** & **0.915** & **0.905** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparisons against _black-box_ (Evasion) attack

Figure 3: Comparisons of LCANets++ and SOTA models on \(L_{\infty}\) norm _white-box_ attacks.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline Attack & Model & \(\epsilon\) & & & & \\  & = 0 & 0.01 & 0.016 & 0.02 & 0.03 \\ \hline \multirow{4}{*}{FGSM} & CNN & 0.866 & 0.439 & 0.196 & 0.108 & 0.017 \\  & LCANet & 0.939 & 0.261 & 0.123 & 0.092 & 0.062 \\  & **LCANet++** & **0.950** & **0.679** & **0.418** & **0.417** & **0.414** \\ \hline \hline \multirow{4}{*}{PGD} & CNN & 0.866 & 0.382 & 0.147 & 0.073 & 0.025 \\  & LCANet & 0.939 & 0.028 & 0.005 & 0.005 & 0.005 \\  & **LCANet++** & **0.950** & **0.588** & **0.585** & **0.579** & **0.567** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparisons against _white-box_ attacks