11institutetext: Analytics Quotient, Bangalore, India
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
22institutetext: Samsung R&D Institute India-Bangalore, Bangalore, India
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>33institutetext:
Birla Institute of Technology & Science, Pilani, Rajasthan, India
33email<EMAIL_ADDRESS>
# Spoken Language Identification using ConvNets

Sarthak 11 Shikhar Shukla 22 Govind Mittal 33

###### Abstract

Language Identification (LI) is an important first step in several speech
processing systems. With a growing number of voice-based assistants, speech LI
has emerged as a widely researched field. To approach the problem of
identifying languages, we can either adopt an implicit approach where only the
speech for a language is present or an explicit one where text is available
with its corresponding transcript. This paper focuses on an implicit approach
due to the absence of transcriptive data. This paper benchmarks existing
models and proposes a new attention based model for language identification
which uses log-Mel spectrogram images as input. We also present the
effectiveness of raw waveforms as features to neural network models for LI
tasks. For training and evaluation of models, we classified six languages
(English, French, German, Spanish, Russian and Italian) with an accuracy of
95.4% and four languages (English, French, German, Spanish) with an accuracy
of 96.3% obtained from the VoxForge dataset. This approach can further be
scaled to incorporate more languages.

###### Keywords:

Language Identification Raw Waveform Convolutional Neural Networks Machine
Learning.

##  1 Introduction

Language Identification (LI) is a problem which involves classifying the
language being spoken by a speaker. LI systems can be used in call centers to
route international calls to an operator who is fluent in that identified
language [12]. In speech-based assistants, LI acts as the first step which
chooses the corresponding grammar from a list of available languages for its
further semantic analysis [1]. It can also be used in multi-lingual voice-
controlled information retrieval systems, for example, Apple Siri and Amazon
Alexa.

Over the years, studies have utilized many prosodic and acoustic features to
construct machine learning models for LI systems [18]. Every language is
composed of _phonemes_ , which are distinct unit of sounds in that language,
such as _b_ of black and _g_ of green. Several prosodic and acoustic features
are based on phonemes, which become the underlying features on whom the
performance of the statistical model depends [20, 5]. If two languages have
many overlapping phonemes, then identifying them becomes a challenging task
for a classifier. For example, the word _cat_ in English, _kat_ in Dutch,
_katze_ in German have different consonants but when used in a speech they all
would sound quite similar.

Due to such drawbacks several studies have switched over to using Deep Neural
Networks (DNNs) to harness their novel auto-extraction techniques [1, 19].
This work follows an implicit approach for identifying six languages with
overlapping phonemes on the VoxForge [23] dataset and achieves 95.4% overall
accuracy.

In previous studies [1, 17, 19], authors use log-Mel spectrum of a raw audio
as inputs to their models. One of our contributions is to enhance the
performance of this approach by utilising recent techniques like Mixup
augmentation of inputs and exploring the effectiveness of _Attention_
mechanism in enhancing performance of neural network. As log-Mel spectrum
needs to be computed for each raw audio input and processing time for
generating log-Mel spectrum increases linearly with length of audio, this acts
as a bottleneck for these models. Hence, we propose the use of raw audio
waveforms as inputs to deep neural network which boosts performance by
avoiding additional overhead of computing log-Mel spectrum for each audio. Our
1D-ConvNet architecture auto-extracts and classifies features from this raw
audio input.

The structure of the work is as follows. In Section 2 we discuss about the
previous related studies in this field. The model architecture for both the
raw waveforms and log-Mel spectrogram images is discussed in Section 3 along
with the a discussion on hyperparameter space exploration. In Section 4 we
present the experimental results. Finally, in Section 5 we discuss the
conclusions drawn from the experiment and future work.

##  2 Related Work

Extraction of language dependent features like prosody and phonemes was a
popular approach to classify spoken languages [29, 16, 6]. Following their
success in speaker verification systems, i-vectors have also been used as
features in various classification networks. These approaches required
significant domain knowledge [4, 16]. Nowadays most of the attempts on spoken
language identification rely on neural networks for meaningful feature
extraction and classification [15, 7].

Revay et al. [19] used the ResNet50 [9] architecture for classifying languages
by generating the log-Mel spectra of each raw audio. The model uses a cyclic
learning rate where learning rate increases and then decreases linearly.
Maximum learning rate for a cycle is set by finding the optimal learning rate
using _fastai_ [11] library. The model classified six languages – English,
French, Spanish, Russian, Italian and German – and achieving an accuracy of
89.0%.

Gazeau et al. [8] in his research showed how Neural Networks, Support Vector
Machine and Hidden Markov Model (HMM) can be used to identify French, English,
Spanish and German. Dataset was prepared using voice samples from Youtube News
[27]and VoxForge [23] datasets. Hidden Markov models convert speech into a
sequence of vectors, was used to capture temporal features in speech. HMMs
trained on VoxForge [23] dataset performed best in comparison to other models
proposed by him on same VoxForge dataset. They reported an accuracy of 70.0%.

Bartz et al. [1] proposed two different hybrid Convolutional Recurrent Neural
Networks for language identification. They proposed a new architecture for
extracting spatial features from log-Mel spectra of raw audio using CNNs and
then using RNNs for capturing temporal features to identify the language. This
model achieved an accuracy of 91.0% on Youtube News Dataset [27]. In their
second architecture they used the Inception-v3 [22] architecture to extract
spatial features which were then used as input for bi-directional LSTMs to
predict the language accurately. This model achieved an accuracy of 96.0% on
four languages which were English, German, French and Spanish. They also
trained their CNN model (obtained after removing RNN from CRNN model) and the
Inception-v3 on their dataset. However they were not able to achieve better
results achieving and reported 90% and 95% accuracies, respectively.

Kumar et al. [12] used Mel-frequency cepstral coefficients (MFCC), Perceptual
linear prediction coefficients (PLP), Bark Frequency Cepstral Coefficients
(BFCC) and Revised Perceptual Linear Prediction Coefficients (RPLP) as
features for language identification. BFCC and RPLP are hybrid features
derived using MFCC and PLP. They used two different models based on Vector
Quantization (VQ) with Dynamic Time Warping (DTW) and Gaussian Mixture Model
(GMM) for classification. These classification models were trained with
different features. The authors were able to show that these models worked
better with hybrid features (BFCC and RPLP) as compared to conventional
features (MFCC and PLP). GMM combined with RPLP features gave the most
promising results and achieved an accuracy of 88.8% on ten languages. They
designed their own dataset comprising of ten languages being Dutch, English,
French, German, Italian, Russian, Spanish, Hindi, Telegu, and Bengali.

Montavon [17] generated Mel spectrogram as features for a time-delay neural
network (TDNN). This network had two-dimensional convolutional layers for
feature extraction. An elaborate analysis of how deep architectures outperform
their shallow counterparts is presented in this reseacrch. The difficulties in
classifying perceptually similar languages like German and English were also
put forward in this work. It is mentioned that the proposed approach is less
robust to new speakers present in the test dataset. This method was able to
achieve an accuracy of 91.2% on dataset comprising of 3 languages – English,
French and German.

In Table 1, we summarize the quantitative results of the above previous
studies. It includes the model basis, feature description, languages
classified and the used dataset along with accuracy obtained. The table also
lists the overall results of our proposed models (at the top). The languages
used by various authors along with their acronyms are English (En), Spanish
(Es), French (Fr), German (De), Russian (Ru), Italian (It), Bengali (Ben),
Hindi (Hi) and Telegu (Tel).

Table 1: Quantitative Review of Previous Studies along with our Results. Year |  Model basis |  Features |  Languages |  Acc. |  Remarks |  Ref.
---|---|---|---|---|---|---
2019 |  1D ConvNet |  Raw Audio |  En, Fr, De, Es, Ru, It |  93.71 |  Evaulation of our 1D ConvNet model with mixup for six languages. |  self
2019 |  2D ConvNet |  log-Mel |  En, Fr, De, Es, Ru, It |  95.41 |  Evaulation of our 2D ConvNet model with mixup for six languages. |  self
2019 |  2D ConvNet-Bi-directional GRU-Attention |  log-Mel |  En, Fr, De, Es, Ru, It |  95.01 |  Result after tuning the hyperparameters of our cnn-bi-directional GRU-attention model and applying mixup |  self
2019 |  2D ConvNet |  log-Mel |  En, Fr, De, Es |  96.31 |  Our evaluation of 2D ConvNet model for four languages. |  self
2019 |  ResNet50 |  log-Mel |  En, Fr, De, Es, Ru, It |  89.01 |  Uses a pretrained ResNet50 architecture and cyclic learner to identify the language. |  [19]
2018 |  SVM-HMM model |  not defined |  En, Fr, Es, De |  70.01 |  HMMs were used to encode speech into sequences of vectors which were then fed into a neural network. |  [8]
2017 |  Inceptionv3 CRNN |  log-Mel |  En, Fr, De, Es |  96.02 |  Used Inception-v3 model followed by bi-directional LSTMs to extract convolutional and temporal features. |  [1]
2017 |  CRNN |  log-Mel |  En, Fr, De, Es |  91.02 |  A new architecture is used to extract spatial features by using CNNs and temporal features using RNNs. |  [1]
2010 |  Gaussian Mixture Models |  Perceptual Linear Prediction |  Dut, En, Fr, De, It, Ru, Es, Ben, Hi and Tel |  88.83 |  Used Gaussian mixture models coupled with RPLP features, which were prepared using MFCC and PLP. |  [12]
2009 |  CNN-TDNN |  log-Mel |  En, Fr , De |  91.21 |  Used a time delay neural network with SGD was used to identify language using log-Mel images as input. |  [17]
Dataset: 1 \- VoxForge [23]; 2 \- Youtube News [27], 3 \- Private |

##  3 Proposed Method

###  3.1 Motivations

Several state-of-the-art results on various audio classification tasks have
been obtained by using log-Mel spectrograms of raw audio, as features [25].
Convolutional Neural Networks have demonstrated an excellent performance gain
in classification of these features [26, 10] against other machine learning
techniques. It has been shown that using _attention_ layers with ConvNets
further enhanced their performance [13]. This motivated us to develop a CNN-
based architecture with _attention_ since this approach hasn’t been applied to
the task of language identification before.

Recently, using raw audio waveform as features to neural networks has become a
popular approach in audio classification [24, 13]. Raw waveforms have several
artifacts which are not effectively captured by various conventional feature
extraction techniques like Mel Frequency Cepstral Coefficients (MFCC),
Constant Q Transform (CQT), Fast Fourier Transform (FFT), etc.

Audio files are a sequence of spoken words, hence they have temporal features
too.A CNN is better at capturing spatial features only and RNNs are better at
capturing temporal features as demonstrated by Bartz et al. [1] using audio
files. Therefore, we combined both of these to make a CRNN model.

We propose three types of models to tackle the problem with different
approaches, discussed as follows.

###  3.2 Description of Features

As an average human’s voice is around 300 Hz and according to Nyquist-Shannon
sampling theorem all the useful frequencies (0-300 Hz) are preserved with
sampling at 8 kHz, therefore, we sampled _raw audio_ files from all six
languages at 8 kHz

The average length of audio files in this dataset was about _10.4 seconds_ and
standard deviation was _2.3 seconds_. For our experiments, the audio length
was set to 10 seconds. If the audio files were shorter than 10 second, then
the data was repeated and concatenated. If audio files were longer, then the
data was truncated.

###  3.3 Model Description

We applied the following design principles to all our models:

  * •

_Every convolutional layer is always followed by an appropriate max pooling
layer_. This helps in containing the explosion of parameters and keeps the
model small and nimble.

  * •

_Convolutional blocks_ are defined as an individual block with multiple pairs
of one convolutional layer and one max pooling layer. _Each convolutional
block is preceded or succeded by a convolutional layer._

  * •

_Batch Normalization and Rectified linear unit activations were applied after
each convolutional layer_. Batch Normalization helps speed up convergence
during training of a neural network.

  * •

Model _ends with a dense layer_ which acts the final output layer.

###  3.4 Model Details: 1D ConvNet

As the sampling rate is 8 kHz and audio length is 10 s, hence the input is
_raw audio_ to the models with input size of (batch size, 1, 80000). In Table
2, we present a detailed layer-by-layer illustration of the model along with
its hyperparameter.

Table 2: Architecture of the 1D-ConvNet model

Layer Name |  | # filters / kernel /
---
stride
output | # of parameters
Conv1 | (128, 3, 3) | (128, 26664) | 384
| (Convolutional Block 1)
---
Conv1D
MaxPool1D
Conv1D
MaxPool1D
| (128, 3, 1)
---
(3, 3)
(128, 3, 1)
(3, 3)
| (128, 26658)
---
(128, 8880)
(128, 8880)
(128, 2960)
| 49152
---
49,152
| Conv1D
---
MaxPool1D
| (256, 3, 1)
---
(3, 3)
| (256, 2954)
---
(256, 984)
98,304
| (Convolutional Block 2)
---
Conv1D
MaxPool1D
| (256, 3, 1)
---
(3, 3)
| (256, 978)
---
(256, 326)
196,608
| Conv1D
---
MaxPool1D
| (512, 3, 1)
---
(106, 3)
| (512, 320)
---
(512, 1)
393,216
Dense Layer | (512, 6) | (6) | 3,072

####  3.4.1 Hyperparameter Optimization:

Tuning hyperparameters is a cumbersome process as the hyperparamter space
expands exponentially with the number of parameters, therefore efficient
exploration is needed for any feasible study. We used the _random search_
algorithm supported by _Hyperopt_ [2] library to randomly search for an
optimal set of hyperparameters from a given parameter space. In Fig. 1,
various hyperparameters we considered are plotted against the validation
accuracy as violin plots. Our observations for each hyperparameter are
summarized below:

_Number of filters in first layer_ : We observe that having 128 filters gives
better results as compared to other filter values of 32 and 64 in the first
layer. A higher number of filters in the first layer of network is able to
preserve most of the characteristics of input.

_Kernel Size_ : We varied the receptive fields of convolutional layers by
choosing the kernel size from among the set of {3, 5, 7, 9}. We observe that a
kernel size of 9 gives better accuracy at the cost of increased computation
time and larger number of parameters. A large kernel size is able to capture
longer patterns in its input due to bigger receptive power which results in an
improved accuracy.

_Dropout_ : Dropout randomly turns-off (sets to 0) various individual nodes
during training of the network. In a deep CNN it is important that nodes do
not develop a co-dependency amongst each other during training in order to
prevent overfitting on training data [21]. Dropout rate of $0.1$ works well
for our model. When using a higher dropout rate the network is not able to
capture the patterns in training dataset.

_Batch Size_ : We chose batch sizes from amongst the set {32, 64, 128}. There
is more noise while calculating error in a smaller batch size as compared to a
larger one. This tends to have a regularizing effect during training of the
network and hence gives better results. Thus, batch size of 32 works best for
the model.

_Layers in Convolutional block 1 and 2_ : We varied the number of layers in
both the convolutional blocks. If the number of layers is low, then the
network does not have enough depth to capture patterns in the data whereas
having large number of layers leads to overfitting on the data. In our
network, two layers in the first block and one layer in the second block give
optimal results.

Figure 1: Effect of hyperparameter variation of the hyperparameter on the
classification accuracy for the case of 1D-ConvNet. Orange colored violin
plots show the most favored choice of the hyperparameter and blue shows
otherwise. One dot represents one sample.

###  3.5 Model Details: 2D ConvNet with Attention and bi-directional GRU

Log-Mel spectrogram is the most commonly used method for converting audio into
the image domain. The audio data was again sampled at 8 kHz. The input to this
model was the log-Mel spectra. We generated log-Mel spectrogram using the
_LibROSA_ [14] library. In Table 3, we present a detailed layer-by-layer
illustration of the model along with its hyperparameter.

#### 3.5.1

We took some specific design choices for this model, which are as follows:

  * •

We added _residual connections_ with each convolutional layer. Residual
connections in a way makes the model selective of the contributing layers,
determines the optimal number of layers required for training and solves the
problem of vanishing gradients. Residual connections or skip connections skip
training of those layers that do not contribute much in the overall outcome of
model.

Table 3: Architecture of the 2D-ConvNet model

Layer Name | Output features | Number of filters / stride / padding | No. of parameters
---|---|---|---
| (ConvBlock 1)
---
Conv2D
Conv2D
AvgPool2D
| (64, 128, 128)
---
(64, 128, 128)
(64, 64, 64)
| (3, 3) / (1, 1) / (1, 1)
---
(3, 3) / (1, 1) / (1, 1)
| 1,728
---
36,864
| (ConvBlock 2)
---
Conv2D
Conv2D
AvgPool2D
| (128, 64, 64)
---
(128, 64, 64)
(128, 32, 32)
| (3, 3) / (1, 1) / (1, 1)
---
(3, 3) / (1, 1) / (1, 1)
| 73,728
---
147,456
| (ConvBlock 3)
---
Conv2D
Conv2D
AvgPool2D
| (256, 32, 32)
---
(256, 32, 32)
(256, 16, 16)
| (3, 3) / (1, 1) / (1, 1)
---
(3, 3) / (1, 1) / (1, 1)
| 294,912
---
589,824
| (ConvBlock 4)
---
Conv2D
Conv2D
AvgPool2D
| (512, 16, 16)
---
(512, 16, 16)
(512, 8, 8)
| (3, 3) / (1, 1) / (1, 1)
---
(3, 3) / (1, 1) / (1, 1)
| 1,179,648
---
235,929
| Bi-directional GRU
---
Embedding Layer
| (8, 1536)
---
(8, 768)
|  | 1,769,472
---
1,179,648
| (Sequential Block)
---
Dropout (0.2)
Linear
Dropout (0.1)
Linear
| (256)
---
(6)
|  | 131,072
---
1,536

  * •

We added _spatial attention_ [3] networks to help the model in focusing on
specific regions or areas in an image. Spatial attention aids learning
irrespective of transformations, scaling and rotation done on the input images
making the model more robust and helping it to achieve better results.

  * •

We added _Channel Attention_ networks so as to help the model to find
interdependencies among color channels of log-Mel spectra. It adaptively
assigns importance to each color channel in a deep convolutional multi-channel
network. In our model we apply channel and spatial attention just before
feeding the input into bi-directional GRU. This helps the model to focus on
selected regions and at the same time find patterns among channels to better
determine the language.

Figure 2: Effect of hyperparameter variation of the six selected
hyperparameter on the classification accuracy for the case of 2D-ConvNet.
Orange colored violin plots show the most favored choice of the hyperparameter
and blue shows otherwise. One dot represents one sample.

####  3.5.2 Hyperparameter Optimization:

We used the _random search_ algorithm supported by _Hyperopt_ [2] library to
randomly search for an optimal set of hyperparameters from a given parameter
space. In Fig. 2 ,various hyperparameters we tuned are plotted against the
validation accuracy. Our observations for each hyperparameter are summarized
below:

_Filter Size_ : 64 filters in the first layer of network can preserve most of
the characteristics of input, but increasing it to 128 is inefficient as
overfitting occurs.

_Kernel Size_ : There is a trade-off between kernel size and capturing complex
non-linear features. Using a small kernel size will require more layers to
capture features whereas using a large kernel size will require less layers.
Large kernels capture simple non-linear features whereas using a smaller
kernel will help us capture more complex non-linear features. However, with
more layers, backpropagation necessitates the need for a large memory. We
experimented with large kernel size and gradually increased the layers in
order to capture more complex features. The results are not conclusive and
thus we chose kernel size of 7 against 3.

_Dropout_ : Dropout rate of 0.1 works well for our data. When using a higher
dropout rate the network is not able to capture the patterns in training
dataset.

_Batch Size_ : There is always a trade-off between batch size and getting
accurate gradients. Using a large batch size helps the model to get more
accurate gradients since the model tries to optimize gradients over a large
set of images. We found that using a batch size of 128 helped the model to
train faster and get better results than using a batch size less than 128.

_Number of hidden units in bi-directional GRU_ : Varying the number of hidden
units and layers in GRU helps the model to capture temporal features which can
play a significant role in identifying the language correctly. The optimal
number of hidden units and layers depends on the complexity of the dataset.
Using less number of hidden units may capture less features whereas using
large number of hidden units may be computationally expensive. In our case we
found that using 1536 hidden units in a single bi-directional GRU layer leads
to the best result.

_Image Size_ : We experimented with log-Mel spectra images of sizes $64\times
64$ and $128\times 128$ pixels and found that our model worked best with
images of size of $128\times 128$ pixels.

We also evaluated our model on data with mixup augmentation [28]. It is a data
augmentation technique that also acts as a regularization technique and
prevents overfitting. Instead of directly taking images from the training
dataset as input, mixup takes a linear combination of any two random images
and feeds it as input. The following equations were used to prepared a mixed-
up dataset:

$\mathrm{Input\\_Image}=\alpha*I_{1}+(1-\alpha)*I_{2},$ (1)

and

$\mathrm{Input\\_Label}=\alpha*L_{1}+(1-\alpha)*L_{2},$ (2)

where $\alpha\in[0,1]$ is a random variable from a $\beta$-distribution,
$I_{1}$.

###  3.6 Model details: 2D-ConvNet

This model is a similar model to 2D-ConvNet with Attention and bi-directional
GRU described in section 3.5 except that it lacks skip connections, attention
layers, bi-directional GRU and the embedding layer incorporated in the
previous model.

###  3.7 Dataset

We classified six languages (English, French, German, Spanish, Russian and
Italian) from the VoxForge [23] dataset. VoxForge is an open-source speech
corpus which primarily consists of samples recorded and submitted by users
using their own microphone. This results in significant variation of speech
quality between samples making it more representative of real world scenarios.

Our dataset consists of 1,500 samples for each of six languages. Out of 1,500
samples for each language, 1,200 were randomly selected as training dataset
for that language and rest 300 as validation dataset using k-fold cross-
validation. To sum up, we trained our model on 7,200 samples and validated it
on 1800 samples comprising six languages. The results are discussed in next
section.

##  4 Results and Discussion

This paper discusses two end-to-end approaches which achieve state-of-the-art
results in both the image as well as audio domain on the VoxForge dataset
[23]. In Table 4, we present all the classification accuracies of the two
models of the cases with and without mixup for six and four languages.

In the audio domain (using raw audio waveform as input), 1D-ConvNet achieved a
mean accuracy of 93.7% with a standard deviation of 0.3% on running k-fold
cross validation. In Fig 3 (a) we present the confusion matrix for the
1D-ConvNet model.

In the image domain (obtained by taking log-Mel spectra of raw audio),
2D-ConvNet with 2D attention (channel and spatial attention) and bi-
directional GRU achieved a mean accuracy of 95.0% with a standard deviation of
1.2% for six languages. This model performed better when mixup regularization
was applied. 2D-ConvNet achieved a mean accuracy of 95.4% with standard
deviation of 0.6% on running k-fold cross validation for six languages when
mixup was applied. In Fig 3 (b) we present the confusion matrix for the
2D-ConvNet model. 2D attention models focused on the important features
extracted by convolutional layers and bi-directional GRU captured the temporal
features.

Table 4: Results of the two models and all its variations Languages | Feature Desc. | Network | Mixup | Accuracy
---|---|---|---|---
En, Es, Fr, De, Ru, It | Raw Waveform | 1D ConvNet | No | 93.7
log-Mel Spectra | 2D ConvNet | No | 94.3
Yes | 95.4
2D ConvNet with Attention and GRU | No | 94.3
Yes | 95.0
En, Es, Fr, De | Raw Waveform | 1D ConvNet | No | 94.4
log-Mel Spectra | 2D ConvNet | No | 96.0
Yes | 96.3
2D ConvNet with Attention and GRU | No | 94.7
Yes | 93.7

####  4.0.1 Misclassification

Several of the spoken languages in Europe belong to the Indo-European family.
Within this family, the languages are divided into three phyla which are
Romance, Germanic and Slavic. Of the 6 languages that we selected Spanish
(Es), French (Fr) and Italian (It) belong to the Romance phyla, English and
German belong to Germanic phyla and Russian in Slavic phyla. Our model also
confuses between languages belonging to the similar phyla which acts as an
insanity check since languages in same phyla have many similar pronounced
words such as _cat_ in English becomes _Katze_ in German and _Ciao_ in Italian
becomes _Chao_ in Spanish.

Our model confuses between French (Fr) and Russian (Ru) while these languages
belong to different phyla, many words from French were adopted into Russian
such as automate (oot-oo-mate) in French becomes ABTOMaT (aff-taa-maat) in
Russian which have similar pronunciation.

(a)

(b)

Figure 3: Confusion matrix for classification of six languages with our (a)
1D-ConvNet and (b) 2D-ConvNet model. Asterisk (*) marks a value less than
0.1%.

####  4.0.2 Future Scope

The performance of raw audio waveforms as input features to ConvNet can be
further improved by applying silence removal in the audio. Also, there is
scope for improvement by augmenting available data through various
conventional techniques like pitch shifting, adding random noise and changing
speed of audio. These help in making neural networks more robust to variations
which might be present in real world scenarios. There can be further
exploration of various feature extraction techniques like Constant-Q transform
and Fast Fourier Transform and assessment of their impact on Language
Identification.

There can be further improvements in neural network architectures like
concatenating the high level features obtained from 1D-ConvNet and 2D-ConvNet,
before performing classification. There can be experiments using deeper
networks with skip connections and Inception modules. These are known to have
positively impacted the performance of Convolutional Neural Networks.

##  5 Conclusion

There are two main contributions of this paper in the domain of spoken
language identification. Firstly, we presented an extensive analysis of raw
audio waveforms as input features to 1D-ConvNet. We experimented with various
hyperparameters in our 1D-ConvNet and evaluated their effect on validation
accuracy. This method is able to bypass the computational overhead of
conventional approaches which depend on generation of spectrograms as a
necessary pre-procesing step. We were able to achieve an accauracy of 93.7%
using this technique.

Next, we discussed the enhancement in performance of 2D-ConvNet using mixup
augmentation, which is a recently developed technique to prevent overfitting
on test data.This approach achieved an accuracy of 95.4%. We also analysed how
_attention_ mechanism and recurrent layers impact the performance of networks.
This approach achieved an accuracy of 95.0%.

## References

  * [1] Bartz, C., Herold, T., Yang, H., Meinel, C.: Language identification using deep convolutional recurrent neural networks. In: International Conference on Neural Information Processing. pp. 880–889. Springer (2017)
  * [2] Bergstra, J., Yamins, D., Cox, D.D.: Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures (2013)
  * [3] Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., Chua, T.S.: Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5659–5667 (2017)
  * [4] Dehak, N., Torres-Carrasquillo, P.A., Reynolds, D., Dehak, R.: Language recognition via i-vectors and dimensionality reduction. In: Twelfth annual conference of the international speech communication association (2011)
  * [5] Endah Safitri, N., Zahra, A., Adriani, M.: Spoken language identification with phonotactics methods on minangkabau, sundanese, and javanese languages. Procedia Computer Science 81, 182–187 (12 2016). https://doi.org/10.1016/j.procs.2016.04.047
  * [6] Ferrer, L., Scheffer, N., Shriberg, E.: A comparison of approaches for modeling prosodic features in speaker recognition. In: 2010 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 4414–4417. IEEE (2010)
  * [7] Ganapathy, S., Han, K., Thomas, S., Omar, M., Segbroeck, M.V., Narayanan, S.S.: Robust language identification using convolutional neural network features. In: Fifteenth annual conference of the international speech communication association (2014)
  * [8] Gazeau, V., Varol, C.: Automatic spoken language recognition with neural networks. Int. J. Inf. Technol. Comput. Sci.(IJITCS) 10(8), 11–17 (2018)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90, https://doi.org/10.1109/CVPR.2016.90
  * [10] Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., et al.: Cnn architectures for large-scale audio classification. In: 2017 ieee international conference on acoustics, speech and signal processing (icassp). pp. 131–135. IEEE (2017)
  * [11] Howard, J., et al.: fastai. https://github.com/fastai/fastai (2018)
  * [12] Kumar, P., Biswas, A., Mishra, A.N., Chandra, M.: Spoken language identification using hybrid feature extraction methods. arXiv preprint arXiv:1003.5623 (2010)
  * [13] Lee, J., Kim, T., Park, J., Nam, J.: Raw waveform-based audio classification using sample-level cnn architectures. arXiv preprint arXiv:1712.00866 (2017)
  * [14] LibROSA: https://librosa.github.io/librosa/. https://librosa.github.io/librosa/, accessed on 16 Jul 2019
  * [15] Lopez-Moreno, I., Gonzalez-Dominguez, J., Plchot, O., Martinez, D., Gonzalez-Rodriguez, J., Moreno, P.: Automatic language identification using deep neural networks. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 5337–5341. IEEE (2014)
  * [16] Martinez, D., Plchot, O., Burget, L., Glembek, O., Matějka, P.: Language recognition in ivectors space. In: Twelfth Annual Conference of the International Speech Communication Association (2011)
  * [17] Montavon, G.: Deep learning for spoken language identification. In: NIPS Workshop on deep learning for speech recognition and related applications. pp. 1–4 (2009)
  * [18] Obuchi, Y., Sato, N.: Language identification using phonetic and prosodic hmms with feature normalization. In: Proceedings.(ICASSP’05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. vol. 1, pp. I–569. IEEE (2005)
  * [19] Revay, S., Teschke, M.: Multiclass language identification using deep learning on spectral images of audio signals. arXiv preprint arXiv:1905.04348 (2019)
  * [20] Rong Tong, Bin Ma, Donglai Zhu, Haizhou Li, Eng Siong Chng: Integrating acoustic, prosodic and phonotactic features for spoken language identification. In: 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings. vol. 1, pp. I–I (May 2006). https://doi.org/10.1109/ICASSP.2006.1659993
  * [21] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1), 1929–1958 (2014)
  * [22] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
  * [23] voxforge.org: Free speech recognition (linux, windows and mac) - voxforge.org. http://www.voxforge.org/, accessed on 16 Jul 2019
  * [24] WEI, Q., LIU, Y., RUAN, X.: A report on audio tagging with deeper cnn, 1d-convnet and 2d-convnet
  * [25] Xu, K., Zhu, B., Kong, Q., Mi, H., Ding, B., Wang, D., Wang, H.: General audio tagging with ensembling convolutional neural networks and statistical features. The Journal of the Acoustical Society of America 145(6), EL521–EL527 (2019)
  * [26] Xu, Y., Huang, Q., Wang, W., Foster, P., Sigtia, S., Jackson, P.J., Plumbley, M.D.: Unsupervised feature learning based on deep models for environmental audio tagging. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25(6), 1230–1241 (2017)
  * [27] Youtube: Retrieved from www.youtube.com. http://www.youtube.com, accessed on 16 Jul 2019
  * [28] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [29] Zissman, M.A.: Comparison of four approaches to automatic language identification of telephone speech. IEEE Transactions on speech and audio processing 4(1), 31 (1996)


As in the continuous case (cf. Proposition 4), the proof proceeds in three
steps.

Step 1 (Weak convergence): By assumption,
$\|\underline{\mathbbmsl{v}}_{h}\|_{\boldsymbol{X}_{\boldsymbol{\tau}},h}\leq
c_{M}\eta_{\sharp}^{\frac{1}{2}}$ for all $h\in\mathcal{H}$. By definition
(62) of the $\|{\cdot}\|_{\boldsymbol{X}_{\boldsymbol{\tau}},h}$-norm, by the
boundedness results (59) and (61), and by the first hybrid Weber inequality
(41), this implies that (i) $(\boldsymbol{v}_{h})_{h\in\mathcal{H}}$ and
$\big{(}\boldsymbol{C}_{h}^{m}(\underline{\mathbbmsl{v}}_{h})\big{)}_{h\in\mathcal{H}}$
are uniformly bounded in $\boldsymbol{L}^{2}(\Omega)$, and (ii)
$\big{(}D_{h}^{p}(\underline{\mathbbmsl{v}}_{h})\big{)}_{h\in\mathcal{H}}$ is
uniformly bounded in $L^{2}(\Omega)$. By weak compactness, we then infer the
existence of $\boldsymbol{v}\in\boldsymbol{L}^{2}(\Omega)$,
$\boldsymbol{c}\in\boldsymbol{L}^{2}(\Omega)$, and $d\in L^{2}(\Omega)$ such
that, along a subsequence (not relabelled), (i)
$\boldsymbol{v}_{h}\rightharpoonup\boldsymbol{v}$ and
$\boldsymbol{C}_{h}^{m}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup\boldsymbol{c}$
weakly in $\boldsymbol{L}^{2}(\Omega)$, and (ii)
$D_{h}^{p}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup d$ weakly in
$L^{2}(\Omega)$.

Let us first prove that
$\boldsymbol{v}\in\boldsymbol{H}_{\boldsymbol{0}}(\operatorname{\bf
curl};\Omega)$ and that $\boldsymbol{c}=\operatorname{\bf
curl}\boldsymbol{v}$. Let
$\boldsymbol{z}\in\boldsymbol{C}^{\infty}(\overline{\Omega})$. There holds

$\begin{split}\big{(}\boldsymbol{C}_{h}^{m}(\underline{\mathbbmsl{v}}_{h}),\boldsymbol{z}\big{)}_{\Omega}&=\sum_{T\in\mathcal{T}_{h}}\big{(}\boldsymbol{C}_{T}^{m}(\underline{\mathbbmsl{v}}_{T}),\boldsymbol{\pi}_{T}^{m}(\boldsymbol{z}_{\mid
T})\big{)}_{T}\\\ &=\sum_{T\in\mathcal{T}_{h}}\bigg{(}\big{(}\operatorname{\bf
curl}\boldsymbol{v}_{T},\boldsymbol{\pi}_{T}^{m}(\boldsymbol{z}_{\mid
T})\big{)}_{T}\\\
&\qquad+\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}\boldsymbol{v}_{T\mid
F}{\times}\boldsymbol{n}_{F}-\boldsymbol{v}_{F,\boldsymbol{\tau}},\boldsymbol{n}_{F}{\times}(\boldsymbol{\pi}^{m}_{T}(\boldsymbol{z}_{\mid
T})_{\mid F}{\times}\boldsymbol{n}_{F})\big{)}_{F}\bigg{)}\\\
&=\big{(}\operatorname{\bf
curl}_{h}\boldsymbol{v}_{h},\boldsymbol{z}\big{)}_{\Omega}+\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}\boldsymbol{v}_{T\mid
F}{\times}\boldsymbol{n}_{F}-\boldsymbol{v}_{F,\boldsymbol{\tau}},\boldsymbol{n}_{F}{\times}(\boldsymbol{z}_{\mid
F}{\times}\boldsymbol{n}_{F})\big{)}_{F}+\mathcal{I}_{\operatorname{\bf
curl},h}\\\ &=\big{(}\boldsymbol{v}_{h},\operatorname{\bf
curl}\boldsymbol{z}\big{)}_{\Omega}+\mathcal{I}_{\operatorname{\bf
curl},h},\end{split}$ (63)

with $\mathcal{I}_{\operatorname{\bf curl},h}$ given by

$\mathcal{I}_{\operatorname{\bf
curl},h}\mathrel{\mathop{:}}=\big{(}\operatorname{\bf
curl}_{h}\boldsymbol{v}_{h},\boldsymbol{\pi}^{m}_{h}(\boldsymbol{z})-\boldsymbol{z}\big{)}_{\Omega}\\\
+\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}\boldsymbol{v}_{T\mid
F}{\times}\boldsymbol{n}_{F}-\boldsymbol{v}_{F,\boldsymbol{\tau}},\boldsymbol{n}_{F}{\times}((\boldsymbol{\pi}^{m}_{T}(\boldsymbol{z}_{\mid
T})-\boldsymbol{z})_{\mid F}{\times}\boldsymbol{n}_{F})\big{)}_{F},$

where we have (i) used that
$\boldsymbol{C}_{T}^{m}(\underline{\mathbbmsl{v}}_{T})\in\boldsymbol{\mathcal{P}}^{m}(T)$
for all $T\in\mathcal{T}_{h}$ in the first line, (ii) applied the definition
(58) of $\boldsymbol{C}_{T}^{m}(\underline{\mathbbmsl{v}}_{T})$ along with an
integration by parts of the term $\big{(}\boldsymbol{v}_{T},\operatorname{\bf
curl}\boldsymbol{\pi}^{m}_{T}(\boldsymbol{z}_{\mid T})\big{)}_{T}$ in the
second line, (iii) added/subtracted $\big{(}\operatorname{\bf
curl}_{h}\boldsymbol{v}_{h},\boldsymbol{z}\big{)}_{\Omega}$ and
$\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}\boldsymbol{v}_{T\mid
F}{\times}\boldsymbol{n}_{F}-\boldsymbol{v}_{F,\boldsymbol{\tau}},\boldsymbol{n}_{F}{\times}(\boldsymbol{z}_{\mid
F}{\times}\boldsymbol{n}_{F})\big{)}_{F}$ in the third line, and (iv)
performed cell-by-cell integration by parts of the term
$\big{(}\operatorname{\bf
curl}_{h}\boldsymbol{v}_{h},\boldsymbol{z}\big{)}_{\Omega}$ and used that
$\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}\boldsymbol{v}_{F,\boldsymbol{\tau}},\boldsymbol{n}_{F}{\times}(\boldsymbol{z}_{\mid
F}{\times}\boldsymbol{n}_{F})\big{)}_{F}=0$ (since $\boldsymbol{z}$ is single-
valued at interfaces and
$\boldsymbol{v}_{F,\boldsymbol{\tau}}\equiv\boldsymbol{0}$ for all
$F\in\mathcal{F}_{h}^{\partial}$ for
$\underline{\mathbbmsl{v}}_{h}\in\underline{\mathbbmsl{X}}_{h,\boldsymbol{\tau}}^{\ell}$)
in the fourth line. For the term $\mathcal{I}_{\operatorname{\bf curl},h}$ in
the right-hand side of (63), standard approximation properties for
$\boldsymbol{\pi}_{h}^{m}$ on mesh cells (cf. e.g. [22, Thm. 1.45]) on the one
side, and repeated applications of the triangle and Cauchy–Schwarz
inequalities along with the estimate (38) and approximation properties for
$\boldsymbol{\pi}_{h}^{m}$ on mesh faces on the other side, yield

$|\mathcal{I}_{\operatorname{\bf curl},h}|\lesssim
h^{m+1}|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{\bf
curl},h}|\boldsymbol{z}|_{m+1}.$

Starting from (63), and using that
$\boldsymbol{v}_{h}\rightharpoonup\boldsymbol{v}$ and
$\boldsymbol{C}_{h}^{m}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup\boldsymbol{c}$
weakly in $\boldsymbol{L}^{2}(\Omega)$, combined with the fact that
$\mathcal{I}_{\operatorname{\bf curl},h}\to 0$ as $h\to 0$ (since
$|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{\bf curl},h}\leq c_{M}$ for
all $h\in\mathcal{H}$ and $m+1\geq 1$), we infer that

$(\boldsymbol{c},\boldsymbol{z})_{\Omega}=(\boldsymbol{v},\operatorname{\bf
curl}\boldsymbol{z})_{\Omega}\qquad\forall\boldsymbol{z}\in\boldsymbol{C}^{\infty}(\overline{\Omega}).$

Since the latter relation is, in particular, valid for all
$\boldsymbol{z}\in\boldsymbol{C}^{\infty}_{\boldsymbol{0}}(\Omega)$, we infer
that $\boldsymbol{v}\in\boldsymbol{H}(\operatorname{\bf curl};\Omega)$ and
that $\boldsymbol{c}=\operatorname{\bf curl}\boldsymbol{v}$. Then, integrating
by parts the left-hand side of the relation, we get that
$\langle\boldsymbol{v}_{\mid\Gamma}{\times}\boldsymbol{n},\boldsymbol{n}{\times}(\boldsymbol{z}_{\mid\Gamma}{\times}\boldsymbol{n})\rangle_{\Gamma}=0$
for all $\boldsymbol{z}\in\boldsymbol{C}^{\infty}(\overline{\Omega})$, which
implies that
$\boldsymbol{v}\in\boldsymbol{H}_{\boldsymbol{0}}(\operatorname{\bf
curl};\Omega)$.

Let us now prove that
$\boldsymbol{v}\in\boldsymbol{H}(\operatorname{div}_{\eta};\Omega)$ and that
$d=\operatorname{div}(\eta\boldsymbol{v})$, along with the weak convergence
result
$\gamma_{n,h}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup(\eta\boldsymbol{v})_{\mid\Gamma}{\cdot}\boldsymbol{n}$
in $H^{-\frac{1}{2}}(\Gamma)$. Let $z\in C^{\infty}(\overline{\Omega})$.
Adapting the arguments used in (63) in the case of the rotational operator to
the case of the divergence operator, we infer that

$\big{(}D_{h}^{p}(\underline{\mathbbmsl{v}}_{h}),z\big{)}_{\Omega}=-\big{(}\eta\,\boldsymbol{v}_{h},\operatorname{\bf
grad}z\big{)}_{\Omega}+\big{(}\gamma_{n,h}(\underline{\mathbbmsl{v}}_{h}),z_{\mid\Gamma}\big{)}_{\Gamma}+\mathcal{I}_{\operatorname{div},h},$
(64)

where $\mathcal{I}_{\operatorname{div},h}$ is given by

$\mathcal{I}_{\operatorname{div},h}\mathrel{\mathop{:}}=\big{(}\operatorname{div}_{h}(\eta\boldsymbol{v}_{h}),\pi^{p}_{h}(z)-z\big{)}_{\Omega}+\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}[\eta
v]_{F,n}-\eta_{T}\boldsymbol{v}_{T\mid
F}{\cdot}\boldsymbol{n}_{F},(\pi_{T}^{p}(z_{\mid T})-z)_{\mid F}\big{)}_{F}.$

Adapting the arguments from the rotational case, it is an easy matter to prove
that

$|\mathcal{I}_{\operatorname{div},h}|\lesssim
h^{p+1}|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{div},h}|z|_{p+1}.$

Let us first choose $z\in C^{\infty}_{0}(\Omega)$. Remark that the boundary
term in (64) vanishes. Starting from (64), and using that
$\boldsymbol{v}_{h}\rightharpoonup\boldsymbol{v}$ weakly in
$\boldsymbol{L}^{2}(\Omega)$ and
$D_{h}^{p}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup d$ weakly in
$L^{2}(\Omega)$, combined with the fact that
$\mathcal{I}_{\operatorname{div},h}\to 0$ as $h\to 0$ (since
$|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{div},h}\leq
c_{M}\eta_{\sharp}$ for all $h\in\mathcal{H}$ and $p+1\geq 1$), we infer that

$(d,z)_{\Omega}=-(\eta\,\boldsymbol{v},\operatorname{\bf
grad}z)_{\Omega}\qquad\forall z\in C^{\infty}_{0}(\Omega).$

The latter relation directly implies that
$\boldsymbol{v}\in\boldsymbol{H}(\operatorname{div}_{\eta};\Omega)$ and that
$d=\operatorname{div}(\eta\boldsymbol{v})$. Now, passing to the limit $h\to 0$
in (64) for a generic $z\in C^{\infty}(\overline{\Omega})$, we infer that

$\lim_{h\to
0}\big{(}\gamma_{n,h}(\underline{\mathbbmsl{v}}_{h}),z_{\mid\Gamma}\big{)}_{\Gamma}=(\operatorname{div}(\eta\boldsymbol{v}),z)_{\Omega}+(\eta\,\boldsymbol{v},\operatorname{\bf
grad}z)_{\Omega}=\langle(\eta\boldsymbol{v})_{\mid\Gamma}{\cdot}\boldsymbol{n},z_{\mid\Gamma}\rangle_{\Gamma},$

which implies that
$\gamma_{n,h}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup(\eta\boldsymbol{v})_{\mid\Gamma}{\cdot}\boldsymbol{n}$
weakly in $H^{-\frac{1}{2}}(\Gamma)$.

Step 2 (Characterization of the limit): This step is identical to Step 2 from
the continuous case (cf. the proof of Proposition 4), up to the replacement of
$m\in\mathbb{N}$ by $h\in\mathcal{H}$ (and $m\to\infty$ by $h\to 0$) and of
$\boldsymbol{v}_{m}\in\boldsymbol{X}_{\boldsymbol{\tau}}(\Omega)$ by
$\boldsymbol{v}_{h}\in\boldsymbol{\mathcal{P}}^{\ell}(\mathcal{T}_{h})$. For
any $h\in\mathcal{H}$, by (7), there exist $\varphi_{h}\in H^{1}_{0}(\Omega)$,
$\boldsymbol{\psi}_{h}\in\boldsymbol{H}^{1}(\Omega)\cap\boldsymbol{L}^{2}_{\boldsymbol{0}}(\Omega)$,
and $\boldsymbol{w}_{h}\in\boldsymbol{H}_{\boldsymbol{\tau}}(\Omega)$, such
that

$\boldsymbol{v}_{h}=\operatorname{\bf
grad}\varphi_{h}+\frac{\eta_{\sharp}}{\eta}\operatorname{\bf
curl}\boldsymbol{\psi}_{h}+\boldsymbol{w}_{h},$ (65)

with $\|\boldsymbol{\psi}_{h}\|_{1}\leq C_{\Omega,1}\|\operatorname{\bf
curl}\boldsymbol{\psi}_{h}\|_{0}$ by (8). Proceeding as in the continuous
case, one can infer the existence of
$\boldsymbol{w}\in\boldsymbol{H}_{\boldsymbol{\tau}}(\Omega)$ such that, up to
extraction (not relabelled), $\boldsymbol{w}_{h}\to\boldsymbol{w}$ strongly in
$\boldsymbol{L}^{2}(\Omega)$, and of $\varphi\in H^{1}_{0}(\Omega)$ and
$\boldsymbol{\psi}\in\boldsymbol{H}^{1}(\Omega)\cap\boldsymbol{L}^{2}_{\boldsymbol{0}}(\Omega)$
such that, up to extractions (not relabelled), $\operatorname{\bf
grad}\varphi_{h}\rightharpoonup\operatorname{\bf grad}\varphi$ and
$\operatorname{\bf curl}\boldsymbol{\psi}_{h}\rightharpoonup\operatorname{\bf
curl}\boldsymbol{\psi}$ weakly in $\boldsymbol{L}^{2}(\Omega)$. By linearity,
we have thus proven that, along a subsequence (not relabelled),
$\boldsymbol{v}_{h}\rightharpoonup\operatorname{\bf
grad}\varphi+\frac{\eta_{\sharp}}{\eta}\operatorname{\bf
curl}\boldsymbol{\psi}+\boldsymbol{w}$ weakly in $\boldsymbol{L}^{2}(\Omega)$.
By uniqueness of the weak limit, $\boldsymbol{v}=\operatorname{\bf
grad}\varphi+\frac{\eta_{\sharp}}{\eta}\operatorname{\bf
curl}\boldsymbol{\psi}+\boldsymbol{w}$.

Step 3 (Strong convergence): Here again, the arguments are inspired by what is
done in Step 3 from the continuous case, but with the important difference
that here
$\boldsymbol{v}_{h}\in\boldsymbol{\mathcal{P}}^{\ell}(\mathcal{T}_{h})\not\subset\boldsymbol{X}_{\boldsymbol{\tau}}(\Omega)$.
By Rellich’s compactness theorem, we actually have that, along the same
subsequence (not relabelled) as in Step 2 above, $\varphi_{h}\to\varphi$
strongly in $L^{2}(\Omega)$ and $\boldsymbol{\psi}_{h}\to\boldsymbol{\psi}$
strongly in $\boldsymbol{L}^{2}(\Omega)$. We now want to prove the strong
convergences of $(\operatorname{\bf grad}\varphi_{h})_{h\in\mathcal{H}}$ and
$(\operatorname{\bf curl}\boldsymbol{\psi}_{h})_{h\in\mathcal{H}}$ in
$\boldsymbol{L}^{2}(\Omega)$ (we remind the reader that we already proved
strong convergence for $(\boldsymbol{w}_{h})_{h\in\mathcal{H}}$).

Recalling the expression of
$\boldsymbol{v}\in\boldsymbol{X}_{\boldsymbol{\tau}}(\Omega)$ derived in Step
2, since the decomposition (65) is
$\boldsymbol{L}^{2}_{\eta}(\Omega)$-orthogonal, there holds

$\begin{split}\|\eta^{\frac{1}{2}}(\operatorname{\bf
grad}\varphi_{h}-\operatorname{\bf
grad}\varphi)\|_{0}^{2}&=\big{(}\eta(\boldsymbol{v}_{h}-\boldsymbol{v}),\operatorname{\bf
grad}(\varphi_{h}-\varphi)\big{)}_{\Omega}\\\
&=\sum_{T\in\mathcal{T}_{h}}\bigg{(}-\big{(}\operatorname{div}(\eta_{T}(\boldsymbol{v}_{T}-\boldsymbol{v})),\varphi_{h}-\varphi\big{)}_{T}\\\
&\qquad+\sum_{F\in\mathcal{F}_{T}}\big{(}\eta_{T}\boldsymbol{v}_{T\mid
F}{\cdot}\boldsymbol{n}_{T,F},(\varphi_{h}-\varphi)_{\mid
F}\big{)}_{F}\bigg{)}\\\
&=-\big{(}\operatorname{div}_{h}(\eta(\boldsymbol{v}_{h}-\boldsymbol{v})),\varphi_{h}-\varphi\big{)}_{\Omega}\\\
&\qquad+\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}\eta_{T}\boldsymbol{v}_{T\mid
F}{\cdot}\boldsymbol{n}_{F}-[\eta v]_{F,n},(\varphi_{h}-\varphi)_{\mid
F}\big{)}_{F},\end{split}$

where, to pass from the first to the second line, we have performed cell-by-
cell integration by parts and used that
$\sum_{T\in\mathcal{T}_{h}}\big{\langle}(\eta\boldsymbol{v})_{\mid\partial
T}{\cdot}\boldsymbol{n}_{\partial T},(\varphi_{h}-\varphi)_{\mid\partial
T}\big{\rangle}_{\partial T}=0$ (as a consequence of the fact that
$\boldsymbol{v}\in\boldsymbol{H}(\operatorname{div}_{\eta};\Omega)$ and that
$\varphi_{h},\varphi\in H^{1}_{0}(\Omega)$) and, to pass from the second to
the third line, we have again used the fact that $(\varphi_{h}-\varphi)$ is
single-valued at interfaces and vanishes on the boundary of the domain to
subtract the zero contribution
$\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}[\eta
v]_{F,n},(\varphi_{h}-\varphi)_{\mid F}\big{)}_{F}$. By Cauchy–Schwarz
inequality, combined with a continuous trace inequality (along with mesh
regularity), and the fact that
$|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{div},h}\leq
c_{M}\eta_{\sharp}$ for all $h\in\mathcal{H}$ and
$\|\operatorname{div}(\eta\boldsymbol{v})\|_{0}\lesssim\lim\inf_{h\to
0}|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{div},h}$ (by weak convergence
of $(D_{h}^{p}(\underline{\mathbbmsl{v}}_{h}))_{h\in\mathcal{H}}$ to
$\operatorname{div}(\eta\boldsymbol{v})$ in $L^{2}(\Omega)$ and the uniform
bound (61)), we infer that

$\|\eta^{\frac{1}{2}}(\operatorname{\bf grad}\varphi_{h}-\operatorname{\bf
grad}\varphi)\|_{0}^{2}\lesssim\eta_{\sharp}\big{(}\|\varphi_{h}-\varphi\|_{0}+h\|\operatorname{\bf
grad}\varphi_{h}-\operatorname{\bf grad}\varphi\|_{0}\big{)}.$ (66)

Since $\|\operatorname{\bf
grad}\varphi_{h}\|_{0}\leq\eta_{\flat}^{-\frac{1}{2}}\|\eta^{\frac{1}{2}}\boldsymbol{v}_{h}\|_{0}$
(by $\boldsymbol{L}^{2}_{\eta}(\Omega)$-orthogonality of the decomposition
(65)), $\|\eta^{\frac{1}{2}}\boldsymbol{v}_{h}\|_{0}\leq
c_{W,1}\sqrt{3}\kappa_{\eta}^{\frac{1}{2}}c_{M}\eta_{\sharp}^{\frac{1}{2}}$
(by (41), (62), and
$\|\underline{\mathbbmsl{v}}_{h}\|_{\boldsymbol{X}_{\boldsymbol{\tau}},h}\leq
c_{M}\eta_{\sharp}^{\frac{1}{2}}$), and $\|\operatorname{\bf
grad}\varphi\|_{0}\leq\lim\inf_{h\to 0}\|\operatorname{\bf
grad}\varphi_{h}\|_{0}$ (since $\operatorname{\bf
grad}\varphi_{h}\rightharpoonup\operatorname{\bf grad}\varphi$ weakly in
$\boldsymbol{L}^{2}(\Omega)$), we infer by the triangle inequality that

$\|\operatorname{\bf grad}\varphi_{h}-\operatorname{\bf grad}\varphi\|_{0}\leq
2c_{W,1}\sqrt{3}\kappa_{\eta}c_{M}.$

Combining this uniform boundedness result to the fact that
$\varphi_{h}\to\varphi$ strongly in $L^{2}(\Omega)$ to pass to the limit $h\to
0$ in (66), we conclude that $\operatorname{\bf
grad}\varphi_{h}\to\operatorname{\bf grad}\varphi$ strongly in
$\boldsymbol{L}^{2}(\Omega)$.

Adapting the latter arguments, there also holds

$\begin{split}\eta_{\sharp}\|\eta^{-\frac{1}{2}}(\operatorname{\bf
curl}\boldsymbol{\psi}_{h}-\operatorname{\bf
curl}\boldsymbol{\psi})\|_{0}^{2}&=\big{(}\boldsymbol{v}_{h}-\boldsymbol{v},\operatorname{\bf
curl}(\boldsymbol{\psi}_{h}-\boldsymbol{\psi})\big{)}_{\Omega}\\\
&=\big{(}\operatorname{\bf
curl}_{h}(\boldsymbol{v}_{h}-\boldsymbol{v}),\boldsymbol{\psi}_{h}-\boldsymbol{\psi}\big{)}_{\Omega}\\\
&\qquad+\sum_{T\in\mathcal{T}_{h}}\sum_{F\in\mathcal{F}_{T}}\varepsilon_{T,F}\big{(}\boldsymbol{v}_{T\mid
F}{\times}\boldsymbol{n}_{F}-\boldsymbol{v}_{F,\boldsymbol{\tau}},\boldsymbol{n}_{F}{\times}((\boldsymbol{\psi}_{h}-\boldsymbol{\psi})_{\mid
F}{\times}\boldsymbol{n}_{F})\big{)}_{F},\end{split}$

where we have used that
$\underline{\mathbbmsl{v}}_{h}\in\underline{\mathbbmsl{X}}_{h,\boldsymbol{\tau}}^{\ell}$,
$\boldsymbol{v}\in\boldsymbol{H}_{\boldsymbol{0}}(\operatorname{\bf
curl};\Omega)$, and
$\boldsymbol{\psi}_{h},\boldsymbol{\psi}\in\boldsymbol{H}^{1}(\Omega)$.
Additionally invoking (38), this implies that

$\eta_{\sharp}\|\eta^{-\frac{1}{2}}(\operatorname{\bf
curl}\boldsymbol{\psi}_{h}-\operatorname{\bf
curl}\boldsymbol{\psi})\|_{0}^{2}\lesssim\big{(}\|\boldsymbol{\psi}_{h}-\boldsymbol{\psi}\|_{0}+h|\boldsymbol{\psi}_{h}-\boldsymbol{\psi}|_{1}\big{)},$

which eventually yields, since
$|\boldsymbol{\psi}_{h}-\boldsymbol{\psi}|_{1}\leq
2C_{\Omega,1}c_{W,1}\sqrt{3}\kappa_{\eta}^{\frac{1}{2}}c_{M}$ (here, we
additionally used the estimate $|\boldsymbol{\psi}_{h}|_{1}\leq
C_{\Omega,1}\|\operatorname{\bf curl}\boldsymbol{\psi}_{h}\|_{0}$ and the fact
that $\boldsymbol{\psi}_{h}\rightharpoonup\boldsymbol{\psi}$ weakly in
$\boldsymbol{H}^{1}(\Omega)$), and $\boldsymbol{\psi}_{h}\to\boldsymbol{\psi}$
strongly in $\boldsymbol{L}^{2}(\Omega)$, that $\operatorname{\bf
curl}\boldsymbol{\psi}_{h}\to\operatorname{\bf curl}\boldsymbol{\psi}$
strongly in $\boldsymbol{L}^{2}(\Omega)$.

Thus, along a subsequence (not relabelled),
$\boldsymbol{v}_{h}\to\boldsymbol{v}$ strongly in
$\boldsymbol{L}^{2}(\Omega)$, yielding the conclusion. ∎

###  5.3 Compactness in $\underline{\mathbbmsl{X}}_{h,n}^{\ell}$

Recall the definitions (33) of the discrete space
$\underline{\mathbbmsl{X}}_{h,n}^{\ell}$, and (35)–(36) of the hybrid semi-
norms $|{\cdot}|_{\operatorname{\bf curl},h}$ and
$|{\cdot}|_{\operatorname{div},h}$. Leveraging the second hybrid Weber
inequality from Theorem 15, we infer that

$\|\underline{\mathbbmsl{v}}_{h}\|_{\boldsymbol{X}_{n},h}^{2}\mathrel{\mathop{:}}=\eta_{\flat}^{-1}|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{div},h}^{2}+\eta_{\sharp}|\underline{\mathbbmsl{v}}_{h}|_{\operatorname{\bf
curl},h}^{2}+\eta_{\flat}^{-1}\sum_{i=1}^{\beta_{1}}\big{|}\big{(}\gamma_{n,h}(\underline{\mathbbmsl{v}}_{h}),1\big{)}_{\Sigma_{i}}\big{|}^{2}$
(67)

defines a norm on $\underline{\mathbbmsl{X}}_{h,n}^{\ell}$, which can be
proven uniformly (in $h$) equivalent to the
$\|{\cdot}\|_{\boldsymbol{X},h}$-norm (37): there is $c_{n}>0$, independent of
$h$ and $\eta$, such that

$(c_{n}\kappa_{\eta})^{-1}\|\underline{\mathbbmsl{v}}_{h}\|_{\boldsymbol{X},h}^{2}\leq\|\underline{\mathbbmsl{v}}_{h}\|_{\boldsymbol{X}_{n},h}^{2}\leq
c_{n}\kappa_{\eta}\|\underline{\mathbbmsl{v}}_{h}\|_{\boldsymbol{X},h}^{2}\qquad\forall\underline{\mathbbmsl{v}}_{h}\in\underline{\mathbbmsl{X}}_{h,n}^{\ell}.$

######  Theorem 18 (Maxwell compactness in
$\underline{\mathbbmsl{X}}_{h,n}^{\ell}$).

Let $\big{(}\mathcal{M}_{h}\big{)}_{h\in\mathcal{H}}$ be a regular mesh family
as defined in Section 3.1. Let
$(\underline{\mathbbmsl{v}}_{h})_{h\in\mathcal{H}}$ be a sequence of elements
of $\underline{\mathbbmsl{X}}_{h,n}^{\ell}$ for which there exists a real
number $c_{M}>0$ (independent of $h$) such that
$\eta_{\sharp}^{-\frac{1}{2}}\|\underline{\mathbbmsl{v}}_{h}\|_{\boldsymbol{X}_{n},h}\leq
c_{M}$ for all $h\in\mathcal{H}$. Then, there exists an element
$\boldsymbol{v}\in\boldsymbol{X}_{n}(\Omega)$ such that, along a subsequence,
as $h\to 0$,

  1. (a)

$\boldsymbol{v}_{h}\rightarrow\boldsymbol{v}$ strongly in
$\boldsymbol{L}^{2}(\Omega)$;

  2. (b)

$\boldsymbol{C}_{h}^{m}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup\operatorname{\bf
curl}\boldsymbol{v}$ weakly in $\boldsymbol{L}^{2}(\Omega)$;

  3. (c)

$D_{h}^{p}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup\operatorname{div}(\eta\boldsymbol{v})$
weakly in $L^{2}(\Omega)$;

  4. (d)

$\boldsymbol{\gamma}_{\boldsymbol{\tau},h}(\underline{\mathbbmsl{v}}_{h})\rightharpoonup\boldsymbol{v}_{\mid\Gamma}{\times}\boldsymbol{n}$
weakly in $\boldsymbol{H}^{-\frac{1}{2}}(\Gamma)$.

###### Proof.

The proof proceeds in three steps, as in the continuous case (cf. Proposition
7). For the sake of brevity, since, up to inverting the roles of the
rotational- and divergence-related parts in the proof, the three steps are
essentially identical to those from Theorem 17, we do not detail them. ∎

## Acknowledgments

The work of the authors is supported by the action “Préservation de l’emploi
de R&D” from the “Plan de Relance” recovery plan of the French State. Support
from the LabEx CEMPI (ANR-11-LABX-0007) is also acknowledged.

##  Appendix A Continuity modulus of $\boldsymbol{\operatorname{div}}^{-1}$ on
star-shaped cells

######  Lemma 19 (Continuity modulus of
$\boldsymbol{\operatorname{div}}^{-1}$).

Let $\ell\in\mathbb{N}$. Let $T\in\mathcal{T}_{h}$, and assume that $T$ is
star-shaped with respect to $\boldsymbol{x}_{T}$. Then, for all
$d\in\mathcal{P}^{\ell-1}(T)$,

$\|\boldsymbol{\operatorname{div}}^{-1}d\|_{0,T}\leq\frac{2}{3}h_{T}\|d\|_{0,T}.$
(68)

###### Proof.

The mapping $\operatorname{div}:\boldsymbol{\mathcal{R}}^{{\rm
c},\ell}(T)\to\mathcal{P}^{\ell-1}(T)$ being an isomorphism, let
$r\in\mathcal{P}^{\ell-1}(T)$ be the unique solution to

$d=\operatorname{div}\big{(}r(\boldsymbol{x}-\boldsymbol{x}_{T})\big{)}.$

Using a simple vector calculus identity to expand the right-hand side, there
holds

$d=3r+\operatorname{\bf grad}r{\cdot}(\boldsymbol{x}-\boldsymbol{x}_{T}).$

Now, multiplying both sides by $\overline{r}$, integrating over $T$, and
taking the real part, we get

$\displaystyle\mathfrak{R}\big{(}(d,r)_{T}\big{)}$
$\displaystyle=3\|r\|_{0,T}^{2}+\frac{1}{2}\big{(}\operatorname{\bf
grad}(|r|^{2}),(\boldsymbol{x}-\boldsymbol{x}_{T})\big{)}_{T}$
$\displaystyle=3\|r\|_{0,T}^{2}-\frac{3}{2}\|r\|_{0,T}^{2}+\frac{1}{2}\big{(}|r|^{2},(\boldsymbol{x}-\boldsymbol{x}_{T}){\cdot}\boldsymbol{n}_{\partial
T}\big{)}_{\partial T}$ $\displaystyle\geq\frac{3}{2}\|r\|_{0,T}^{2},$

where we have used an integration by parts formula to pass to the second line,
and the fact that $T$ is star-shaped with respect to $\boldsymbol{x}_{T}$ to
conclude. Finally, applying the Cauchy–Schwarz inequality to $(d,r)_{T}$, we
infer

$\|\boldsymbol{\operatorname{div}}^{-1}d\|_{0,T}=\|r(\boldsymbol{x}-\boldsymbol{x}_{T})\|_{0,T}\leq
h_{T}\|r\|_{0,T}\leq\frac{2}{3}h_{T}\|d\|_{0,T},$

which proves (68). ∎

## References

  * [1] C. Amrouche, C. Bernardi, M. Dauge, and V. Girault.  Vector potentials in three-dimensional non-smooth domains.  Math. Methods Appl. Sci., 21(9):823–864, 1998.
  * [2] D. N. Arnold.  Finite element exterior calculus, volume 93 of CBMS-NSF Regional Conference Series in Applied Mathematics.  Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2018.
  * [3] D. N. Arnold, R. S. Falk, and R. Winther.  Finite element exterior calculus, homological techniques, and applications.  Acta Numer., 15:1–155, 2006.
  * [4] F. Assous, P. Ciarlet Jr., and S. Labrunie.  Mathematical foundations of computational electromagnetism, volume 198 of Applied Mathematical Sciences.  Springer, Cham, 2018.
  * [5] B. Ayuso de Dios, K. Lipnikov, and G. Manzini.  The nonconforming virtual element method.  ESAIM Math. Model. Numer. Anal., 50(3):879–904, 2016.
  * [6] S. Bauer, D. Pauly, and M. Schomburg.  The Maxwell compactness property in bounded weak Lipschitz domains with mixed boundary conditions.  SIAM J. Math. Anal., 48(4):2912–2943, 2016.
  * [7] L. Beirão da Veiga, F. Brezzi, A. Cangiani, G. Manzini, L. D. Marini, and A. Russo.  Basic principles of virtual element methods.  Math. Models Methods Appl. Sci., 23(1):199–214, 2013.
  * [8] F. Bonaldi, D. A. Di Pietro, J. Droniou, and K. Hu.  An exterior calculus framework for polytopal methods, 2023.  Submitted, preprint arXiv:2303.11093.
  * [9] A. Bonito, J.-L. Guermond, and F. Luddens.  Regularity of the Maxwell equations in heterogeneous media and Lipschitz domains.  J. Math. Anal. Appl., 408(2):498–512, 2013.
  * [10] A. Bossavit.  Computational electromagnetism – Variational formulations, complementarity, edge elements.  Electromagnetism. Academic Press, Inc., San Diego, CA, 1998.
  * [11] D. Braess and J. Schöberl.  Equilibrated residual error estimator for edge elements.  Math. Comp., 77(262):651–672, 2008.
  * [12] F. Chave, D. A. Di Pietro, and S. Lemaire.  A discrete Weber inequality on three-dimensional hybrid spaces with application to the HHO approximation of magnetostatics.  Math. Models Methods Appl. Sci., 32(1):175–207, 2022.
  * [13] G. Chen, J. Cui, and L. Xu.  Analysis of a Hybridizable Discontinuous Galerkin method for the Maxwell operator.  ESAIM Math. Model. Numer. Anal., 53(1):301–324, 2019.
  * [14] H. Chen, W. Qiu, K. Shi, and M. Solano.  A superconvergent HDG method for the Maxwell equations.  J. Sci. Comput., 70(3):1010–1029, 2017.
  * [15] S. H. Christiansen and M. W. Licht.  Poincaré–Friedrichs inequalities of complexes of discrete distributional differential forms.  BIT Numer. Math., 60(2):345–371, 2020.
  * [16] M. Cicuttin, A. Ern, and N. Pignet.  Hybrid high-order methods – A primer with applications to solid mechanics.  SpringerBriefs in Mathematics. Springer, Cham, 2021.
  * [17] B. Cockburn, D. A. Di Pietro, and A. Ern.  Bridging the Hybrid High-Order and Hybridizable Discontinuous Galerkin methods.  ESAIM Math. Model. Numer. Anal., 50(3):635–650, 2016.
  * [18] B. Cockburn, J. Gopalakrishnan, and R. Lazarov.  Unified hybridization of discontinuous Galerkin, mixed, and continuous Galerkin methods for second order elliptic problems.  SIAM J. Numer. Anal., 47(2):1319–1365, 2009.
  * [19] M. Costabel.  A remark on the regularity of solutions of Maxwell’s equations on Lipschitz domains.  Math. Methods Appl. Sci., 12(4):365–368, 1990.
  * [20] M. Costabel, M. Dauge, and S. Nicaise.  Singularities of Maxwell interface problems.  M2AN Math. Model. Numer. Anal., 33(3):627–649, 1999.
  * [21] R. Dautray and J.-L. Lions.  Mathematical analysis and numerical methods for science and technology – Vol. 3: Spectral theory and applications.  Springer-Verlag, Berlin, 1990.  With the collaboration of Michel Artola and Michel Cessenat. Translated from the French by John C. Amson.
  * [22] D. A. Di Pietro and J. Droniou.  The Hybrid High-Order method for polytopal meshes – Design, analysis, and applications, volume 19 of Modeling, Simulation and Applications.  Springer, Cham, 2020.
  * [23] D. A. Di Pietro and J. Droniou.  An arbitrary-order Discrete De Rham complex on polyhedral meshes: Exactness, Poincaré inequalities, and consistency.  Found. Comput. Math., 23(1):85–164, 2023.
  * [24] D. A. Di Pietro and A. Ern.  A hybrid high-order locking-free method for linear elasticity on general meshes.  Comput. Methods Appl. Mech. Engrg., 283:1–21, 2015.
  * [25] D. A. Di Pietro, A. Ern, and S. Lemaire.  An arbitrary-order and compact-stencil discretization of diffusion on general meshes based on local reconstruction operators.  Comput. Methods Appl. Math., 14(4):461–472, 2014.
  * [26] S. Du and F.-J. Sayas.  A unified error analysis of Hybridizable Discontinuous Galerkin methods for the static Maxwell equations.  SIAM J. Numer. Anal., 58(2):1367–1391, 2020.
  * [27] A. Ern and J.-L. Guermond.  Quasi-optimal nonconforming approximation of elliptic PDEs with contrasted coefficients and $H^{1+r}$, $r>0$, regularity.  Found. Comput. Math., 22(5):1273–1308, 2022.
  * [28] P. Fernandes and G. Gilardi.  Magnetostatic and electrostatic problems in inhomogeneous anisotropic media with irregular boundary and mixed boundary conditions.  Math. Models Methods Appl. Sci., 7(7):957–991, 1997.
  * [29] V. Girault and P.-A. Raviart.  Finite element methods for Navier–Stokes equations – Theory and algorithms, volume 5 of Springer Series in Computational Mathematics.  Springer-Verlag, Berlin, 1986.
  * [30] P. W. Gross and P. R. Kotiuga.  Electromagnetic theory and computation: a topological approach, volume 48 of Mathematical Sciences Research Institute Publications.  Cambridge University Press, Cambridge, 2004.
  * [31] F. Jochmann.  A compactness result for vector fields with divergence and curl in $L^{q}(\Omega)$ involving mixed boundary conditions.  Appl. Anal., 66(1-2):189–203, 1997.
  * [32] F. Jochmann.  Regularity of weak solutions of Maxwell’s equations with mixed boundary conditions.  Math. Methods Appl. Sci., 22(14):1255–1274, 1999.
  * [33] C. Lehrenfeld and J. Schöberl.  High order exactly divergence-free hybrid discontinuous Galerkin methods for unsteady incompressible flows.  Comput. Methods Appl. Mech. Engrg., 307:339–361, 2016.
  * [34] S. Lemaire.  Bridging the Hybrid High-Order and Virtual Element methods.  IMA J. Numer. Anal., 41(1):549–593, 2021.
  * [35] M. W. Licht.  Complexes of discrete distributional differential forms and their homology theory.  Found. Comput. Math., 17(4):1085–1122, 2017.
  * [36] P. Monk.  Finite element methods for Maxwell’s equations.  Numerical Mathematics and Scientific Computation. Oxford University Press, New York, 2003.
  * [37] L. Mu, J. Wang, X. Ye, and S. Zhang.  A weak Galerkin finite element method for the Maxwell equations.  J. Sci. Comput., 65(1):363–386, 2015.
  * [38] J.-C. Nédélec.  Mixed finite elements in $\mathbb{R}^{3}$.  Numer. Math., 35(3):315–341, 1980.
  * [39] J.-C. Nédélec.  A new family of mixed finite elements in $\mathbb{R}^{3}$.  Numer. Math., 50(1):57–81, 1986.
  * [40] N. C. Nguyen, J. Peraire, and B. Cockburn.  Hybridizable Discontinuous Galerkin methods for the time-harmonic Maxwell’s equations.  J. Comput. Phys., 230(19):7151–7175, 2011.
  * [41] J. Wang and X. Ye.  A weak Galerkin finite element method for second-order elliptic problems.  J. Comput. Appl. Math., 241:103–115, 2013.
  * [42] C. Weber.  A local compactness theorem for Maxwell’s equations.  Math. Methods Appl. Sci., 2(1):12–25, 1980.
  * [43] N. Weck.  Maxwell’s boundary value problem on Riemannian manifolds with nonsmooth boundaries.  J. Math. Anal. Appl., 46:410–437, 1974.
  * [44] H. Whitney.  Geometric integration theory.  Princeton University Press, Princeton, N. J., 1957.


Send correspondence to<EMAIL_ADDRESS>
# The Blue Multi Unit Spectroscopic Explorer (BlueMUSE) on the VLT:
characterization of two VPHG prototypes based on dichromated gelatin and
photopolymer recording materials

Alexandre Jeanneau  Université Lyon, Université Claude Bernard Lyon 1, Ens de
Lyon, CNRS, Centre de Recherche Astrophysique de Lyon, F-69230, Saint-Genis-
Laval, France  Andrea Bianco  INAF – Osservatorio Astronomico di Brera, via E.
Bianchi 46, 23807, Merate, LC, Italy  Andrew Clawson  Wasatch Photonics, 1305
North 1000 West, Suite 120, Logan, UT 84321 USA  Michele Frangiamore  INAF –
Osservatorio Astronomico di Brera, via E. Bianchi 46, 23807, Merate, LC, Italy
Elroy Pearson  Wasatch Photonics, 1305 North 1000 West, Suite 120, Logan, UT
84321 USA  Laurent Pinard  Université Lyon, Université Claude Bernard Lyon 1,
CNRS, Laboratoire des Matériaux Avancés, F-69622, Villeurbanne, France  Jürgen
Schmoll  Centre for Advanced Instrumentation (CfAI), Durham University, South
Road, Durham DH1 3LE, United Kingdom  Johan Richard  Université Lyon,
Université Claude Bernard Lyon 1, Ens de Lyon, CNRS, Centre de Recherche
Astrophysique de Lyon, F-69230, Saint-Genis-Laval, France  Rémi Giroud
Université Lyon, Université Claude Bernard Lyon 1, Ens de Lyon, CNRS, Centre
de Recherche Astrophysique de Lyon, F-69230, Saint-Genis-Laval, France
Florence Laurent  Université Lyon, Université Claude Bernard Lyon 1, Ens de
Lyon, CNRS, Centre de Recherche Astrophysique de Lyon, F-69230, Saint-Genis-
Laval, France  Roland Bacon  Université Lyon, Université Claude Bernard Lyon
1, Ens de Lyon, CNRS, Centre de Recherche Astrophysique de Lyon, F-69230,
Saint-Genis-Laval, France

###### Abstract

Volume-phase holographic gratings (VPHGs) are widely used in astronomical
spectrographs due to their adaptability and high diffraction efficiency. Most
VPHGs in operation use dichromated gelatin as a recording material, whose
performance is sensitive to the coating and development process, especially in
the near-UV. In this letter, we present the characterization of two UV-blue
VPHG prototypes for the BlueMUSE integral field spectrograph on the VLT, based
on dichromated gelatin and the Bayfol®HX photopolymer film as recording
materials. Our measurements show that both prototypes meet the required
diffraction efficiency and exhibit similar performance with a wavelength-
average exceeding 70% in the 350-580 nm range. Deviations from theoretical
models increase towards 350 nm, consistently with previous studies on similar
gratings. We also report similar performances in terms spatial uniformity and
grating-to-grating consistency. Likewise, no significant differences in
wavefront error or scattered light are observed between the prototypes.

###### keywords:

VPHG, dichromated gelatin, photopolymer, diffraction efficiency, spectrograph,
BlueMUSE

##  1 Introduction

Volume-phase holographic gratings (VPHGs) operate through bulk refractive
index modulations, which are created by holographically exposing a
photosensitive material such as dichromated gelatin (DCG). Despite their
widespread use in astronomical spectrographs over the past decades — owing to
their adaptability and high diffraction efficiency[1, 2] — the choice of VPHG
suppliers is rather limited. This is partly due to the sensitivity of the DCG
coating and development process[3], particularly towards the near-UV. Indeed,
measurements of 36 VPHGs for the Dark Energy Spectroscopic Instrument (DESI)
suggest higher processing-induced efficiency variations in the UV–blue
gratings with respect to other bands[4]. Measurements of four VPHG prototypes
for the Visible Integral field Replicable Unit Spectrograph (VIRUS) reveal
increasing efficiency deviations to the theoretical model towards 350 nm,
which are attributed to process-induced absorption or scatter in the DCG
layer[5]. However, Ref. 6 reports both minimal grating-to-grating scatter and
minimal spatial variations towards 350 nm for the full suite of 170 VIRUS
VPHGs.

Laminated self-processing photopolymers seem well-suited to facilitate VPHG
manufacturing, and offer an increasingly broad range of thicknesses and
achievable refractive index modulations[7]. This solution is particularly
interesting for near-UV/visible instruments such as BlueMUSE[8] on the VLT,
which requires a high and consistent transmission across 16 replicated
integral-field units.

In this letter we compare two VPHG prototypes for BlueMUSE based on
dichromated gelatin (DCG) and the Bayfol®HX photopolymer film[9] as recording
materials. We assess manufacturing losses between modeled and measured
diffraction efficiencies and cross-check characterization results provided by
the manufacturers using an identical full aperture test setup. We also compare
the transmitted wavefront error as well as the bidirectional transmittance
distribution function.

##  2 Method

###  2.1 Manufacturing

Two VPHG prototypes are manufactured according to the design parameters listed
in Table 1: the DCG-based prototype is manufactured by Wasatch Photonics and
another prototype based on the Bayfol®HX photopolymer is manufactured by the
Osservatorio Astronomico di Brera (OAB). The resulting gratings are shown in
Figure 1.

(a) (b)

Figure 1: DCG (a) and photopolymer (b) prototypes, back illuminated by a handheld LED light. Table 1: Design parameters. Design parameter | Value
---|---
Wavelength range | 350 - 580 nm
Angle of incidence (AOI, in air) |  $13.72$° $\pm$ $0.7$°
Line density |  $1027\pm 1$ line/mm
Clear aperture |  $110\times 70$ mm ellipse
Diffraction efficiency | $T(350\text{ nm})\gtrsim 60\%$
(excl. Fresnel losses) | $T(580\text{ nm})>40\%$
| Goal: $T_{\text{avg}}>70\%$
Wavefront error (+1 order) |  $<1266$ nm ($2\lambda$) PV
Substrate | Fused Silica (uncoated)
Substrate size |  $130\times 90$ mm

###  2.2 Diffraction efficiency test

Independent diffraction efficiency measurements have been carried out by
Wasatch Photonics, OAB and the Centre de Recherche Astrophysique de Lyon
(CRAL). Unlike the setups at Wasatch Photonics and OAB which rely on scanning
a monochromatic test beam over the aperture and across the wavelength range,
the method at CRAL is based on a full aperture test setup shown in Figure 2
and akin to the one described in Ref 10. This choice was motivated by the
desire to grasp diffraction efficiency variations over the clear aperture.

Briefly, the test beam is generated by a 150 W Xenon light source (Newport
#6256 arc lamp) and filtered by a monochromator (Oriel Cornerstone CS130B) to
a bandwidth of 4.5 nm varied from 330 to 600 nm. The beam is then directed
towards a $f=650$ mm – F/5 Newtonian telescope (Skywatcher Explorer 130PDS),
used as a collimator. We mount the grating on a rotary stage and set it
perpendicular to the collimated beam, using an autocollimation off the
uncoated substrate. We establish this reference angle using a theodolite
aligned with the monochromator slit. The grating is then rotated to the
desired angle of incidence and an identical telescope, used as an objective,
is rotated as well to collect the diffracted light. A CCD camera (ZWO ASI174MM
Mini) is placed after the telescope focus in order to image the grating clear
aperture.

Figure 2: Diffraction efficiency setup at CRAL.

We divide the diffracted image by a reference image, where the grating is
removed and both telescopes are aligned. In order to increase the overall S/N,
we stack 5 exposures per diffracted and reference image. Finally, we smooth
the resulting diffraction efficiency map using a $1\times 1$ mm boxcar filter.
We assess the accuracy of this setup to $\sim 2\%$ using repeated transmission
measurements of an uncoated fused silica window. The setup is probably limited
by straylight and lamp stability between the diffracted and reference images,
which are separated by less than a minute.

##  3 Results

###  3.1 Diffraction efficiency

Internal diffraction efficiency results (i.e., corrected for Fresnel losses)
for unpolarized light are presented in Figure 3. Both prototypes comply with
the requirements presented in Section 2, albeit with different optimizations:
the DCG prototype exhibits a lower average (72% versus 76%) but a better
diffraction efficiency at 350 nm (62% versus 55%). We emphasize that average
diffraction efficiency and diffraction efficiency at 350 nm were given the
same importance in the optimization of those prototypes, considering the
combined impact of atmosphere cutoff and internal glass absorption on the
overall transmission of BlueMUSE. We observe that measurements at CRAL agree
with data from Wasatch Photonics and OAB to within 1% rms.

(a) (b) (c)

Figure 3: Diffraction efficiency results: spatial uniformity of the DCG (a)
and photopolymer (b) prototypes (AOI = 13.72°), and angular selectivity (c).

We note that both prototypes almost reach the performance predicted by
Rigorous Coupled Wave Analysis (RCWA) models[11] at the red end but
increasingly deviate towards the blue, particularly for the DCG prototype. The
overall deviation for an angle of incidence of 13.72° is 23% for DCG and 12%
for Bayfol®HX at 350 nm, compared to 9% and 7% on average, respectively.
Finally, we find 16th-84th percentile variations ($\pm 1\sigma$) of 6% across
the clear aperture for DCG and 10% for Bayfol®HX at 350 nm, compared to 9% on
average over the full wavelength range for both prototypes.

###  3.2 Wavefront error

We measure the transmitted wavefront error in 0th and +1 order using a Fizeau
interferometer (4” Zygo Verifire XPD, $\lambda=632.8$ nm) in a double-pass
configuration, with the prototype set at its working angle and a $\lambda/25$
flat mirror closing the cavity. As the prototype clear aperture is slightly
larger than the 4-inch test beam, we stitch three measurements per diffraction
order. The resulting wavefront error maps are shown in Figure 4. Both the DCG
and photopolymer prototypes are well within the required wavefront error
($<2\lambda$ PV) with 451 nm PV (70 nm rms) and 523 nm PV (78 nm rms),
respectively.

Figure 4: Wavefront error (WFE) maps.

###  3.3 Scattering

Finally, we measure the Bidirectional Transmittance Distribution Function
(BTDF) in the dispersion direction and at the center of the clear aperture
using a complete angle scatter instrument (TSW CASI scatterometer). The probe
beam is a s-polarized HeNe laser with a $\sim$ 1 mm spot diameter. The
measured BTDF is shown in Figure 5. Both prototypes have similar BTDFs within
the limits of the probe beam signature, which spans $\sim$10 BlueMUSE pixels
compared to a slit width of 2 pixels. Therefore, this measurement mainly
probes the outer scatter halo surrounding the point spread function.

Figure 5: Measured BTDFs in the dispersion direction, compared with the probe
beam signature (black line) and models of smooth air-glass interfaces with
varying microroughness (colored lines). Internal reflections at $\sim$0.3° and
$\sim$0.6° are not shown.

##  4 Discussion

###  4.1 Diffraction efficiency uniformity and consistency

Our measurements are consistent with expectations from Ref. 5 which reports
increasing deviations from RCWA models towards the blue, with deviations up to
10-30% ($\sim$20% median) at 350 nm, for DCG-based gratings with a similar
bandpass. According to Ref. 5, these deviations are likely due to processing-
related absorption or scatter, as they vary between production batches. The
photopolymer prototype is at the lower end, with a 12% deviation at 350 nm,
compared to a median value of 23% for the DCG prototype.

Although photopolymers are anticipated to provide more uniform performance
across the clear aperture, our measurements show similar spatial variations
for both the photopolymer and DCG prototypes. Notably, spatial variations
above the median are smaller than those below the median, particularly for the
DCG prototype. This aligns with Ref. 6, which attributes this to the higher
likelihood of encountering a less favorable set of recording material
properties (e.g., thickness or refractive index modulation) compared to the
set optimized by design. Interestingly, the photopolymer prototype shows more
balanced spatial variations around the median, even at peak diffraction
efficiency (i.e., close to an optimal set of recording material properties).
However, it is difficult to draw broad conclusions from a single realization
of each manufacturing process.

To investigate grating-to-grating uniformity, OAB conducted a process
repeatability test, recording and measuring four times a full aperture
photopolymer film before encapsulation. The achieved diffraction efficiency
demonstrated remarkable consistency, with wavelength-averaged variations of
only 2%. For comparison, similar studies such as Ref. 4 and Ref. 6 report
grating-to-grating variations of 7% rms and 3-6% rms respectively, for
comparable but fully assembled DCG-based gratings. This consistency suggests
that most of the spatial variations arise from very repeatable edge effects.
Indeed, we observe that most of the efficiency variation is located at the
edge of the clear aperture as in Ref. 4.

###  4.2 Wavefront error

We note that the wavefront error in 0th order is 3-4 times smaller than in the
+1 order. This suggests that hologram recording errors are the primary
contributors to the overall wavefront error, outweighing errors due to the
substrate, recording material, or bonding, which tend to affect both orders
similarly. In the photopolymer prototype, a significant portion of the
diffracted wavefront error is due to a power term, which may be corrected
during spectrograph alignment. This error likely arises from a slight
collimation error in the recording beam.

###  4.3 Scattering

We approximate the measured straylight levels presented above by an equivalent
surface microroughness, a useful quantity for generating simple yet realistic
straylight models. We compare the prototypes against model BTDFs for smooth
air-glass interfaces ($\Delta n=0.5$) of varying microroughness
($\sigma=2,5,10,25$ nm), following the methodology of Ref 12. We find an
equivalent microroughness between 25 and 10 nm up to 0.10°, between 10 and 5
nm up to 0.25°, and below 5 nm for angles larger than 0.25°. This is a good
straylight performance compared to the different grating technologies with
similar line densities tested in Ref 12, including a DCG-based VPHG which
reaches an equivalent microroughness below 5 nm for angles larger than 1.5°.

##  5 Summary and conclusions

In this letter we have compared two UV-blue VPHG prototypes developped for
BlueMUSE, based on dichromated gelatin (DCG) and the Bayfol®HX photopolymer as
recording materials:

  1. 1.

We have presented a full aperture test setup at CRAL, which provides
measurements in agreement to within 1% rms with respect to data from Wasatch
Photonics and OAB.

  2. 2.

Both prototypes comply with the required diffraction efficiency and achieve
similar performance exceeding the wavelength-averaged goal of 70% over the
BlueMUSE bandpass.

  3. 3.

We observe that both prototypes increasingly depart from their RCWA model
towards 350 nm, which confirms a trend reported in Ref. 5 for VIRUS
prototypes.

  4. 4.

Furthermore, we note that both prototypes have $\sim 10$% spatial efficiency
variations. A repeatability test at OAB shows a remarkably consistent grating-
to-grating performance with wavelength-averaged deviations of only 2%.

  5. 5.

No significant differences in terms of wavefront error ($<1\lambda$ PV) or
scattered light are observed.

Although our measurements are based on a single realization for each
manufacturing process, they offer valuable insights into the expected
performance of the full suite of 16 VPHGs for BlueMUSE. Future statistical
analyses of multiple photopolymer-based VPHGs will provide a more
comprehensive understanding of the advantages and disadvantages of this
recording material compared to DCG.

###### Acknowledgements.

We acknowledge financial support from the Commission spécialisée Astronomie
Astrophysique (CSAA) of CNRS / INSU. We also acknowledge support from the
FRAMA (Fédération de Recherche André-Marie Ampère) as well as the Labex-LIO
(Lyon Institute of Origins) under Grant No. ANR-10-LABX-66 (Agence Nationale
pour la Recherche).

## References

  * [1] Barden, S. C., Arns, J. A., and Colburn, W. S., “Volume-phase holographic gratings and their potential for astronomical applications,” in [Optical Astronomical Instrumentation ], D’Odorico, S., ed., 3355, 866 – 876, International Society for Optics and Photonics, SPIE (1998).
  * [2] Barden, S. C., Arns, J. A., Colburn, W. S., and Williams, J. B., “Volume‐phase holographic gratings and the efficiency of three simple volume‐phase holographic gratings,” Publications of the Astronomical Society of the Pacific 112, 809 (jun 2000).
  * [3] Blanche, P.-A., Gailly, P., Habraken, S. L., Lemaire, P. C., and Jamar, C. A. J., “Volume phase holographic gratings: large size and high diffraction efficiency,” Optical Engineering 43(11), 2603 – 2612 (2004).
  * [4] Ishikawa, Y., Sirk, M. M., Edelstein, J., Jelinsky, P., Brooks, D., Tarle, G., and Collaboration), D., “Comprehensive measurements of the volume-phase holographic gratings for the Dark Energy Spectroscopic Instrument,” The Astrophysical Journal 869, 24 (dec 2018).
  * [5] Chonis, T. S., Hill, G. J., Clemens, J. C., Dunlap, B., and Lee, H., “Methods for evaluating the performance of volume phase holographic gratings for the VIRUS spectrograph array,” in [Ground-based and Airborne Instrumentation for Astronomy IV ], McLean, I. S., Ramsay, S. K., and Takami, H., eds., 8446, 84465H, International Society for Optics and Photonics, SPIE (2012).
  * [6] Chonis, T. S., Frantz, A., Hill, G. J., Clemens, J. C., Lee, H., Tuttle, S. E., Adams, J. J., Marshall, J. L., DePoy, D. L., and Prochaska, T., “Mass production of volume phase holographic gratings for the VIRUS spectrograph array,” in [Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation ], Navarro, R., Cunningham, C. R., and Barto, A. A., eds., 9151, 91511J, International Society for Optics and Photonics, SPIE (2014).
  * [7] Bianco, A., Frangiamore, M., Zanutta, A., Oggioni, L., Pariani, G., Garzon, F., and Insausti, M., “Improvements in VPHGs for astronomy based on photopolymers,” in [Holography: Advances and Modern Trends VIII ], Fimia, A. and Hrabovský, M., eds., 12574, 125740H, International Society for Optics and Photonics, SPIE (2023).
  * [8] Richard, J., Giroud, R., Laurent, F., Krajnović, D., Jeanneau, A., Bacon, R., Abreu, M., Adamo, A., Araujo, R., Bouché, N., Brinchmann, J., Cai, Z., Castro, N., Calcines, A., Chapuis, D., Claeyssens, A., Cortese, L., Daddi, E., Davison, C., Harris, R., Hayes, M., Jauzac, M., Kelz, A., Kneib, J.-P., Goodwin, M., Lanotte, A. A., Lawrence, J., Bouteiller, V. L., Breton, R. L., Lehnert, M., Sanchez, A. L., McGregor, H., McLeod, A. F., Monteiro, M., Morris, S., Opitom, C., Pécontal, A., Robertson, D., Smith, R., Steinmetz, M., Swinbank, M., Urrutia, T., der Sande, J. V., Verhamme, A., Weilbacher, P. M., Wendt, M., Wildi, F., and Zheng, J., “The Blue Multi Unit Spectroscopic Explorer (BlueMUSE) on the VLT: science drivers and overview of instrument design,” in [Ground-based and Airborne Instrumentation for Astronomy X ], Bryant, J. J., Motohara, K., and Vernet, J. R. R., eds., Proc. SPIE 13096 (2024).
  * [9] Bruder, F.-K., Fäcke, T., and Rölle, T., “The chemistry and physics of Bayfol® HX film holographic photopolymer,” Polymers 9(10) (2017).
  * [10] Barkhouser, R. H., Arns, J., and Gunn, J. E., “Volume phase holographic gratings for the Subaru Prime Focus Spectrograph: performance measurements of the prototype grating set,” in [Ground-based and Airborne Instrumentation for Astronomy V ], Ramsay, S. K., McLean, I. S., and Takami, H., eds., 9147, 91475X, International Society for Optics and Photonics, SPIE (2014).
  * [11] Moharam, M. G. and Gaylord, T. K., “Rigorous coupled-wave analysis of planar-grating diffraction,” J. Opt. Soc. Am. 71, 811–818 (Jul 1981).
  * [12] Harnisch, B., Deep, A., Vink, R., and Coatantiec, C., “Grating scattering BRDF and imaging performances: A test survey performed in the frame of the flex mission,” in [International Conference on Space Optics — ICSO 2012 ], Cugny, B., Armandillo, E., and Karafolas, N., eds., 10564, 105642P, International Society for Optics and Photonics, SPIE (2017).