# SALADnet: Self-Attentive multisource Localization
in the Ambisonics Domain

###### Abstract

In this work, we propose a novel self-attention based neural network for
robust multi-speaker localization from Ambisonics recordings. Starting from a
state-of-the-art convolutional recurrent neural network, we investigate the
benefit of replacing the recurrent layers by self-attention encoders,
inherited from the Transformer architecture. We evaluate these models on
synthetic and real-world data, with up to 3 simultaneous speakers. The
obtained results indicate that the majority of the proposed architectures
either perform on par, or outperform the CRNN baseline, especially in the
multisource scenario. Moreover, by avoiding the recurrent layers, the proposed
models lend themselves to parallel computing, which is shown to produce
considerable savings in execution time.

Index Terms—  Sound source localization, neural networks, self-attention,
Ambisonics, parallel computing.

##  1 Introduction

Sound source localization (SSL) is an active research field with various
applications such as source separation [1], speech recognition [2] or human-
robot interaction [3]. Traditional methods to address SSL are based on
eigenvalue decomposition of a multichannel signal covariance matrix [4], time-
difference of arrival (TDOA) estimation [5], or beamforming [6]. These
methods, and their variants, enjoy widespread popularity, but are known to
lack robustness in noisy and reverberant environments.

More recently, machine learning methods have greatly improved the performance
of SSL systems. In particular, deep neural networks have been proposed to
improve single-source [7], as well as multi-source DOA estimation [8, 9]. They
were shown to improve the robustness of SSL in challenging conditions compared
to traditional methods. Among deep learning models, different architectures
have been proposed: convolutional neural networks (CNNs) [10], convolutional
recurrent neural networks (CRNNs) [8, 9], U-net architectures [1],
autoencoders (AEs) [11] or attention-based neural networks [12]. Also, various
types of input features have been used, such as raw signal waveforms [10],
features based on the short-time Fourier transform (STFT) [13], correlation-
based features [7, 11], Ambisonics features [9, 8], or combinations of
different input features [14, 15]. These neural network-based systems function
either in regression [7, 10, 12] or in classification mode [9, 8, 11]. On the
one hand, regression-based SSL provides DOA estimates with any arbitrary
resolution, at the expense of dealing with the well-known source permutation
problem in the case of multiple sources localization [16]. On the other hand,
classification-based SSL directly provides (multiple) DOA estimates on a
discrete spatial grid.

When considering multiple sound source localization for Ambisonic signals, the
CRNN-based neural networks perform particularly well, as reported in _e.g._
[8, 9, 17, 18]. The major downside of such models is their sequential nature,
_i.e._ their recurrent layers cannot be efficiently parallelized. However,
these recurrent layers are important for modeling the temporal dynamics,
therefore, it is suboptimal to have them simply discarded or replaced by
convolutional layers, as it it leads to significant drop in localization
accuracy [17, 9]. Beyond computational concerns, another potential issue is
that the recurrent layers are essentially based on the first order Markov
model, meaning that their output depends only on the hidden state and the
current input, hence they do not directly incorporate information from the
other elements of a sequence.

Recently, the _attention mechanism_ has emerged as a promising alternative to
model temporal dependencies. Originally proposed in [19] in conjunction with
recurrent neural networks (RNNs) for natural language processing (NLP),
attention efficiently learns the interdependencies of elements (_e.g._
vectors) between two sequences. It has been lately applied to the localization
problem in [12], in the context of sound event detection and localization
(SELD). _Self-attention_ is a particular variant of attention that analyses a
single sequence, _i.e._ it models the similarity of its elements. In [20], the
authors proposed to couple a CRNN with self-attention mechanism, also in order
to improve the SELD baseline.

In this work, we are interested in adapting the CRNNs used for multiple
speaker localization (thus, different from SELD). We propose to replace the
recurrent layers by the encoder modules of the so-called _Transformer_
architecture [21]. Transformer uses standalone self-attention blocks, i.e. it
does not include recurrent layers at all. Originally applied to NLP tasks, it
has since become very popular, as it can actually surpass the performance of
classical RNNs. We thus propose to replace the bidirectional long short-term
memory (BiLSTM) layers of a state-of-the-art CRNN [18] by one or several self-
attention encoders, depending on the model. As demonstrated on the simulated
and real data, this modification actually improves the overall accuracy of the
considered CRNN, while avoiding the recurrent layers and thus being better
suited for parallel computing. We term these new architectures _Self-Attentive
multisource Localization in the Ambisonics Domain (SALAD) nets_.

##  2 Proposed Method

In this section, we present the proposed SSL system, including input features,
output format and network architecture.

###  2.1 Input features

We use the same input features as in [9, 18], namely the intensity vector from
the first-order Ambisonics (FOA) representation of the audio signal. The
Ambisonics format has proven to be well-suited for SSL [9, 8] as it provides a
convenient way of representing the spatial properties of the soundfield. The
FOA consists of the omnidirectional channel $W(t,f)$, and three figure-of-
eight channels, aligned with Cartesian coordinate axes, denoted as $X(t,f)$,
$Y(t,f)$ and $Z(t,f)$ ($t$ and $f$ denote the STFT time and frequency bins,
respectively).
footnoteWe adopt the N3D Ambisonics normalization standard.

We can then derive the expressions for the active and reactive intensity
vectors in the FOA representation [jarrett2017theory] (indexes $t$ and $f$ are
omitted for concision, and ∗ denotes the complex conjugate):

$\boldsymbol{I}_{a}=\begin{bmatrix}\operatorname{Re}\\{WX^{*}\\}\\\
\operatorname{Re}\\{WY^{*}\\}\\\
\operatorname{Re}\\{WZ^{*}\\}\end{bmatrix},\quad\boldsymbol{I}_{r}=\begin{bmatrix}\operatorname{Im}\\{WX^{*}\\}\\\
\operatorname{Im}\\{WY^{*}\\}\\\ \operatorname{Im}\\{WZ^{*}\\}\end{bmatrix}.$
(1)

For each time-frequency (TF) bin, the above active and reactive intensity
vectors are concatenated to form a 6-channel vector which is normalized by the
sound power given by
$|W(t,f)|^{2}+\frac{1}{3}(|X(t,f)|^{2}+|Y(t,f)|^{2}+|Z(t,f)|^{2})$. The
resulting vector is reminiscent of the so-called Frequency Domain Velocity
Vector representation [22]. Finally, the input features are given as the
collection of these vectors for all time-frequency bins, assembled into a
tensor of size $N\times F\times 6$, with $N$ the number of frames and $F$ the
number of frequency bins. In our experiments, the signals are sampled at $16$
kHz and we use a 1,024-point ($64$ ms) STFT (i.e. $F=513$) with sinusoidal
analysis window and $50$ % overlap. The input sequences contains $25$ frames
(representing $800$ ms of signal), hence the shape input feature is $25\times
513\times 6$.

###  2.2 Output

As in [18], we treat SSL as a classification problem. The 2D sphere is divided
into a quasi-uniform grid of candidate DOA regions with elevation
$\phi_{i}\in[-90,90]$ and azimuth $\theta^{j}_{i}\in[-180,180]$ given by:

$\begin{cases}\phi_{i}=-90+\frac{i}{I}\times 180&\text{with
$i\in\\{0,...,I\\}$}\\\ \theta_{i}^{j}=-180+\frac{j}{J^{i}+1}\times
360&\text{with $j\in\\{0,...,J^{i}\\}$},\\\ \end{cases}$ (2)

where $I=\lfloor\frac{180}{\alpha}\rfloor$ and
$J^{i}=\lfloor\frac{360}{\alpha}cos\phi_{i}\rfloor$ with $\alpha$ the grid
resolution in degrees. Using classification-based localization, each region
correspond to a class and the output vector $y$ is of size $C$ (which is the
total number of classes). For example for the target, $y(c)=1$ if a source is
present in a zone corresponding to class $c$, and $y(c)=0$ otherwise. We set
$\alpha=10$°, resulting in a total of $C=429$ classes.

###  2.3 Network architecture

Figure 1: Proposed architecture based on convolutional layers and self-
attention encoders. Left: overall architecture, with $L$ stacked self-
attention encoders; Right: Detail of a self-attention encoder.

As stated above, the proposed architecture is based on the architecture called
Model 6-4 in [18], which is composed of a convolutional module followed by a
recurrent module and ending with feed-forward layers for classification.
Taking into account the limitations of the recurrent layers mentioned in the
introduction, and the empirical observation that certain frames are more
informative than others with regards to the SSL task [9], we propose to
replace the recurrent part of the model in [18] with a self-attention-based
encoder module. While recurrent layers only perform the analysis of temporal
sequences in a sequential way, the use of self-attention allows for comparison
of all111In the present study, we do not use positional encoding _c.f._ [21].
frames with each other. Fig. 1 shows the diagram of the proposed neural
network architecture, which is detailed in the remainder of this section.

Convolutional module: The $N\times F\times 6$ input features are first
provided to a convolutional module identical to the one of Model 6-4 in [18],
which was chosen due to its good trade-off between localization accuracy and
model size. It is composed of 6 convolutional blocks, each block being made of
two consecutive convolutional layers with 64 filters of size $3\times 3$,
followed by a max-pooling layer with various size on the second dimension.
This module performs feature extraction by reducing the overall feature size
while keeping the temporal dimension ($N=25$) of the input intact. After the
convolutional module, the feature maps are concatenated together to form a 2D
tensor of size $N\times G$ (reshaping module; here $G=128$).

Self-attention module: The sequence of reshaped features goes into a stack of
$L$ self-attention encoders. Each self-attention encoder follows the structure
shown in the right side of Fig. 1, which is the encoder of the Transformer
architecture [21]. It is composed first of a multi-head self-attention layer
(classical multi-head or cross multi-head according to the experiment,
detailed below) which produces an encoded feature for each feature of the
input $N$-sequence. Each encoded feature is then added to the corresponding
input feature with a residual connection, and the sum is normalized. After
that, each sequence item goes through the same two-layer feed-forward module,
the first layer being linear and the second layer having a ReLU activation.
Then again, a residual connection and normalization are applied. The output of
such a self-attention encoder has the same dimension as its input (here
$N\times G$).

Self-attention has been originally proposed for sequential data, and works as
following (see [21] for more details). For each vector $\mathbf{x}_{i}$ of the
input sequence of length $N$, three vectors $\mathbf{q}_{i}$, $\mathbf{k}_{i}$
and $\mathbf{v}_{i}$ (usually labelled as query, key and value, respectively)
are computed by multiplying $\mathbf{x}_{i}$ with learnt matrices
$\mathbf{W}^{Q}$, $\mathbf{W}^{K}$ and $\mathbf{W}^{V}$. Then, for each vector
$\mathbf{x}_{i}$, an encoded vector $\mathbf{z}_{i}$ is calculated via
weighted sum of all $\\{\mathbf{v}_{j}\\}$:
$\mathbf{z}_{i}=\sum_{j=1}^{N}s_{ij}\mathbf{v}_{j}$, where $s_{ij}$ is the
score of $\mathbf{x}_{i}$ against $\mathbf{x}_{j}$. The score $s_{ij}$ is
calculated as the softmax of a scaled dot product of $\mathbf{q}_{i}$ and
$\mathbf{k}_{j}$, so that the sum of $s_{ij}$ over $j$ is equal to $1$:

$s_{ij}=\text{softmax}\Big{(}\frac{\mathbf{q}_{i}\cdot\mathbf{k}_{j}}{\sqrt{G}}\Big{)}$
(3)

In summary, self-attention applied on the temporal axis computes at each time
index $i$ an output vector $\mathbf{z}_{i}$ that takes into account the input
vector $\mathbf{x}_{i}$ at time $i$ and its dependencies (the scores) with the
other (past and future) vectors $\mathbf{x}_{j}$ in the sequence. Note that
$\mathbf{x}_{i}$ and $\mathbf{z}_{i}$ are vectors of the same size.

In the Transformer paper [21], the authors also introduce the concept of
multi-head self-attention, which consists of learning $H$ independent sets of
matrices $\mathbf{W}^{Q}_{h}$, $\mathbf{W}^{K}_{h}$ and $\mathbf{W}^{V}_{h}$,
with $h\in[1,H]$ and $H$ the number of heads, thus leading to $H$ vectors
$\mathbf{q}_{ih}$, $\mathbf{k}_{ih}$ and $\mathbf{v}_{ih}$ for each input
$\mathbf{x}_{i}$. The score are computed independently for each head $h$ :

$s_{ijh}=\text{softmax}\Big{(}\frac{\mathbf{q}_{ih}\cdot\mathbf{k}_{jh}}{\sqrt{G}}\Big{)}$
(4)

In this case, a multi-head version of $\mathbf{z}_{i}$ is calculated as a
weighted sum of all $\mathbf{v}_{jh}$ :
$\mathbf{z}_{ih}=\sum_{j=1}^{N}s_{ijh}\mathbf{v}_{jh}$. The $\mathbf{z}_{ih}$
are finally concatenated in the head dimension and another learnt matrix
$\mathbf{W}^{O}$ is used to output one vector $\mathbf{z}_{i}$ for each input
vector $\mathbf{x}_{i}$ as in single-head self-attention. This multi-head
aspect provides more flexibility in the self-attention mechanism.

We also evaluate the use of a more general way of computing the multi-head
attention scores, where a score is calculated independently for each head pair
$(h,h^{\prime})$ (while in [21] the scores are calculated head-wise) :

$s_{ijhh^{\prime}}=\text{softmax}\Big{(}\frac{\mathbf{q}_{ih}\cdot\mathbf{k}_{jh^{\prime}}}{\sqrt{G}}\Big{)}$
(5)

Then $\mathbf{z}_{ih}$ is obtained as a weighted sum on $j$ and $h^{\prime}$ :
$\mathbf{z}_{ih}=\sum_{j=1}^{N}\sum_{h^{\prime}=1}^{H}s_{ijhh^{\prime}}\mathbf{v}_{jh^{\prime}}$.
We call this method cross multi-head (CMH) self-attention to distinguish it
from original multi-head self-attention [21]. In our experiments , $H$ is a
hyperparameter.

Feed-forward fully-connected module: At the end, each vector of the output
sequence of the self-attention module is sent to the same two-layer feed-
forward network, with $429$ units in each layer. The first layer is linear,
and the second one has the sigmoid activation, to provide output values
homogeneous to probability of having a source present in the corresponding DOA
region (see Section 2.2).

##  3 Experiments

###  3.1 Data

To generate data for training and testing, we used the same methodology as in
[18]. Beforehand, we adapted the spatial room impulse response (SRIR)
generator of [23] (based on the image-source method [24]) to synthesize FOA
impulse responses for many different room configurations: room lengths, widths
and heights randomly drawn in $[2,10]$ m, $[2,10]$ m and $[2,3]$ m,
respectively; RT60s lie in $[200,800]$ ms; microphone and source positions are
randomly picked in the room with the microphone at least $0.5$ m from the
walls. These synthetic SRIRs are used for the training dataset as well as for
the first test dataset. For a second more realistic test dataset, we also
recorded SRIRs in our acoustic lab (RT60 $\approx$ 500 ms), using all
combinations from 36 microphone positions and 16 static loudspeaker positions.
We generated microphone signals by convolving the SRIRs (either real or
synthetic) with TIMIT speech signals [25]. For each room configuration, we
created signals corresponding to a single source, 2- and 3- source mixture.
The signal-to-interference ratio between speakers is randomly drawn between 0
and 10 dB. Some diffuse babble noise, obtained by averaging the late
reverberation part of two random synthetic RIRs, was added to those mixtures
with a random SNR between 0 and 20 dB. The resulting training dataset was
composed of 247,400 sequences for each number of speakers (1 to 3), i.e. a
total of 772,200 training sequences (about 172 hours of signals). Each test
set (synthetic and real data) contains 1152 sequences for each number of
speakers, corresponding to about 45 minutes of signals.

###  3.2 Training procedure & parameters

We designed and trained the proposed model using Keras framework on Nvidia GTX
1080 GPUs. We use the Nadam optimizer [26] with default parameters. Early
stopping was applied with a patience of 20 epochs by monitoring the accuracy
on a validation set, and the learning rate was divided by two when the
validation accuracy was not improving after 10 epochs.

###  3.3 Configurations

We trained and tested several variants of the proposed model to evaluate the
capacity of the self-attention module to replace the recurrent layers for SSL.
In the presented results, we use the naming convention
“$X$-$N_{\mathrm{enc}}$enc-$H$H”, where $X$ is MH or CMH whether multi-head or
cross multi-head is used, respectively, $N_{\text{enc}}$ denotes the number of
encoders, and $H$ is the number of attention heads. First, we tested
$H\in\\{1,2,3,10\\}$ using classical multi-head attention. Then, we evaluated
the benefit of using cross multi-head self-attention for $H\in\\{3,10\\}$.
Finally, we investigated the use of two cross multi-head self-attention
encoders instead of 1, with $H\in\\{5,10\\}$.

###  3.4 Metrics, evaluation and baseline

During the inference, the output of the neural network is averaged over the
frames, meaning that we end up with one probability distribution for the whole
input sequence. Knowing the number of sources $S$ in the test signal, we
extract the $S$ highest peaks in the probability distribution, in which a peak
represents a local maximum within the spherical geometry. The DOA estimation
task is evaluated by computing the sequence-wise accuracy, i.e. the percentage
of sequences with a predicted DOA angular error lower than a given tolerance
(the higher the better). Since the minimal angular separation between two
points in our grid is 7°, we set tolerance threshold to 10° or 15°. We also
evaluated the performance with the mean and median angular error, averaged on
all test sequences (the lower the better), thus we indicate the standard
deviation as well. As the proposed architectures are based on the CRNN model
called Model 6-4 in [18], we use this model as a common baseline, referred as
Baseline (CRNN). A DNN-free baseline is also referred as Baseline (TRAMP):
this method is based on the histogram - over all frames and frequencies of the
sequence - of the DOAs derived from the pseudointensity vector; each DOA is
weighted by an ad-hoc plane-wave indicator (see [27] for more details).

###  3.5 Results

Model | 1 source | 2 sources | 3 sources
---|---|---|---
Acc. <10° | Acc. <15° | Mean | Med. | Std. | Acc. <10° | Acc. <15° | Mean | Med. | Std. | Acc. <10° | Acc. <15° | Mean | Med. | Std.
Baseline (TRAMP) | 43.7 | 64.2 | 16.7 | 11.5 | 20.3 | 27.9 | 39.9 | 44.0 | 21.9 | 44.6 | 22.4 | 33.6 | 48.4 | 30.0 | 44.9
Baseline (CRNN) | 98.6 | 99.7 | 4.4 | 4.1 | 4.6 | 88.3 | 93.3 | 7.7 | 4.7 | 19.4 | 74.7 | 83.4 | 12.8 | 5.9 | 29.7
MH-1enc-1H | 98.3 | 99.7 | 4.5 | 4.2 | 4.5 | 87.9 | 93.6 | 7.9 | 4.9 | 13.8 | 71.6 | 81.5 | 13.5 | 6.5 | 21.1
MH-1enc-2H | 98.1 | 99.7 | 4.7 | 4.2 | 5.2 | 87.1 | 93.0 | 8.7 | 4.9 | 15.3 | 72.2 | 80.6 | 15.8 | 6.2 | 23.9
MH-1enc-3H | 98.1 | 99.6 | 4.7 | 4.2 | 5.5 | 87.7 | 92.7 | 8.8 | 4.8 | 16.0 | 72.9 | 80.7 | 16.2 | 6.0 | 25.1
MH-1enc-10H | 98.5 | 99.6 | 4.5 | 4.1 | 5.5 | 90.4 | 94.5 | 7.4 | 4.7 | 15.4 | 77.3 | 84.7 | 12.3 | 5.6 | 24.2
CMH-1enc-3H | 98.5 | 99.8 | 4.4 | 4.1 | 5.1 | 89.3 | 94.1 | 7.6 | 4.8 | 15.1 | 75.9 | 84.1 | 12.7 | 5.9 | 23.6
CMH-1enc-10H | 98.4 | 99.5 | 4.5 | 4.1 | 5.1 | 89.9 | 94.5 | 6.8 | 4.7 | 14.6 | 78.2 | 85.7 | 11.3 | 5.6 | 23.0
CMH-2enc-5H | 98.5 | 99.6 | 4.6 | 4.2 | 4.9 | 90.4 | 94.7 | 7.0 | 4.7 | 14.3 | 75.6 | 84.2 | 12.4 | 6.0 | 22.8
CMH-2enc-10H | 97.9 | 99.3 | 4.8 | 4.2 | 5.0 | 88.7 | 94.9 | 7.3 | 5.0 | 14.1 | 72.8 | 83.7 | 13.0 | 6.5 | 22.5

Table 1: SSL results on the test dataset generated with synthetic SRIRs (best
results are in bold).

Model | 1 source | 2 sources | 3 sources
---|---|---|---
Acc. <10° | Acc. <15° | Mean | Med. | Std. | Acc. <10° | Acc. <15° | Mean | Med. | Std. | Acc. <10° | Acc. <15° | Mean | Med. | Std.
Baseline (TRAMP) | 52.9 | 72.8 | 14.5 | 9.4 | 19.2 | 30.5 | 42.9 | 37.0 | 19.1 | 38.6 | 25.1 | 35.6 | 43.0 | 26.0 | 41.1
Baseline (CRNN) | 79.0 | 93.7 | 7.6 | 6.1 | 10.3 | 68.2 | 84.7 | 11.9 | 7.2 | 24.8 | 56.8 | 73.3 | 17.3 | 8.7 | 32.9
MH-1enc-1H | 77.0 | 93.5 | 7.5 | 6.2 | 6.5 | 67.4 | 83.5 | 11.5 | 7.2 | 15.8 | 53.8 | 69.9 | 18.9 | 9.1 | 26.1
MH-1enc-2H | 76.8 | 93.4 | 7.6 | 6.3 | 6.8 | 67.9 | 83.8 | 12.7 | 7.5 | 18.3 | 53.6 | 68.5 | 22.5 | 9.1 | 29.2
MH-1enc-3H | 76.2 | 92.7 | 8.1 | 6.3 | 8.1 | 67.4 | 84.9 | 12.3 | 7.3 | 18.4 | 54.6 | 68.3 | 21.8 | 9.1 | 29.4
MH-1enc-10H | 77.3 | 93.0 | 8.3 | 6.2 | 9.0 | 68.2 | 86.3 | 11.2 | 7.3 | 18.0 | 57.8 | 74.0 | 16.9 | 8.5 | 28.2
CMH-1enc-3H | 77.5 | 92.6 | 8.0 | 6.3 | 9.1 | 68.6 | 85.6 | 10.7 | 7.3 | 17.3 | 56.4 | 72.3 | 18.2 | 8.9 | 27.8
CMH-1enc-10H | 77.0 | 92.6 | 8.0 | 6.2 | 9.1 | 68.6 | 85.8 | 10.5 | 7.3 | 16.7 | 58.2 | 74.8 | 15.2 | 8.5 | 26.8
CMH-2enc-5H | 75.7 | 92.6 | 8.4 | 6.3 | 9.4 | 70.0 | 87.1 | 10.4 | 7.2 | 16.4 | 57.7 | 74.2 | 16.3 | 8.6 | 26.3
CMH-2enc-10H | 75.7 | 91.1 | 8.8 | 6.2 | 9.9 | 69.0 | 86.9 | 10.6 | 7.2 | 16.2 | 56.3 | 73.3 | 17.1 | 8.9 | 26.0

Table 2: SSL results on the test dataset generated with real SRIRs (best results are in bold). Model | real-time % | # parameters
---|---|---
Baseline (CRNN) | 437 | 913,907
MH-1enc-1H | 244 | 796,125
MH-1enc-2H | 244 | 862,045
MH-1enc-3H | 244 | 927,965
MH-1enc-10H | 244 | 1,389,405
CMH-1enc-3H | 244 | 927,965
CMH-1enc-10H | 244 | 1,389,405
CMH-2enc-5H | 281 | 1,653,341
CMH-2enc-10H | 281 | 2,312,541
Table 3: Real-time percentage for inference and number of parameters for the
different models (in our experiments, frame length = 0.032 s).

The results obtained on the synthetic and real SRIRs test datasets are shown
in Table 1 and Table 2, respectively. First, we can see that all neural-based
systems largely surpass the DNN-free baseline from [27]. Second, we can see
that the performance of all the tested self-attention-based neural networks
are roughly similar to the performance of the baseline CRNN, sometimes a bit
below and sometimes better, depending on the configuration (we detail below).
This shows that it is possible to replace the recurrent layers with self-
attention encoders without losing in performance. Importantly, we can see in
Table 3 that the gain in inference time over the baseline is quite
significant. The median inference time using self-attention-based networks is
44% lower than for the baseline (when using one self-attention encoder; it is
33% lower when using 2 encoders). We also see that this inference time is not
correlated with the number of parameters, but rather with the number of self-
attention encoders, which is to be expected.

Generally, the MH attention-based networks with 1, 2 or 3 heads perform
slightly below the baseline. However, the MH self-attention network with 10
attention heads outperforms the baseline in the multi-source configurations:
for example, on 3-source mixtures with synthetic SRIRs, the accuracy (< 10°)
is 78.2 % vs 74.7 % for the baseline, and the mean angular error is lower by
1.5°. The same trend can be observed for the real SRIRs configuration, though
with a more moderate performance gain. This suggests that adding more heads to
the self-attention mechanism may be particularly beneficial for multi-source
localization.

The results for the models with CMH self-attention are also globally better
than the baseline (in the multi-source configurations), and also better than
MH self-attention when comparison is made with the same number of heads. CMH
self-attention leads to an improvement for both 3 and 10 heads over MH self-
attention, especially in terms of mean and median angular error. For example,
going from MH to CMH self-attention with 10 heads reduce the mean error by
0.6° and 1° for 2-source and 3-source mixtures (synthetic SRIRs),
respectively, and by 0.7° and 1.7° for 2-source and 3-source mixtures (real
SRIRs). The accuracy is also improved, up to 78.2% (< 10°) for synthetic SRIRs
and 58.2% for real SRIRs, representing a performance gain of 3.5% and 1.4%
over the baseline, respectively. Allowing the score calculation across all
heads seems to provide more flexibility to the self-attention mechanism thus
leading to more robust multi-source localization.

Finally, the results of Models CMH-2enc-5H and CMH-2enc-10H shows what happens
when we stack two CMH self-attention encoders (for the same number of heads).
We can see that the performance of model CMH-2enc-5H model is better than the
one of model CMH-1enc-10H for 2-source mixtures (for both synthetic or real
SRIRs), but lose performance on 3-source signals. The mean error increases by
1.1°. When dealing with 2 encoders with 10 attention heads each, the
performance gets even worse, somehow going back near to the baseline
performance. The shows that adding more encoders does not necessarily improve
the performance, possibly because the complexity of the network becomes too
high for this task or for the amount of training data available.

##  4 Conclusion

We have presented a novel multi-head self-attentive neural network for SSL of
up to 3 speakers. We first showed that self-attention encoders are suitable
for replacing recurrent layers of a state-of-the-art CRNN without losing in
localization performance, while saving computations and inference time. We
also compared the performance for different configurations of the self-
attention module. Overall, the best performance was obtained with a cross
multi-head self-attention model with one 10-head encoder, which clearly
outperformes the baseline state-of-the-art CRNN in the multi-source
configuration (again for a lower computation cost). Future work will further
investigate the performance of these models, e.g. why the increase of the
number of stacked encoders does not improve the performance in our
experiments.

## References

  * [1] S. E. Chazan, H. Hammer, G. Hazan, J. Goldberger, and S. Gannot, “Multi-Microphone Speaker Separation based on Deep DOA Estimation,” in _2019 27th European Signal Processing Conference (EUSIPCO)_ , Sept. 2019.
  * [2] H. Lee, J. Cho, M. Kim, and H. Park, “DNN-Based Feature Enhancement Using DOA-Constrained ICA for Robust Speech Recognition,” _IEEE Signal Processing Letters_ , vol. 23, no. 8, pp. 1091–1095, Aug. 2016\.
  * [3] X. Li, L. Girin, F. Badeig, and R. Horaud, “Reverberant Sound Localization with a Robot Head Based on Direct-Path Relative Transfer Function,” _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 2819–2826, Oct. 2016.
  * [4] R. Schmidt, “Multiple emitter location and signal parameter estimation,” _IEEE Transactions on Antennas and Propagation_ , vol. 34, no. 3, pp. 276–280, Mar. 1986.
  * [5] C. Knapp and G. Carter, “The generalized correlation method for estimation of time delay,” _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 24, no. 4, pp. 320–327, Aug. 1976.
  * [6] J. P. Dmochowski, J. Benesty, and S. Affes, “A Generalized Steered Response Power Method for Computationally Viable Source Localization,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 15, no. 8, pp. 2510–2526, Nov. 2007.
  * [7] F. Vesperini, P. Vecchiotti, E. Principi, S. Squartini, and F. Piazza, “A neural network based algorithm for speaker localization in a multi-room environment,” in _IEEE International Workshop on Machine Learning for Signal Processing_ , 2016, pp. 1–6.
  * [8] S. Adavanne, A. Politis, and T. Virtanen, “Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network,” _arXiv:1710.10059_ , Sept. 2018.
  * [9] L. Perotin, R. Serizel, E. Vincent, and A. Guérin, “CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 13, no. 1, pp. 22–33, Mar. 2019.
  * [10] J. M. Vera-Diaz, D. Pizarro, and J. Macias-Guarasa, “Towards End-to-End Acoustic Localization using Deep Learning: from Audio Signal to Source Position Coordinates,” _Sensors_ , vol. 18, no. 10, p. 3418, 2018.
  * [11] Z.-M. Liu, C. Zhang, and P. S. Yu, “Direction-of-Arrival Estimation Based on Deep Neural Networks With Robustness to Array Imperfections,” _IEEE Transactions on Antennas and Propagation_ , vol. 66, no. 12, pp. 7315–7327, Dec. 2018.
  * [12] C. Schymura, T. Ochiai, M. Delcroix, K. Kinoshita, T. Nakatani, S. Araki, and D. Kolossa, “Exploiting Attention-based Sequence-to-Sequence Architectures for Sound Event Localization,” in _2020 28th European Signal Processing Conference (EUSIPCO)_ , 2021.
  * [13] N. Yalta, K. Nakadai, and T. Ogata, “Sound Source Localization Using Deep Learning Models,” _Journal of Robotics and Mechatronics_ , vol. 29, no. 1, pp. 37–48, 2017.
  * [14] W. Xue, Y. Tong, C. Zhang, G. Ding, X. He, and B. Zhou, “Sound Event Localization and Detection Based on Multiple DOA Beamforming and Multi-Task Learning,” in _Interspeech 2020_ , Oct. 2020.
  * [15] Y. Cao, T. Iqbal, Q. Kong, M. B. Galindo, W. Wang, and M. D. Plumbley, “Two-Stage Sound Event Localization And Detection Using Intensity Vector And Generalized Cross-Correlation,” Tech. Rep., 2019\.
  * [16] A. S. Subramanian, C. Weng, S. Watanabe, M. Yu, and D. Yu, “Deep Learning based Multi-Source Localization with Source Splitting and its Effectiveness in Multi-Talker Speech Recognition,” _arXiv:2102.07955_ , Feb. 2021.
  * [17] D. Krause, A. Politis, and K. Kowalczyk, “Comparison of Convolution Types in CNN-based Feature Extraction for Sound Source Localization,” in _2020 28th European Signal Processing Conference (EUSIPCO)_ , Jan. 2021, pp. 820–824.
  * [18] P.-A. Grumiaux, S. Kitic, L. Girin, and A. Guérin, “Improved feature extraction for CRNN-based multiple sound source localization,” in _European Signal Processing Conference_ , 2021\.
  * [19] D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” _arXiv:1409.0473_ , May 2016.
  * [20] H. Phan, L. Pham, P. Koch, N. Q. K. Duong, I. McLoughlin, and A. Mertins, “Audio Event Detection and Localization with Multitask Regression Network,” Tech. Rep., 2020.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” _arXiv:1706.03762_ , Dec. 2017.
  * [22] J. Daniel and S. Kitic, “Time Domain Velocity Vector for Retracing the Multipath Propagation,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2020, pp. 421–425.
  * [23] E. A. P. Habets, “Room impulse response generator,” Technische Universiteit Eindhoven, Tech. Rep., 2006.
  * [24] J. B. Allen and D. A. Berkley, “Image method for efficiently simulating small-room acoustics,” _The Journal of the Acoustical Society of America_ , vol. 65, no. 4, pp. 943–950, 1979.
  * [25] J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, N. L. Dahlgren, and V. Zue, “TIMIT Acoustic-Phonetic Continuous Speech Corpus,” Tech. Rep., 1993.
  * [26] T. Dozat, “Incorporating Nesterov Momentum into Adam,” _ICLR 2016_ , 2016.
  * [27] S. Kitic and A. Guérin, “TRAMP: TRacking by a realtime AMbisonic-based Particle filter,” _LOCATA Challenge Workshop_ , 2018.


††thanks: Current affiliation: National Research Council of Canada, 100 Sussex
Drive, K1N 5A2, Ottawa, Ontario, Canada

# Spatially twisted liquid-crystal devices

Alicia Sit  Nexus for Quantum Technologies, University of Ottawa, K1N 5N6,
Ottawa, ON, Canada  Francesco Di Colandrea  Nexus for Quantum Technologies,
University of Ottawa, K1N 5N6, Ottawa, ON, Canada  Alessio D’Errico  Nexus for
Quantum Technologies, University of Ottawa, K1N 5N6, Ottawa, ON, Canada
Ebrahim Karimi<EMAIL_ADDRESS>Nexus for Quantum Technologies, University
of Ottawa, K1N 5N6, Ottawa, ON, Canada

###### Abstract

Nematic liquid-crystal devices are a powerful tool to structure light in
different degrees of freedom, both in classical and quantum regimes. Most of
these devices exploit either the possibility of introducing a position-
dependent phase retardation with a homogeneous alignment of the optic
axis—e.g., liquid-crystal-based spatial light modulators—or conversely, with a
uniform but tunable retardation and patterned optic axis, e.g., $q$-plates.
The pattern is the same in the latter case on the two alignment layers. Here,
a more general case is considered, wherein the front and back alignment layers
are patterned differently. This creates a non-symmetric device which can
exhibit different behaviours depending on the direction of beam propagation
and effective phase retardation. In particular, we fabricate multi-$q$-plates
by setting different topological charges on the two alignment layers. The
devices have been characterized by spatially resolved Stokes polarimetry, with
and without applied electric voltage, demonstrating new functionalities.

## Introduction

When nematic liquid crystals are placed between parallel glass plates with
differing alignment directions, the bulk will twist in order to match the
boundary conditions De Gennes and Prost (1993). This phenomenon of a twisted
nematic liquid crystal (TNLC) cell—in particular with $90^{\circ}$ twists—has
been used extensively for the development of everyday liquid-crystal displays
Shanks (1982); Schadt (1988); Yeh and Gu (2009). With properly chosen
birefringent liquid crystals and fabrication techniques, incident linearly
polarized light will rotate through the cell, following the twist structure.
When a sufficiently strong voltage is applied across the cell, the twist
structure disappears as the liquid crystals are aligned in the field
direction, negating any polarization rotation. However, there has been limited
study of the twisted cell beyond the 90∘ twist case for general polarization
manipulation Gooch and Tarry (1975); Ong (1987).

On the other hand, in the context of experimental optics, spatially patterned
liquid-crystal-based devices are an efficient and compact method for
structuring the polarization and spatial degrees of freedom of light, but
studies have been limited to symmetric elements, i.e., the front and back
patterns on the alignment layers are identical. For example, $q$-plates—part
of the general class of Pancharatnam–Berry phase optical elements (PBOE)—are
such that the liquid-crystal layer is aligned to have a semi-integer
topological charge of $q$ – note that such a topological structure, similar to
$q$-plates, may be formed by liquid crystal droplets Brasselet  _et al._
(2009). This allows for the coupling of photonic spin to orbital angular
momentum. $q$-plates have found applications in both classical and quantum
optics Marrucci  _et al._ (2011); Rubano  _et al._ (2019), in particular STED
microscopy Yan  _et al._ (2015), metrology D’Ambrosio  _et al._ (2013), high-
dimensional classical Karimi  _et al._ (2012) and quantum communication Sit
_et al._ (2017), and quantum simulations Cardano  _et al._ (2015). For the
case of non-symmetric spatially patterned devices, there have been only a few
implementations, including polarization converters which convert linear
polarization into vector vortex modes Stalder and Schadt (1996), a
functionality still achievable via standard $q$-plates Cardano  _et al._
(2012). However, these spatially twisted elements operate with no externally
applied field. Only recently, the voltage-dependent behaviour of non-symmetric
devices patterned with different gratings has been observed Nys  _et al._
(2022).

This article aims to bridge the above gaps by investigating the behaviour of
liquid crystals with the full range of possible twist angles from
$-90^{\circ}$ to $90^{\circ}$, under the influence of externally applied
electric fields. We first analyze the derived Jones matrix for a static TNLC
cell, i.e., with no applied field, and discuss its expected behaviour in the
so-called adiabatic following regime for different effective phase
retardations $\Gamma$. There is a potential dual behaviour that a TNLC cell
exhibits—which has not previously been reported—and it has wide-reaching
implications for their spatially varying extensions. An incident circularly
polarized beam may acquire three unique phase distributions from,
respectively, $\Gamma=0$, $\Gamma=\pi$, and $\Gamma=0$ with reversed-plate
orientation. Dual-plates (DP), as we will call them, thus promise a switch-
like capability between phase distributions. Moreover, an externally applied
electric field ultimately enables a transition in the effective topological
charge of generated polarization patterns. Proof-of-principle spatially
twisted liquid-crystal devices are fabricated and compared with the above two
case studies.

Figure 1: Twisted nematic liquid-crystal cells. a. Illustration of liquid
crystals twisting between two glass plates, uniformly aligned at $0^{\circ}$
and $\alpha$ for the front and back layers, respectively, spaced apart by a
distance of $L$. The action of this configuration is shown on the Poincaré
sphere on a horizontally polarized input state (black dot) for varying twist
angles $\alpha$ between $-\pi/2$ and $+\pi/2$, with a birefringence of b.
$\Gamma=\pi$, and c. $\Gamma=1001\pi$ in the adiabatic following regime.

## Results

### No applied field

A TNLC cell of thickness $L$ without any externally applied field can be
modelled as a stack of infinitely thin waveplates which are gradually twisted
from one to the next. With a total twist angle of $\alpha$, and assuming
strong anchoring, the twist distribution with propagation is linear,
$\phi(z)=\alpha z/L$, where $z$ is the axis normal to the cell Yamauchi
(2005). For positively uniaxial nematic liquid crystals with extraordinary and
ordinary refractive indices $n_{e}$ and $n_{o}$, respectively, we can define
the parameter $\Gamma=2\pi(n_{e}-n_{o})L/\lambda$ ($\lambda$ is the wavelength
of the incident light) which corresponds to the total phase retardation for a
zero-twist cell. The Jones matrix describing the TNLC configuration, shown in
Fig. 1-a, can be put in the form Yariv and Yeh (2003); Yamauchi (2005)

$\displaystyle\mathbf{T}_{0}(\alpha,\Gamma)$ $\displaystyle=$
$\displaystyle\mathbf{R}(-\alpha)\begin{bmatrix}\cos X-\frac{i\Gamma}{2X}\sin
X&\frac{\alpha}{X}\sin X\\\ -\frac{\alpha}{X}\sin X&\cos
X+\frac{i\Gamma}{2X}\sin X\end{bmatrix}$ (1) $\displaystyle=:$
$\displaystyle\mathbf{R}(-\alpha)\mathbf{M}_{0}(\alpha,\Gamma),$

with $X=\sqrt{\alpha^{2}+(\Gamma/2)^{2}}$, and the rotation matrix
$\mathbf{R}(\cdot)$ is

$\mathbf{R}(\cdot)=\begin{bmatrix}\cos(\cdot)&\sin(\cdot)\\\
-\sin(\cdot)&\cos(\cdot)\end{bmatrix}.$ (2)

For the case where the fast axes of the front and back layers are aligned at
angles $\phi_{f}$ and $\phi_{b}$, respectively, the general twisted matrix,
after setting $\alpha=\phi_{b}-\phi_{f}$, becomes

$\displaystyle\mathbf{T}_{\phi_{f}}(\alpha,\Gamma)$ $\displaystyle=$
$\displaystyle\mathbf{R}(-\phi_{f})\left[\mathbf{R}(\phi_{f}-\phi_{b})\mathbf{M}_{0}(\alpha,\Gamma)\right]\mathbf{R}(\phi_{f})$
(3) $\displaystyle=$
$\displaystyle\mathbf{R}(-\alpha)\mathbf{M}_{\phi_{f}}(\alpha,\Gamma),$

with
$\mathbf{M}_{\phi_{f}}(\alpha,\Gamma)=\mathbf{R}(-\phi_{f})\mathbf{M}_{0}(\alpha,\Gamma)\mathbf{R}(\phi_{f})$.

The optical action associated with Eq. (1) can be conveniently visualized on
the Poincaré sphere (PS). We recall that the positive (negative) points of the
three principal axes $\hat{S}_{1}$, $\hat{S}_{2}$, $\hat{S}_{3}$ on the PS
correspond, respectively, to horizontal (vertical), diagonal (anti-diagonal),
and left-hand circular (right-hand circular) polarization states. Figure 1-b
demonstrates the action of a TNLC on a horizontally polarized input for twist
angles between $-\pi/2$ and $\pi/2$ and a global phase retardation of
$\Gamma=\pi$. In this case, the output is always elliptical, wherein the
handedness is determined by the sign of $\alpha$.

Remarkably, if $\Gamma\gg\alpha$, then $\mathbf{M}_{0}(\alpha,\Gamma)$
simplifies to a waveplate $\mathbf{W}_{0}(\Gamma)$ with its fast axis along
the horizontal:

$\mathbf{W}_{0}(\Gamma)=\begin{bmatrix}e^{-i\tfrac{\Gamma}{2}}&0\\\
0&e^{i\tfrac{\Gamma}{2}}\end{bmatrix},$ (4)

and Eq. (3) becomes

$\mathbf{T}_{\phi_{f}}(\alpha,\Gamma)\approx\mathbf{R}(-\alpha)\mathbf{W}_{\phi_{f}}(\Gamma),$
(5)

where
$\mathbf{W}_{\phi_{f}}(\Gamma)=\mathbf{R}(-\phi_{f})\mathbf{W}_{0}(\Gamma)\mathbf{R}(\phi_{f})$
is a typical waveplate of retardance $\Gamma$, with fast axis oriented at
$\phi_{f}$. Consequently, an input beam that is linearly polarized either
parallel or orthogonal to $\phi_{f}$ will be rotated by $\alpha$ in the
counterclockwise direction. This is referred to as adiabatic following. Figure
1-c demonstrates this behaviour for a horizontally polarized input state for a
range of twist angles, wherein the output states are calculated via Eq. (1)
with $\Gamma=1001\pi$. The horizontal input—parallel to the input alignment
layer—is rotated by $\alpha$ toward other linear states along the equator of
the PS.

Figure 2: Fabricated samples. False colour images of a. discretized DP(0,1/2),
and b. DP(1,2) between crossed polarizers under a microscope illuminated with
white light. The topological pattern on each glass plate is also shown. Note
that the $q=1/2$ pattern is discretized into 16 slices for the patterning
process.

Furthermore, the effective phase retardation is defined modulo $2\pi$ (see Eq.
(4)). Accordingly, Eq. (5) simplifies further for two cases: $\Gamma\
(\mathrm{mod}\ 2\pi)=0$ and $\Gamma\ (\mathrm{mod}\ 2\pi)=\pi$. For $\Gamma\
(\mathrm{mod}\ 2\pi)=0$, $\mathbf{W}_{\phi_{f}}(0)$ reduces to the identity
matrix, leaving

$\mathbf{T}_{\phi_{f}}(\alpha,0)=\begin{bmatrix}\cos{\alpha}&-\sin{\alpha}\\\
\sin{\alpha}&\cos{\alpha}\end{bmatrix}.$ (6)

Left-handed (L) and right-handed (R) circular polarizations are rotation-
invariant states (up to a global phase), so explicitly acting this on the
circular basis yields
$\\{\ket{L},\ket{R}\\}\rightarrow\\{e^{-i\alpha}\ket{L},e^{i\alpha}\ket{R}\\}$,
where we have adopted bra-ket notation to denote the complex polarization
spinor. For the case where $\Gamma\ (\mathrm{mod}\ 2\pi)=\pi$, Eq. (5)
simplifies to

$\mathbf{T}_{\phi_{f}}(\alpha_{\pi},\pi)=\begin{bmatrix}\cos{\alpha_{\pi}}&\sin{\alpha_{\pi}}\\\
\sin{\alpha_{\pi}}&-\cos{\alpha_{\pi}}\end{bmatrix},$ (7)

with $\alpha_{\pi}=\phi_{b}+\phi_{f}$. This is the Jones matrix for a half-
wave plate oriented at $\alpha_{\pi}/2$, yielding
${\\{\ket{L},\ket{R}\\}\rightarrow\\{e^{i\alpha_{\pi}}\ket{R},e^{-i\alpha_{\pi}}\ket{L}\\}}$.
Of course, we can also consider the case for an arbitrary $\Gamma$. In this
case, Eq. (5) can be rewritten explicitly:

$\mathbf{T}_{\phi_{f}}(\alpha,\Gamma)=\frac{(1+e^{i\Gamma})}{2}\mathbf{T}_{\phi_{f}}(\alpha,0)+\frac{(1-e^{i\Gamma})}{2}\mathbf{T}_{\phi_{f}}(\alpha_{\pi},\pi).$
(8)

The general case is, therefore, a superposition of the cases $\Gamma\
(\mathrm{mod}\ 2\pi)=0$ and $\Gamma\ (\mathrm{mod}\ 2\pi)=\pi$, and the
relative amplitudes are determined by the phase retardation.

In the following, we extend this analysis to spatially varying dual-plates,
allowing for a spatial distribution of the fast-axis orientation of the front
and back plates in the transverse plane, $\Phi_{f}(r,\varphi)$ and
$\Phi_{b}(r,\varphi)$, respectively, described in cylindrical coordinates. An
incident circularly polarized beam will acquire a spatially varying phase
proportional to either
$\alpha(r,\varphi)=\Phi_{b}(r,\varphi)-\Phi_{f}(r,\varphi)$, or
$\alpha_{\pi}(r,\varphi)=\Phi_{b}(r,\varphi)+\Phi_{f}(r,\varphi)$, i.e., the
difference or sum of the two fast axis distributions, depending on if $\Gamma\
(\mathrm{mod}\ 2\pi)=0$ or $\Gamma\ (\mathrm{mod}\ 2\pi)=\pi$. Therefore, we
can use the phase retardation of a dual-plate to toggle between two different
behaviours. Additionally, for a given dual-plate, if the front and back layers
are reversed—i.e., the orientation of the plate is flipped, or the beam enters
from the back—a distinct third phase pattern could be acquired at $\Gamma\
(\mathrm{mod}\ 2\pi)=0$, defined by $\alpha^{(-)}=\Phi_{f}-\Phi_{b}=-\alpha$.
Of course, we could also consider the inverse problem wherein we desire a
particular twist distribution; in this case,
$\Phi_{b}=(\alpha_{\pi}+\alpha)/2$ and $\Phi_{f}=(\alpha_{\pi}-\alpha)/2$.

One potential challenge that arises when dealing with the general class of
non-symmetric inhomogeneous liquid-crystal plates is that nematic liquid-
crystals only favourably twist between $-90^{\circ}$ and $90^{\circ}$. For
example, if $\phi_{f}=22.5^{\circ}$ and $\phi_{b}=157.5^{\circ}$, then
$\alpha=-45^{\circ}$, and not $135^{\circ}$, in order to achieve the lowest
possible twist. At locations where $\phi_{f}$ and $\phi_{b}$ are orthogonal,
there is an ambiguity as to whether $\alpha=+90^{\circ}$ or $-90^{\circ}$.
This leads to discontinuities in the twist distribution, which results in
$\pi$-phase jumps appearing in these locations. Consequently, a spatially
varying global phase distribution will be imparted to any input beam,
regardless of the birefringence setting.

An interesting example of a spatially varying dual-plate is the
multi–$q$–plate. The functionality of a $q$-plate is equivalent to that of a
half-wave plate with a fast-axis distribution of
$\Phi_{\text{QP}}(\varphi)=q\varphi+\varphi_{0}$, where $q$, the topological
charge of the plate, is either a full or half-integer, and $\varphi_{0}$ is an
offset angle. Its Jones matrix is identical to Eq. (7) with
$\alpha_{\pi}=2\Phi$, and circularly polarized input will experience spin-to-
orbital angular momentum coupling. For example, an input photon with a spin of
$+(-)\hbar$ along the axis of propagation will gain $+(-)2q\hbar$ units of
orbital angular momentum (OAM), where $\hbar$ is the reduced Planck constant,
at the expense of switching the spin to be $-(+)\hbar$. We have adopted the
convention that a spin of $+(-)\hbar$ corresponds to left (right) circularly
polarized light. A $q$-plate can thus span the two-dimensional vector space
$\\{\ket{R,2q},\ket{L,-2q}\\}$, where we have used Dirac notation with the
labels corresponding to a photon’s polarization and OAM, respectively.

Figure 3: Stokes vectors reconstruction. Reconstructed average Stokes vectors
in each of the 16 slices (coloured points) for the cardinal input states
(black points), and theoretical fit (line) using the TNLC Jones matrix with
$\Gamma_{\text{fit}}=51.7$ for a. DP(0,1/2), and b. DP(1/2,0). The
experimental Stokes vectors plotted are the average values in each slice of
the discretized sample.

However, in order to have access to states created from a $-q$-plate—thus
accessing a four-dimensional vector space—it is necessary to place a half-wave
plate, or an equivalent device, to impart a $\lambda/2$ retardation, after the
original $q$-plate. Additionally, it is impossible to modify the topology of a
$q$-plate once this is fabricated. An appropriately patterned dual-plate is
capable of performing both of these tasks without the aid of additional
optical elements. For example, define the front and back distributions to be
$\Phi_{f}(\varphi)=q_{f}\varphi$ and $\Phi_{b}(\varphi)=q_{b}\varphi$,
respectively (we have dropped the offset angles without loss of generality).
If $q_{f}=q_{b}$, we straightforwardly recover the behaviour of a regular
$q=q_{b}$-plate so that we will assume $q_{f}\neq q_{b}$, and we get the
following cases.

(1) If $q_{f}=0$, then $\alpha=\alpha_{\pi}=q_{b}\varphi$, and we lose the
dual behaviour of the dual-plate. (2) However, if $q_{b}=0$, then
$\alpha=-q_{f}\varphi$ and $\alpha_{\pi}=q_{f}\varphi$; we can thus use the
phase retardation of the dual-plate to toggle between the behaviour of
oppositely charged $q_{f}$-plates. We note that cases (1) and (2) are the same
dual-plate; however, the orientation of the device—or equivalently the beam
propagation—is reversed. So, while we obtain two distinct behaviours through
$\alpha$ and $\alpha_{\pi}$, we do not gain a third behaviour since
$\alpha^{(-)}=\alpha$ in case (2). (3) In general, if $|q_{f}|,|q_{b}|>0$,
then $\alpha\neq\alpha_{\pi}$ and $\alpha^{(-)}\neq\alpha$, thus creating a
multi-$q$-plate with three possible behaviours. We note that the same concept
can be applied to any phase distribution. Indeed, we are not limited to
$\Phi_{f}$ and $\Phi_{b}$ being of the same class of phases. In this way, we
can create arbitrary dual-functionality dual-setting devices.

Figure 4: Stokes vectors reconstruction. For horizontally polarized input
light, the theoretical and experimentally reconstructed local Stokes vectors
(arrows) for a. DP(1,2) and b. DP(2,1). The arrow color is a measure of the
local polarization’s ellipticity, with left-hand circular as red, right-hand
circular as blue, and linear as green.

To explore the validity of the TNLC model, several dual-plates are fabricated
and characterized (see Materials and Methods). The first device is a
discretized multi $q$-plate DP($q_{b}$=0, $q_{f}$=1/2), with $\sim$35 $\mu$m
spacers, where DP stands for dual-plate, and the labels are the topologies of
the front and back plate. Figure 2a shows the fabricated sample as it appears
between crossed polarizers. The $q=1/2$ topology is discretized into 16 slices
such that a range of twist angles from $[-90^{\circ},90^{\circ}]$ can be
efficiently characterized with enough room in each slice to average
imperfections from the assemblage. The second sample fabricated and tested is
a DP(1,2), shown in Fig. 2b. The experimentally calculated average Stokes
vectors for each of the 16 slices of DP(0,1/2) with different input polarized
light are plotted on the Poincaré sphere, along with theoretical fits (see
Fig. 3a). The results for DP(1/2,0) are shown in Fig. 3b. Figures 4a-b show
the locally reconstructed Stokes vectors for horizontally polarized input
light on DP(1,2) and DP(2,1), respectively. A very nice agreement is observed
for the two cases, despite the singularity misalignment deriving from the
fabrication process.

Figure 5: Externally applied voltage on DP(0,1/2). a. Reconstructed local
Stokes vectors (arrow) from a horizontally polarized input for voltage:
$V_{pp}=3.00$, 6.00, 8.00, and 12.00 V. The color corresponds to the
polarization ellipse angle $\psi$ with respect to the horizontal. b. The
experimental data (dots) are the output Stokes vectors.

The diameters of the spacers placed between the glass plates of the cell range
from 32–38 $\mu$m, with an average of 35 $\mu$m. This gives our sample an
average phase retardation of $\Gamma_{avg}=51.7$ for 6CHBT liquid crystals
with $\Delta n=0.151$ Nowinowski-Kruszelnicki  _et al._ (2012). This
$\Gamma_{avg}$ is used when plotting the theoretical fits using the TNLC Jones
matrix $\mathbf{T}_{\phi_{f}}(\alpha,\Gamma)$ of Eqs. (3) in Figs. 3a-b. As a
measure of good fit, the average state overlap across all twist angles for a
given input polarization is used. The overlap is calculated as
$|\mathbf{S}_{\text{exp}}\cdot\mathbf{S}_{\text{TNLC}}|^{2}$, where
$\mathbf{S}_{\text{exp}}$ and $\mathbf{S}_{\text{TNLC}}$ are the
experimentally-reconstructed and theoretical Stokes vectors, respectively. The
uncertainties on each average Stokes vector are the standard deviations from
the area used for averaging, which are less than $10\%$ for each Stokes
parameter in all cases. The theoretical fits using $\Gamma_{avg}=51.7$
imitates the experimental data for all input polarizations and the two
orientation cases very well, with total average overlaps of 89% and 87% for
DP(0,1/2) and DP(1/2,0), respectively.

In the plots obtained for horizontally and vertically polarized inputs, we can
observe a slight deviation from the adiabatic following regime, which predicts
a mere rotation toward other linear polarizations. In this case, the adiabatic
following regime could be better enforced by employing more birefringent
nematic liquid crystals or larger spacers.

###  .1 Externally applied field

When an electric voltage is applied across the DP(0,1/2) sample, the resulting
effect on polarized light can be surprising. Figure 5a shows the reconstructed
local Stokes vectors at different voltages from a horizontally polarized input
state. The color is encoded to show the azimuthal angle
$\psi=\arctan({S_{2}/S_{1}})$ of the Stokes vectors on the Poincaré sphere. We
experimentally observe that the overall topological charge can be tuned from
$q\sim 1/2$ to $q\sim 1$ to $q\sim 0$, and back to $q\sim 1$ as we increase
the field strength. While detuning to $q=0$ is observed with standard
$q$-plates, this apparent charge-doubling is never observed nor achievable.
This behaviour is also not accounted for by simply varying $\Gamma$ in
$\mathbf{T}_{\phi_{f}}(\alpha,\Gamma)$, and we must, therefore, extend our
model.

Liquid crystals may be regarded as a continuous medium with a set of elastic
constants. As such, the elastic continuum theory De Gennes and Prost (1993)
has been an excellent way to describe the influence of boundary conditions and
externally applied fields—whether electric or magnetic—on these systems. Since
liquid crystals are electrically polarizable, diamagnetic, and anisotropic in
their electric/magnetic properties, an applied field will cause the molecules
to align with the field direction. In general, the tilt and twist
distributions of the liquid-crystal directors within the bulk become non-
trivial, and typically non-analytical. These are determined by minimizing the
Frank–Oseen free-energy density of the system Deuling (1993). The solution
that arises is a coupled set of highly singular integrals to be solved
simultaneously. In the high-voltage limit ($V/V_{T0}\geqslant 4$), certain
approximations can be made to simplify solving these integrals around their
singularities, with some analytical approximations when $V\gg V_{T0}$ Preist
_et al._ (1989). $V_{T0}$ is the characteristic threshold voltage of the
system, dependent on material parameters; here, $V_{T0}=0.966~{}V$ or
$V_{pp}=2.733~{}V$. To analyze more general cases, we have opted to use a
numerical minimization approach based on evolutionary methods, specifically a
genetic algorithm, to also look at the response of the system for field
strengths below the high-voltage limit. Remarkably, this method does not
require any a-priori hypothesis on the applied field. The details of our
numerical approach will be found in a separate technical paper Sit  _et al._
(2023).

Figure 6: Experimental setups a. Setup to pattern the glass plates for
fabricating liquid-crystal devices. The inset shows the cross-section of the
constructed LC cell b. Setup to characterize the fabricated samples. DMD =
digital micromirror device, PBS = polarizing beamsplitter, FG = function
generator.

With the numerically calculated tilt and twist distributions, the overall
Jones matrix $\mathbf{J}_{\phi_{f}}(\alpha,V)$ is derived by multiplying
together the $N$ twisted liquid crystal cells of thickness $d$ using the form
of Eq. (3):

$\mathbf{J}_{\phi_{f}}(\alpha,V)=\mathbf{R}(-\phi_{b})\left[\prod_{j=0}^{N}\mathbf{M}_{0}(\alpha^{\prime}(z_{j}),\Gamma(z_{j}))\right]\mathbf{R}(\phi_{f}),$
(9)

where $\alpha^{\prime}(z_{j})=\phi(z_{j+1})-\phi(z_{j})$, $z_{j}=jd$,
$\phi_{f}=\phi(0)$ and $\phi_{b}=\phi(L)$ are the front and back alignment
angles, respectively, and $\alpha=\phi(L)-\phi(0)$ is the total twist angle.
The phase retardation $\Gamma(z_{j})$ within the $j^{\text{th}}$ slice is
calculated from the tilt distribution $\theta(z)$ using the trapezoidal rule
in favour of left or right Riemann sums for a better estimate,

$\Gamma(z_{j})=\frac{\pi\Delta
nd}{\lambda}[\cos^{2}\theta(z_{j+1})+\cos^{2}\theta(z_{j})].$ (10)

A cell thickness of $L=35~{}\mu$m is used, with $N=100$ slices for $d=L/N$.
Figure 5b compares the experimentally reconstructed Stokes data for DP(0,1/2)
and horizontally polarized input light with numerical predictions.

This model reproduces quite accurately the experimental results. This would
not be possible by simply assuming a linear twist distribution and varying
$\Gamma$. In particular, the non-linear twist distributions account for the
peculiar phenomenon of the evolving topological charge is shown in Fig. 5a.
Further investigations will be required to determine if we achieve an actual
evolution of the topological charge for the continuous version of DP(0,1/2).

## Discussion and Conclusions

We have shown the versatility of patterned twisted nematic cells, focusing on
the action of dual $q$-plates on polarized light. The case in which no
external field is applied has been analyzed in detail, and an analytic Jones
matrix description of the dual-plates has been derived. We demonstrated that
one can observe different functionalities depending on the effective
retardation $\Gamma$ or the plate orientation. It must be noted that $\Gamma$
can either be fixed or tuned by means of temperature control Karimi  _et al._
(2009). Another way of controlling the effective retardation is via an applied
field. However, in the case of dual-plates, an external field introduces
additional deformations in the liquid-crystal medium which affects the overall
distribution of the molecular director. As a consequence, the resulting Jones
matrix can be much more complicated. At the same time, this can lead to
intriguing unexpected effects, for instance, the generation of polarization
patterns with apparent topologies which depend on the applied voltage.
Typically, a change in topology induced by a continuous parameter is
associated with the existence of an intermediate critical point. However, we
observe that dual $q$-plates always exhibit singular lines in the liquid-
crystal orientation pattern, which are transferred to the polarization
distribution of the transmitted light. Accordingly, no topological charge can
be rigorously associated with the polarization distribution. Nevertheless, the
observed patterns display distinguishable features, like lemon and azimuthal
patterns. Dual-plates open a new avenue for optical functionalities based on
liquid crystals. Here, we focused on plates with different topological
structures patterned on the two alignment layers. Other possibilities will
include dual Fresnel lenses, axicons, gratings, and magic windows. The use of
these structures will also allow novel studies and realizations of devices
introducing 3D geometric phases Slussarenko  _et al._ (2016).

## Materials and methods

The process to fabricate dual-plates is similar to that for $q$-plates Rubano
_et al._ (2019); Larocque  _et al._ (2016). First, we start with two glass
substrates, each with a conductive layer of indium tin oxide (ITO). A drop of
an azobenzene-based dye (PAAD-22, provided by BEAM Co) is deposited on top of
the ITO; the sample is then spin-coated for 30 s at 4000 rpm, and baked at
120∘ for 5 minutes. When exposed to the light of a wavelength around the peak
of the azodye’s absorption spectrum, the molecules will photoalign themselves
according to the light linear polarization. Here, we use a 430-nm laser.
Figure 6a shows the setup to pattern the sample. In particular, a digital
micro-mirror device (DLP3000 DLP® 0.3 WVGA Series 220 DMD) is programmed to
reflect a tailored intensity pattern. The resolution of the DMD is 608x684
with a micromirror pitch of 7.6 $\mu$m. A HWP can then be rotated to adjust
the polarization to the required orientation.

When fabricating $q$-plates, the two glass substrates are first glued
together—with spacers between them to create a uniform cavity for the liquid
crystals—and then the sample is exposed with the desired pattern. For dual-
plates, each substrate is separately exposed with the front and back plate
patterns, respectively; then they are glued together, taking care to overlap
the two patterns as best as possible by hand. For the samples presented here,
silica microspheres with diameters between 32 $\mu$m and 38 $\mu$m are used as
the spacers. Note, the two substrates are glued with a lateral offset such
that wires can be soldered to the conductive ITO layer; this will allow for a
voltage to be applied across the sample. Finally, the nematic liquid crystals
(here, 6CHBT) are injected into the cavity and the remaining sides are sealed
off with glue. The completed sample is heated to $\sim$$100^{\circ}$C on a hot
plate and cooled to room temperature once more to cement the alignment of the
liquid crystals with the desired patterns.

The setup to characterize the fabricated sample is shown in Fig. 6b. A red
635-nm diode laser is used to illuminate the sample; it is expanded using a
telescoping lens system to completely cover the patterned area. A polarizing
beamsplitter (PBS), half-wave plate (HWP), and quarter-wave plate (QWP) are
used to prepare the input polarization state. A second set of QWP, HWP, and
PBS is used to project the output polarization state from the sample onto a
different polarization state. Another telescoping lens system (not shown) is
used to image the sample plane and shrink the beam down to fit onto a CCD
camera in order to record the intensity measurement. A function generator (FG)
applies a sinusoidal waveform with 4 kHz frequency and peak-to-peak voltages
$V_{pp}$ between $0~{}V$ and $20~{}V$ to the prepared sample. Polarization
state tomography is thus performed for the six cardinal polarizations as
inputs on the sample. This consists of projecting the output polarization
state onto the six cardinal states and recording the intensity using the CCD
camera, i.e., there are 6 measurements for each input state, for 36
measurements (images) total per configuration of the sample.

## Acknowledgements

A.S. acknowledges the financial support of the Vanier graduate scholarship of
the NSERC. This work was supported by the Canada Research Chairs (CRC) program
and the Natural Sciences and Engineering Research Council of Canada (NSERC).

## References

  * De Gennes and Prost (1993) P. G. De Gennes and J. Prost,  _The Physics of Liquid Crystals_ , 2nd ed. (Oxford: Clarendon Press, 1993).
  * Shanks (1982) I. A. Shanks, Contemporary Physics 23, 65 (1982).
  * Schadt (1988) M. Schadt, Molecular Crystals and Liquid Crystals 165, 405 (1988).
  * Yeh and Gu (2009) P. Yeh and C. Gu,  _Optics of liquid crystal displays_ , Vol. 67 (John Wiley & Sons, 2009).
  * Gooch and Tarry (1975) C. Gooch and H. Tarry, Journal of Physics D: Applied Physics 8, 1575 (1975).
  * Ong (1987) H. L. Ong, Applied Physics Letters 51, 1398 (1987).
  * Brasselet  _et al._ (2009) E. Brasselet, N. Murazawa, H. Misawa, and S. Juodkazis, Phys. Rev. Lett. 103, 103903 (2009).
  * Marrucci  _et al._ (2011) L. Marrucci, E. Karimi, S. Slussarenko, B. Piccirillo, E. Santamato, E. Nagali, and F. Sciarrino, Journal of Optics 13, 064001 (2011).
  * Rubano  _et al._ (2019) A. Rubano, F. Cardano, B. Piccirillo, and L. Marrucci, Journal of the Optical Society of America B 36, 70 (2019).
  * Yan  _et al._ (2015) L. Yan, P. Gregg, E. Karimi, A. Rubano, L. Marrucci, R. W. Boyd, and S. Ramachandran, Optica 2, 900 (2015).
  * D’Ambrosio  _et al._ (2013) V. D’Ambrosio, N. Spagnolo, L. Del Re, S. Slussarenko, Y. Li, L. C. Kwek, L. Marrucci, S. P. Walborn, L. Aolita, and F. Sciarrino, Nature Communications 4, 1 (2013).
  * Karimi  _et al._ (2012) E. Karimi, L. Marrucci, C. de Lisio, and E. Santamato, Optics Letters 37, 127 (2012).
  * Sit  _et al._ (2017) A. Sit, F. Bouchard, R. Fickler, J. Gagnon-Bischoff, H. Larocque, K. Heshami, D. Elser, C. Peuntinger, K. Günther, B. Heim, C. Marquardt, G. Leuchs, R. W. Boyd, and K. E., Optica 4, 1006 (2017).
  * Cardano  _et al._ (2015) F. Cardano, F. Massa, H. Qassim, E. Karimi, S. Slussarenko, D. Paparo, C. de Lisio, F. Sciarrino, E. Santamato, R. W. Boyd, and L. Marrucci, Science Advances 1, e150087 (2015).
  * Stalder and Schadt (1996) M. Stalder and M. Schadt, Optics Letters 21, 1948 (1996).
  * Cardano  _et al._ (2012) F. Cardano, E. Karimi, S. Slussarenko, L. Marrucci, C. de Lisio, and E. Santamato, Applied Optics 51, C1 (2012).
  * Nys  _et al._ (2022) I. Nys, B. Berteloot, and K. Neyts, Materials 15, 2453 (2022).
  * Yamauchi (2005) M. Yamauchi, Applied optics 44, 4484 (2005).
  * Yariv and Yeh (2003) A. Yariv and P. Yeh, Optical waves in crystals: propagation and control of laser radiation (2003).
  * Nowinowski-Kruszelnicki  _et al._ (2012) E. Nowinowski-Kruszelnicki, J. Kedzierski, Z. Raszewski, L. Jaroszewicz, M. A. Kojdecki, W. Piecek, P. Perkowski, M. Olifierczuk, E. Miszczyk, K. Ogrodnik, and P. Morawiak, Opto-electronics Review 20, 255 (2012).
  * Deuling (1993) H. J. Deuling, Molecular Crystals and Liquid Crystals 27, 81 (1993).
  * Preist  _et al._ (1989) T. W. Preist, K. R. Welford, and J. R. Sambles, Liquid Crystals 4, 103 (1989).
  * Sit  _et al._ (2023) A. Sit, F. Di Colandrea, A. D’Errico, and E. Karimi, In Preparation (2023).
  * Karimi  _et al._ (2009) E. Karimi, B. Piccirillo, E. Nagali, L. Marrucci, and E. Santamato, Applied Physics Letters 94 (2009).
  * Slussarenko  _et al._ (2016) S. Slussarenko, A. Alberucci, C. P. Jisha, B. Piccirillo, E. Santamato, G. Assanto, and L. Marrucci, Nature Photonics 10, 571 (2016).
  * Larocque  _et al._ (2016) H. Larocque, J. Gagnon-Bischoff, F. Bouchard, R. Fickler, J. Upham, R. W. Boyd, and E. Karimi, Journal of Optics 18, 124002 (2016).