# towards a unified Conformer structure: from ASR to ASV task

###### Abstract

Transformer has achieved extraordinary performance in Natural Language
Processing and Computer Vision tasks thanks to its powerful self-attention
mechanism, and its variant Conformer has become a state-of-the-art
architecture in the field of Automatic Speech Recognition (ASR). However, the
main-stream architecture for Automatic Speaker Verification (ASV) is
convolutional Neural Networks, and there is still much room for research on
the Conformer based ASV. In this paper, firstly, we modify the Conformer
architecture from ASR to ASV with very minor changes. Length-Scaled Attention
(LSA) method and Sharpness-Aware Minimization (SAM) are adopted to improve
model generalization. Experiments conducted on VoxCeleb and CN-Celeb show that
our Conformer based ASV achieves competitive performance compared with the
popular ECAPA-TDNN. Secondly, inspired by the transfer learning strategy, ASV
Conformer is natural to be initialized from the pretrained ASR model. Via
parameter transferring, self-attention mechanism could better focus on the
relationship between sequence features, brings about 11% relative improvement
in EER on test set of VoxCeleb and CN-Celeb, which reveals the potential of
Conformer to unify ASV and ASR task. Finally, we provide a runtime in ASV-
Subtools to evaluate its inference speed in production scenario. Our code is
released at https://github.com/Snowdar/asv-
subtools/tree/master/doc/papers/conformer.md.

Index Terms—  speaker verification, Conformer, transfer learning, runtime

##  1 Introduction

Automatic Speaker Verification (ASV) is a task to verify the identity of the
speaker by voice, which has been well-developed and widely applied in many
real-world scenarios. Currently, x-vector proposed by Snyder et al. [1] is the
most popular framework for ASV systems. It includes two parts, where an
embedding extractor maps utterances with variable duration to fixed-
dimensional speaker representations, and then the similarity of the speaker
representation can be calculated by back-end scoring method. Many prior works
focused on DNN-based structure have improved the performance of ASV systems
(e.g., ResNet, Res2Net, ECAPA-TDNN) [2, 3, 4, 5]. Most of above networks are
Convolutional Neural Networks (CNNs), which have the inherent ability of
emphasizing the local information.

Recently, self-attention mechanisms that directly capture the global
information have been explored, and it has helped Transformer [6] achieve
remarkable success in Natural Language Processing (NLP) and Computer Vision
(CV) areas [7, 8]. However, unlike CNNs, Transformer lacks some of the
inductive biases, such as translation equivariance and locality, which
degrades performance when trained on insufficient data. It is difficult to
achieve competitive results by directly applying Transformer to ASV tasks [9,
10]. Conformer [11] is a hybrid architecture which combines self-attention
with convolutions, i.e., self-attention learns the global interaction while
convolutions capture the local information. It has become a state-of-art model
in Automatic Speech Recognition (ASR). MFA-Conformer [12] utilizes the Multi-
scale Feature Aggregation method in ECAPA-TDNN, successfully introduces
Conformer into ASV for the first time. However, the uniformity between ASV an
ASR deserves further attention. The same Conformer establishes connections
between ASV and other tasks, which will not only facilitate better research on
the link between ASV and ASR, but may also be a foundation of future multi-
task learning or multimodal machine learning. Hence, in this paper, we mainly
concentrate on ASV Conformer which matches ASR encoder.

In addition, several studies have injected phonetic information into the DNN
structure of the ASV extractor through multi-task learning [13, 14, 15],
indicates that there exists some positive interdependence between the speaker
identities and ASR tasks when sharing some of the low-level computation.
Meanwhile, in the field of Language Identification (LID), providing
informative speech representation by a pretrained ASR model in LID system,
proved to be effective for the downstream LID task [16]. It is worth mentioned
that [17] adopts transfer learning scheme, that is, pretrains a U2++ encoder-
decoder [18] model and then further finetunes the encoder for the LID task,
won the first place in the OLR 2021 [19]. It can be well explained by the fact
that the ASR encoder already has a strong capability to discriminate
languages, since the supervised training labels for ASR are language-related.
Although the association with ASR information in ASV task is not as apparent
as in LID, e.g., different speakers can say the same words. Their deeper
dependencies could be digged by appropriate methods. Inspired by these works,
we propose a parameter transferring strategy, which can make use of a typical
ASR model to improve the performance of ASV system.

At last, for the purpose of bridging the gap between production and research,
we provide a C++ based runtime tool to evaluate our models’ inference speed in
production environment. With Torch Just In Time (JIT) and LibTorch, models
trained by Pytorch can be converted to TorchScrip, and then employed in C++
applications. Our main contributions in this paper are as follows.

  * •

We modify Conformers of different configures from ASR to ASV system. To
improve model generalization ability, Length-Scaled Attention (LSA) method
[20] enables self-attention to better generalize to various length inputs, and
Sharpness-Aware Minimization (SAM) [21] prevents the loss from falling into
the local minima during training. Our system yields competitive results in
popular VoxCeleb [22, 23] and CN-Celeb [24, 25].

  * •

Through a parameter transferring strategy, we show that ASV Conformer could
benefit from ASR information. Parts of the ASR encoder is selected to
initialize ASV Conformer, then we retrain the model rather than finetune it.
This method allows model to learn the deep relationship with ASR.

  * •

We provide a runtime to conform the production value of our models, make it
easier and more convenient to deploy ASV models to real applications.

##  2 Methods

###  2.1 Model architecture

The overview of ASV Conformer system is shown in Fig. 1. Rotary Position
Embedding [26], which incorporates explicit relative position dependency in
the form of absolution position embedding, encode the position information
into the self-attention mechanism. On one hand, a stack of Conformer blocks
model the frame-level speaker representation. On the other hand, an Attentive
Statistics Pooling [27] layer process all the information across the time
dimension, resulting in a segment-level vector. After linear layer, the
segment-level vector is further projected to fixed-dimensional x-vector. In
training stage, the speaker embedding extractor is optimized by AAM-Softmax
[28] loss.

The Conformer block consists of two Macaron-like feed forward modules (FNN)
with half residual connections, between them lies the Multi-head self-
attention (MHSA) and Convolution module (Conv). MHSA means to model the global
information, and Conv gives inductive bias to network.

Fig. 1: Schematic diagram of ASV Conformer.

Without loss of generality, we assume MHSA has one attention head. An standard
self attention can be described as mapping a query and a set of key-value
pairs to an output, and the matrix of outputs is computed as the weighted sum
over the value representation:

Fig. 2: Schematic diagram of ASR transferring.

$\operatorname{Att}(Q,K,V)=\operatorname{Softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V$
(1)

where $d$ is the embedding dimension, and its detailed form is:

$\boldsymbol{o}_{i}=\sum_{j=1}^{n}a_{i,j}\boldsymbol{v}_{j},\quad
a_{i,j}=\frac{exp\left({\frac{\boldsymbol{q}_{i}\cdot\boldsymbol{k}_{j}}{\sqrt{d}}}\right)}{\sum_{j=1}^{n}exp\left({\frac{\boldsymbol{q}_{i}\cdot\boldsymbol{k}_{j}}{\sqrt{d}}}\right)}\vspace{-0.5em}$
(2)

where $n$ denotes the sequence length, $\boldsymbol{q}$, $\boldsymbol{k}$ and
$\boldsymbol{v}$ correspond to query, key and value vector respectively, and
$a_{i,j}$ means the attention score weight between time position $i$ and $j$.
The softmax function normalizes the attention weight across all position to a
probability distribution. However, there exits varying length inputs in
inference stage, and this mismatching may hurt the original relationship of
different position embeddings (e.g., a longer inputs diluting the attention
weight, making the value in position $i$ is inclined to be easier to be
influenced by unrelated position values). From the perspective of entropy
[29], equation 1 is modified to a length-scaled version as equation 3 to
stabilize the uncertainty of attention distribution.

$\operatorname{Att}(Q,K,V)=\operatorname{Softmax}\left(\frac{\log
n}{s{\sqrt{d}}}{QK^{T}}\right)V$ (3)

where $n$ denotes the sequence length, $s$ is set to a learnable scalar as a
temperature of softmax in different layers. In this way, the attention weights
after softmax function are sharpened by $\log n$ in long sequences, while
smoothed in short sequences.

###  2.2 Sharpness-Aware Minimization training

While Conformer’s superior representational capacity enables ASV system to
memorize the training set easily, it also leads to overfitting problems,
especially when training data is insufficient. It has been studied [30, 31]
that a model converging to sharp minimas of loss might results poorer
generalization. Sharpness-Aware Minimization (SAM), which leverages the
connection between geometry of the loss landscape and generalization, can seek
a flatter minima, and is adopted to improve generalization ability of our
models.

Intuitively, SAM aims to find the parameter $w$ whose entire neighbours have
low training loss $L_{train}$, it can be defined as:

$\min_{w}\max_{\|\epsilon\|_{2}\leq\rho}L_{train}(w+\epsilon)$ (4)

where $\rho\geq 0$ denotes the radius of seeking region. A two-step
approximation is applied to solve this minimax optimization:

$\left\\{\begin{array}[]{l}\epsilon_{t}\approx\rho\nabla
L_{train}\left(w_{t}\right)/\left\|\nabla
L_{train}\left(w_{t}\right)\right\|_{2}\\\ \left.w_{t+1}\approx
w_{t}-\alpha_{t}\nabla
L_{train}(w_{t})\right|_{w_{t}+\epsilon_{t}}\end{array}\right.$ (5)

where $\alpha_{t}$ is the learning rate in training position $t$. The first
step is an efficient approximation of

$\epsilon_{t}=\arg\max_{\|\epsilon\|_{2}\leq\rho}L_{{train}}(w_{t})+\epsilon_{t}^{T}\nabla_{w_{t}}L_{train}(w_{t})$

and in the second step, SAM updates weights based on the gradient in
$w_{t}+\epsilon_{t}$.

###  2.3 ASR transferring

In order to integrate ASR information, we pretrain the U2++[18] ASR model
first, and then optimize it to ASV task. However, unlike LID task [17], ASR
representation is not so related with speaker classification, so in the second
stage we apply the same strategy as training the original ASV Conformer. In
addition, it was previously found that ASR information is more helpful in
shallow layers [13, 14], we infer that transferring parts of ASR encoder to
ASV Conformer is compatible. Notably, the datasets used for ASR training are
independent and the model configure is almost same as it in ASV, giving
flexibility and convenient to the transfer scheme, i.e., a model for ASR task
can be directly transferred to ASV. As depicted in Fig. 2, the right part is
U2++ architecture, which can be either an open source pretrained model or
trained from scratch. Then the shared encoder is further to be trained to ASV
task.

##  3 Experimental setup

###  3.1 Datasets

We conducted our proposed experiments on CN-Celeb [24, 25] and VoxCeleb [22,
23] respectively. The CN-Celeb corpus contains speech from Chinese
celebrities. The entire dataset can be divided into CN-Celeb.T and CNC-Eval:
the former involves 632,736 utterance from 2,793 speakers with total 1,285
hours, and is used for training; the latter is test set. The VoxCeleb is one
of the most classic English dataset in the field of speaker verification. For
the VoxCeleb, we only used the VoxCeleb2 which involves 1,092,009 utterance
from 5994 speakers with total 2,300 hours as training set, and employed three
available test trials: VoxCeleb1-O-clean, VoxCeleb1-E-clean and VoxCeleb-H-
clean to verify our proposed method.

The pretrained ASR model involves three datasets, GigaSpeech [32], Multi-
CN[33] and Wenetspeech [34]. GigaSpeech is English speech recognition corpus
with 10,000 hours, while Multi-CN and WenetSpeech are Chinese datasets with
2,825 and 10,005 hours respectively.

###  3.2 Experimental Configuration

We explored various Conformer configures, with a major emphasis on attention
dimension, layer number and subsampling rate. The feed-forward dimension is
set to 2048.

On-the-fly approach is adopted for data preprocessing. Data augmentation is
applied to enrich the training data and no voice activity detection is
performed. Data augmentation involves additive noise, reverberation and speed
perturb. Additive noise from MUSAN [35] dataset is mixed with original signal,
the room impulse responses from RIRs [36] dataset is used to inject
reverberation via convolution operation. We extract 80-dimensional Mel-
filterbanks as acoustic feature. Then ASV models are trained on chunks of 300
randomly selected from whole feature map. Cepstral mean-normalization (CMN) is
applied before model training.

For ASV training, models are trained with AdamW optimizer with a total batch
of 512 on 4 Nvidia V100 GPUs. The learning rate increases to a peak during the
warmup stage and then decays to small as model converges. The total training
process lasts about 30 epochs. For ASR pre-training, we follow the recipes in
Wenet [18].

More training details can refer to ASV-Subtools[37].

###  3.3 Model evaluation

During the test, each utterance is chunked with about 300 frames. Embeddings
extracted from chunks are averaged to the final x-vector. For back-end, we
choose cosine similarity to score the extracted x-vectors. Evaluation
performance is measured by Equal Error Rate(EER) and minimum normalized
detection cost (minDCF).

From a practical standpoint, we evaluate the real-time-factor (RTF) of the
models on an Intel(R) Xeon(R) E5-2643 v4 CPU. The runtime is implemented based
on LibTorch so as to conveniently deploy Pytorch models for production
scenarios. Only one thread is used for CPU threading and TorchScript
inference.

##  4 Results and analysis

Table 1: Results of ASV Conformer on VoxCeleb & CN-Celeb. LSA means Length-
Scaled Attention. For the configure, 6L-256D-4H-2Sub means a Conformer with 6
layers (L) blocks, 256 attention dim (D), 4 attention heads (H), and 2-factor
subsampling (2Sub, default:4Sub)

Model | Configure | Prarams | RTF | VoxCeleb-O | VoxCeleb-E | VoxCeleb-H | CNC-Eval
---|---|---|---|---|---|---|---
EER(%) | EER(%) | EER(%) | EER(%) | $\text{minDCF}_{0.01}$
ECAPA-TDNN | C1024 | 16.0M | 0.071 | 0.925 | 1.231 | 2.321 | 9.45 | 0.5059
\+ asnorm[38] |  |  |  | 0.856 |  |  |  |
original paper[4] |  |  |  | 0.87 |  |  |  |
Conformer | 6L-256D-4H (w/o LSA) | 18.8M | 0.025 | 1.143 | 1.321 | 2.359 | 8.71 | 0.4747
| 6L-256D-4H |  |  | 1.026 | 1.280 | 2.278 | 8.39 | 0.4748
| 6L-256D-4H-2Sub | 22.5M | 0.070 | 0.915 | 1.177 | 2.034 | 8.30 | 0.4504
| 12L-256D-4H | 34.2M | 0.030 | 1.133 | 1.382 | 2.463 | 9.40 | 0.5195
| 6L-512D-8H | 46.4M | 0.080 | 1.356 | 1.492 | 2.571 | 8.38 | 0.4826

###  4.1 Results of Conformer on VoxCeleb and CN-Celeb

In this section, we generalize Conformer to ASV system. Table 1 presents the
performance of different networks on VoxCeleb and CN-Celeb. We reproduced the
1024 channels ECAPA-TDNN as baseline system. Score Normalization [38] (asnorm)
here is just for a fair comparison with original paper and the results suggest
the confidence of our baseline. For better compatibility with general ASR
encoder setting, the attention dimension and head have two types (i.e.,
256D-4H and 512D-8H). The efficacy of LSA is shown in the first two rows. LSA
automatically adapts attention weight distribution, and generalizes model to
various length inputs. Despite more parameters, the deeper 12 layers and wider
512D-8H networks fail to get further improvement, probably because 6L-256D-4H
has enough model capacity to model the size of VoxCeleb and CN-Celeb. More
complex structures are accompanied by harder convergence or overfitting.
Compared with 4-factor, 2-factor subsampling generates longer/richer feature
map before Conformer blocks, significantly improves performance, with the
trade of computation cost.

From the aspect of inference speed, due to the quadratic-complexity of self-
attention mechanism, 2-factor subsampling increases the RTF by
2$\times$-3$\times$ than 4-factor. The 512D also leads to about 4 times more
computation compared to 256D. At last, compared with ECAPA-TDNN,
6L-256D-4H-Sub with comparable model size and RTF shows ideal improvement in
EERs. While 6L-256D-4H achieves the remarkable relative reduction in RTF of
64.8%, with desirable performance sacrifice.

###  4.2 Effect of SAM training

When it comes to an overparameterized circumstance, the
6L-512D-8H-6Sub1116L-512D-8H-6Sub matches the open source pretrained ASR
model. in Table 2 obtains unsatisfactory performance on VoxCeleb-O. A large
regularization of 0.2 alleviates model overfitting, and a comparable efficacy
can be achieved through SAM training. What’s more, SAM training combined with
appropriate regularization further improves the model generalization.

To investigate the effective of ASR transferring, we downloaded the available
GigaSpeech ASR checkpoint [39] as a pretrained encoder. During training, we
observed that the model transferred from the ASR encoder yields a lower value
of training loss than training from scratch. However, the EER on test set
increases from 1.431% to 1.521% as shown in the fifth and last line of Table
2. It seems that prior ASR classification increases the risk of falling into a
sharp minima on the travel from ASR to ASV, leading to suboptimal quality. SAM
aims to seek out parameters whose entire neighborhoods have both low loss
value, preventing model from dramatic shaking by inputs perturbation. Thus, we
attempted to introduce SAM training to provide robustness. As a result, SAM
apparently improved model generalization as the EER value decreases from
1.521% to 1.218%. Compared with standard ASV Conformer, ASR tranferring
achieved 11.22% relative reduction in EER under the same training procedure.
Note that we didn’t pre-train a 256D-4H ASR model on GigaSpeech, and ASR
transferring is further discussed in sec 4.3.

Table 2: Effect of SAM training. ASR transfer means transferred from an ASR encoder trained on GigaSpeech. Model | SAM | weight decay | VoxCeleb-O
---|---|---|---
EER(%)
6L-256D-4H | ✗ | 0.05 | 1.026
| ✓ | 0.05 | 0.983
6L-512D-8H-6Sub | ✗ | 0.05 | 1.771
| ✓ | 0.05 | 1.459
| ✗ | 0.2 | 1.431
| ✓ | 0.2 | 1.372
\+ ASR transfer | ✓ | 0.2 | 1.218
| ✗ | 0.2 | 1.521

###  4.3 Effect of ASR transferring

Experiments conducted on CN-Celeb further reports the effect of ASR
transferring. All transferred models are applied SAM. The 6L-256D-4H is chosen
as baseline system. As shown in Table 3, Conformers of all different
configurations benefit greatly from the ASR pretrained encoder, proving that
our proposed method is effective enough for integrating ASR information. As
WenetSpeech is much larger than Multi-CN, a more robust ASR encoder
contributes better performance. When applying full ASR encoder to ASV system,
the EER is improved by -1.582% but the minDCF is degraded by +0.0617. This
result might be due to the rear layers of the ASR encoder being too biaed
towards ASR classification, making it more difficult for the network to learn
speaker representation. Compared with training from scratch, transferring 6
blocks from the WenetSpeech ASR encoder achieves 11.56% relative reduction in
EER and 6.76% relative reduction in minDCF respectively on CNC-Eval, revealing
the capabilities of Conformer to model ASR and ASV tasks.

Table 3: Performance of ASR transferring on CN-Celeb. Configure | pretrain ASR | CNC-Eval
---|---|---
EER(%) | $\text{minDCF}_{0.01}$
6L-256D-4H | — | 8.39 | 0.4748
| Multi-CN | 7.95 | 0.4534
| WenetSpeech | 7.42 | 0.4427
12L-256D-4H | — | 9.40 | 0.5195
| WenetSpeech | 7.82 | 0.5812
6L-512D-8H | — | 8.38 | 0.4826
| WenetSpeech | 7.83 | 0.4551

##  5 Conclusions

To bridge the gap between ASR and ASV, we concentrate on applying a unified
architecture Conformer in ASV system. First of all, we adopt LSA and SAM
training on ASV Conformer system to improve the model generalization ability.
LSA enables model to generate to variable lengths inputs and SAM seeks a
flatter loss landscape, proved to be effective especially when the model is
severely overfitted. Experiments conducted on Voxceleb & CN-Celeb indicated
that the Conformer is well-suited for ASV task, and achieved competitive
performance compared with the popular ECAPA-TDNN. Moreover, a simply ASR
transferring method is introduced to integrate ASR information. ASR
transferring outperformed standard ASV Conformer, gave a relative improvement
of about 11% in EER on both VoxCeleb and CN-Celeb. Unfortunately, the
underlying relationship between ASR and ASV remains unclear and should be
further investigated in future. In addition, we provide a runtime to better
deploy models for production. The RTF of ASV Conformer evaluated on runtime
environment verified its industrial value. Last but not least, this work
leaves plenty room to extend advanced exploration of attention-based
technology to ASV system, which will benefit ASV tasks both in industry and
academia.

##  6 Acknowledgements

This work was funded by the National Natural Science Foundation of China
(Grant No.61876160 and No.62001405) and in part by the Science and Technology
Key Project of Fujian Province, China (Grant No. 2020HZ020005).

## References

  * [1] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-Vectors: Robust DNN Embeddings for Speaker Recognition,” in _Proc. IEEE ICASSP_ , 2018.

  * [2] H. Zeinali, S. Wang, A. Silnova, P. Matějka, and O. Plchot, “BUT System Description to VoxCeleb Speaker Recognition Challenge 2019,” _arXiv preprint arXiv:1910.12592_ , 2019.
  * [3] T. Zhou, Y. Zhao, and J. Wu, “Resnext and res2net structures for speaker verification,” in _Proc. IEEE SLT_ , 2021.
  * [4] B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” in _Proc. Interspeech_ , 2020.
  * [5] M. Zhao, Y. Ma, M. Liu, and M. Xu, “The speakin system for voxceleb speaker recognition challange 2021,” _arXiv preprint arXiv:2109.01989_ , 2021.
  * [6] A. Vaswani, N. Shazeer, N. Parmar _et al._ , “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _Proc. ICLR_ , 2021.
  * [9] P. Safari, M. India, and J. Hernando, “Self-attention encoding and pooling for speaker recognition,” in _Proc. Interspeech_ , 2020.
  * [10] B. Han, Z. Chen, and Y. Qian, “Local information modeling with self-attention for speaker verification,” in _Proc. IEEE ICASSP_ , 2022.
  * [11] A. Gulati, J. Qin, C.-C. Chiu _et al._ , “Conformer: Convolution-augmented Transformer for Speech Recognition,” in _Proc. Interspeech_ , 2020.
  * [12] Y. Zhang, Z. Lv, H. Wu _et al._ , “Mfa-conformer: Multi-scale feature aggregation conformer for automatic speaker verification,” _arXiv preprint arXiv:2203.15249_ , 2022.
  * [13] S. Wang, J. Rohdin, L. Burget _et al._ , “On the usage of phonetic information for text-independent speaker embedding extraction,” in _Proc. Interspeech_ , 2019.
  * [14] S. Sigtia, E. Marchi, S. Kajarekar _et al._ , “Multi-task learning for speaker verification and voice trigger detection,” in _Proc. IEEE ICASSP_ , 2020.
  * [15] D. Liao, J. Li, Y. Zhi, S. Li, Q. Hong, and L. Li, “An integrated framework for two-pass personalized voice trigger,” in _Proc. Interspeech_ , 2021.
  * [16] H. Yu, J. Zhao, S. Yang _et al._ , “Language recognition based on unsupervised pretrained models,” in _Proc. Interspeech_ , 2021.
  * [17] A. Lyu, Z. Wang, and H. Zhu, “Ant multilingual recognition system for olr 2021 challenge,” in _Proc. Interspeech_ , 2022.
  * [18] B. Zhang, D. Wu, Z. Peng _et al._ , “Wenet 2.0: More productive end-to-end speech recognition toolkit,” _arXiv preprint arXiv:2203.15455_ , 2022.
  * [19] B. Wang, W. Hu, J. Li _et al._ , “Olr 2021 challenge: Datasets, rules and baselines,” in _Proc. APSIPA ASC_ , 2021.
  * [20] D. Chiang and P. Cholak, “Overcoming a theoretical limitation of self-attention,” _arXiv preprint arXiv:2202.12172_ , 2022.
  * [21] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in _Proc. ICLR_ , 2021.
  * [22] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” in _Proc. Interspeech_ , 2017.
  * [23] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” in _Proc. Interspeech_ , 2018.
  * [24] Y. Fan, J. Kang, L. Li _et al._ , “CN-CELEB: a challenging Chinese speaker recognition dataset,” in _Proc. IEEE ICASSP_ , 2020.
  * [25] L. Li, R. Liu, J. Kang _et al._ , “CN-Celeb: Multi-genre speaker recognition,” _Speech Communication_ , vol. 137, 2022.
  * [26] J. Su, Y. Lu, S. Pan _et al._ , “Roformer: Enhanced transformer with rotary position embedding,” _arXiv preprint arXiv:2104.09864_ , 2022.
  * [27] K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive statistics pooling for deep speaker embedding,” in _Proc. Interspeech_ , 2018.
  * [28] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proc. IEEE CVPR_ , 2019.
  * [29] J. Su, “Scale operation of attention from entropy invariance,” 2022. [Online]. Available: https://spaces.ac.cn/archives/8823
  * [30] N. Keskar, D. Mudigere, J. Nocedal _et al._ , “On large-batch training for deep learning: Generalization gap and sharp minima,” _arXiv preprint arXiv:1609.04836_ , 2016.
  * [31] Y. Jiang, B. Neyshabur, H. Mobahi _et al._ , “Fantastic generalization measures and where to find them,” _arXiv preprint arXiv:1912.02178_ , 2019\.
  * [32] G. Chen, S. Chai, G. Wang _et al._ , “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” in _Proc. Interspeech_ , 2021.
  * [33] https://github.com/wenet-e2e/wenet/blob/main/examples/multi_cn/s0/README.md.
  * [34] B. Zhang, H. Lv, P. Guo _et al._ , “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,” in _Proc. IEEE ICASSP_ , 2022.
  * [35] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” _arXiv preprint arXiv:1510.08484_ , 2015.
  * [36] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _Proc. IEEE ICASSP_ , 2017.
  * [37] F. Tong, M. Zhao, J. Zhou _et al._ , “ASV-Subtools: Open source toolkit for automatic speaker verification,” in _Proc. IEEE ICASSP_ , 2021\.
  * [38] P. Matejka, O. Novotný, O. Plchot _et al._ , “Analysis of score normalization in multilingual speaker recognition,” in _Proc. Interspeech_ , 2017.
  * [39] https://github.com/wenet-e2e/wenet/blob/main/docs/pretrained_models.en.md.


matrix element is reduced to the parameter $\gamma_{1}$.

$\left(\begin{matrix}\omega_{1}+v_{1}k-i(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{1+}}\\\
\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\kappa_{1}e^{i\phi}+\beta_{1}\kappa_{2}e^{-i\phi}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}-i(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{1-}}\\\
\kappa_{d}-ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{2+}}\\\
\kappa_{c}-ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{2-}}\end{matrix}\right).$
(72)

The matrix element in red accounts for the diffractive coupling between
$a_{1+}$ and $a_{1-}$ (optical field transfer from $a_{1+}$ to $a_{1-}$) at
the 2nd Brillouin zone boundary induced by the corrugation of grating 1
(factor $\kappa_{1}$) and by the corrugation of grating 2 (factor
$\kappa_{2}$, weighted by the parameter $\beta_{1}$). The coefficients $e^{\pm
i\phi}$ correspond to the 2nd-order diffraction phase-shift at the 2nd
Brillouin zone boundary between the two gratings due to the lateral offset
$\delta$.

$\left(\begin{matrix}\omega_{1}+v_{1}k-i(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{1+}}\\\
\kappa_{1}e^{i\phi}+\beta_{1}\kappa_{2}e^{-i\phi}-i\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{1-}}\\\
\kappa_{d}-ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{2+}}\\\
\kappa_{c}-ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{2-}}\end{matrix}\right).$
(73)

The matrix element in red expresses the optical field transfer from $a_{1+}$
to $a_{1-}$, mediated by the $1^{st}$ order diffraction processes at the
$\Gamma$ point occurring between the guided waves $a_{1+}$ and $a_{1-}$, via
the radiation continuum. This transfer takes place in two successive steps, as
illustrated in the figure below. The 1st step is identical to the previous
case; for the $2^{nd}$ step, insertion of the radiated light in the backward
direction is accounted for in the different phase parameters.

Figure 22: $a_{1+}/a_{1-}$ radiative coupling

$\left(\begin{matrix}\omega_{1}+v_{1}k-i(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{1+}}\\\
\kappa_{1}e^{i\phi}+\beta_{1}\kappa_{2}e^{-i\phi}-i(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{1-}}\\\
\kappa_{d}-\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{2+}}\\\
\kappa_{c}-ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{2-}}\end{matrix}\right).$
(74)

The matrix element in red expresses the optical field transfer from $a_{1+}$
to $a_{2+}$, mediated by the $1^{st}$ order diffraction processes at the
$\Gamma$ point occurring between the guided waves $a_{1+}$ and $a_{2+}$, via
the radiation continuum. This transfer takes place in three successive steps,
as illustrated in the figure below. The $1^{st}$ step is identical to the
previous cases; the $2^{nd}$ step consists in the radiative flying of plane
waves from grating 1 to grating 2, resulting in the phase shift $\psi$ ; the
$3^{rd}$ step corresponds to light insertion in grating 2 in the forward
direction.

Figure 23:  $a_{1+}/a_{2+}$ radiative coupling

$\left(\begin{matrix}\omega_{1}+v_{1}k-i(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{1+}}\\\
\kappa_{1}e^{i\phi}+\beta_{1}\kappa_{2}e^{-i\phi}-i(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{1-}}\\\
\kappa_{d}-ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{-i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{i\frac{\phi}{2}})e^{-i\varphi_{2+}}\\\
\kappa_{c}-\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}ie^{i\psi}(\sqrt{\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\beta_{1}\gamma_{2}}e^{-i\frac{\phi}{2}})e^{i\varphi_{1+}}(\sqrt{\beta_{2}\gamma_{1}}e^{i\frac{\phi}{2}}+\sqrt{\gamma_{2}}e^{-i\frac{\phi}{2}})e^{-i\varphi_{2-}}\end{matrix}\right).$
(75)

The matrix element in red expresses the optical field transfer from $a_{1+}$
to $a_{2-}$, mediated by the $1^{st}$ order diffraction processes at the
$\Gamma$ point occurring between the guided waves $a_{1+}$ and $a_{2-}$, via
the radiation continuum. This transfer takes place also in three steps and is
similar to the previous case; the only difference concerns the $3^{rd}$ step,
where light insertion in grating 2 occurs along in the backward direction,
which is accounted for with the various phase parameters.

##  Appendix C Fully symmetrical structures

###  C.1 Eigenvalues

Eigenvalues can be obtained from diagonalization of the Hamiltonian given in
section III.1 and expressed in the base formed by vectors
$(a_{1+},a_{1-},a_{2+},a_{2-})$. The Hamiltonian can be rewritten in a new
base formed by even and odd (along the transverse direction), forward and
backward vectors $(a_{e+},a_{e-},a_{o+},a_{o-})$ with
$a_{e\pm}=\frac{1}{\sqrt{2}}(a_{1\pm}+a_{2\pm})$ and
$a_{o\pm}=\frac{1}{\sqrt{2}}(a_{1\pm}-a_{2\pm})$. Since the eigenmodes are
also even and odd (the coupled grating being symmetrical transversally), this
writing of equations results in a new Hamitonien which is a priori both more
physically readable and mathematically tractable. The new 4×4 Hamiltonien is
as below,

$H=\left(\begin{matrix}\omega_{0}+vk+\kappa_{d}-iG(1+e^{i\psi})&K+\kappa_{c}-iG(1+e^{i\psi})&0&0\\\
K+\kappa_{c}-iG(1+e^{i\psi})&\omega_{0}-vk+\kappa_{d}-iG(1+e^{i\psi})&0&0\\\
0&0&\omega_{0}+vk-\kappa_{d}-iG(1-e^{i\psi})&K-\kappa_{c}-iG(1-e^{i\psi})\\\
0&0&K-\kappa_{c}-iG(1-e^{i\psi})&\omega_{0}-vk-\kappa_{d}-iG(1-e^{i\psi})\end{matrix}\right),$
(76)

and can be rewritten as :

$H=\left(\begin{matrix}H_{e}&0\\\ 0&H_{o}\end{matrix}\right),$ (77)

where $H_{e}$, $H_{o}$ are 2×2 Hamiltonians whose diagonalization provides,
separately, the couples of even and odd eigenvalues. It can be easily shown
that for both couples of even and odd eigenvalues, one eigenvalue is
systematically dark (called lateral BIC in the main text) at the $\Gamma$
point ($k=0$), while the other is bright in general except when
$\psi=\pi(0)mod2\pi$, for the even (odd) mode (in the transverse direction).
The systematic observation of a lateral BIC at the $\Gamma$ point features
structures with lateral symmetry as commented in the main text.

###  C.2 Expansion of complex eigenvalues in the vicinity of the $\Gamma$
point

Results of the expansion of complex eigenvalues of even modes (Eq. (4) with
$\eta=1$) are given below:

  * •

Bright branch of the dispersion characteristics:

$\omega=\omega_{R}+\kappa_{d}+K+\kappa_{c}+2G\sin\psi_{0}-2iG(1+\cos\psi_{0})+C\frac{k^{2}}{2}.$
(78)

$C=C_{R}+iC_{I}$ is the complex curvature (second derivative versus $k$) of
the complex eigenvalue.

$C_{R}=\frac{d^{2}\omega_{R}}{dk^{2}}=\frac{\frac{v^{2}}{|K+\kappa_{c}-U|^{2}}(K+\kappa_{c}+G\sin\psi_{0})-\cos\psi_{0}\frac{\tau_{rad}}{\tau_{wg}}C_{rad}}{1-\cos\psi_{0}\frac{\tau_{rad}}{\tau_{wg}}},$
(79)
$C_{I}=\frac{d^{2}\omega_{I}}{dk^{2}}=\frac{v^{2}}{|K+\kappa_{c}-U|^{2}}G(1+\cos\psi_{0})-\sin\psi_{0}\frac{\tau_{rad}}{\tau_{wg}}\left(C_{rad}-\frac{d^{2}\omega_{R}}{dk^{2}}\right),$
(80)

with $U=iG(1+e^{i\psi_{0}})$ , $\psi_{0}=\psi(k=0)$ and where
$\tau_{wg}=\frac{1}{2G}$ and $\tau_{rad}=\frac{L_{opt}}{c}$, are respectively
the average lifetime of photons in the wave-guided state before being emitted
into the continuum, and in the radiated state during a one way trip between
the two gratings. $C_{rad}=\frac{(\frac{c}{n})^{2}}{\omega_{R0}}$ , is named
the radiated curvature: it is identical to the curvature of the dispersion
characteristic of a Fabry Perot cavity, with n intra-cavity optical index and
with perfect metallic reflectors, at the $\Gamma$ point around the resonance
frequency $\omega_{R0}$.

  * •

Dark branch (with lateral BIC at the $\Gamma$ point) of the dispersion
characteristics:

$\omega=\omega_{R}+i\omega_{i}\cong\omega_{0}+\kappa_{d}-K-\kappa_{c}+C\frac{k^{2}}{2},$
(81)
$C_{R}=\frac{d^{2}\omega_{R}}{dk^{2}}=\frac{-v^{2}}{|K+\kappa_{c}-U|^{2}}(K+\kappa_{c}+G\sin\psi_{0}),$
(82)
$C_{I}=\frac{d^{2}\omega_{I}}{dk^{2}}=\frac{-v^{2}}{|K+\kappa_{c}-U|^{2}}G(1+\cos\psi_{0}).$
(83)

As explained in section III.3.1 and expressed by Eqs. (78), (81) and (82), the
real eigenvalues of the bright and the dark branches are degenerated at the
$\Gamma$ point when $K+\kappa_{c}+G\sin\psi_{0}=0$, and the real dark branch
is flat (zero real curvature).

###  C.3 Imaginary dispersion characteristic around the wave vector $k_{BIC}$
of a simple transverse BIC; BIC merging processes: double transverse BIC from
merging of 2 simple transverse BICs and triple BIC from merging of two simple
transverse BICs and a lateral BIC

Expression of the imaginary dispersion characteristic around the wave-vector
$k=k_{BIC}$ can be derived from expansion of Eqs. (4) and (7). It is given by
the relation below, for the even mode:

$\omega_{I}=-\frac{G}{2}\left(1\pm\frac{K+\kappa_{c}}{\sqrt{v^{2}k_{BIC}^{2}+(K+\kappa_{c})^{2}}}\right)\left[\frac{\frac{L_{opt}^{2}}{(2p+1)\pi}\frac{k_{BIC}}{n^{2}}\left(1\pm\frac{\omega_{RBIC}v^{2}}{\frac{c^{2}}{n^{2}}\sqrt{v^{2}k_{BIC}^{2}+(K+\kappa_{c})^{2}}}\right)}{1+\frac{L_{opt}^{2}}{(2p+1)\pi}\frac{\omega_{RBIC}}{c^{2}}G\left(1\pm\frac{K+\kappa_{c}}{\sqrt{v^{2}k_{BIC}^{2}+(K+\kappa_{c})^{2}}}\right)}\right]^{2}\left(k-k_{BIC}\right)^{2},$
(84)

with:

$L_{opt}\sqrt{\left(\frac{\omega_{RBIC}}{c}\right)^{2}-\left(\frac{k_{BIC}}{n}\right)^{2}}=(2p+1)\pi.$
(85)

The sign + (-) corresponds to the air (dielectric) branch. If $K$>0 the air
(dielectric) branch is bright (dark) at the $\Gamma$ point, and the other way
around if $K$<0\. The second derivative, or the curvature, of the imaginary
dispersion characteristic is written as:

$\frac{d^{2}\omega_{I}}{dk^{2}}|_{(k=k_{BIC})}\cong-\frac{G}{2}\left(1\pm\frac{K+\kappa_{c}}{\sqrt{v^{2}k_{BIC}^{2}+(K+\kappa_{c})^{2}}}\right)\left[\frac{\frac{L_{opt}^{2}}{(2p+1)\pi}\frac{k_{BIC}}{n^{2}}\left(1\pm\frac{\omega_{RBIC}v^{2}}{\frac{c^{2}}{n^{2}}\sqrt{v^{2}k_{BIC}^{2}+(K+\kappa_{c})^{2}}}\right)}{1+\frac{L_{opt}^{2}}{(2p+1)\pi}\frac{\omega_{RBIC}}{c^{2}}G\left(1\pm\frac{K+\kappa_{c}}{\sqrt{v^{2}k_{BIC}^{2}+(K+\kappa_{c})^{2}}}\right)}\right]^{2}.$
(86)

When the transverse BIC approaches the $\Gamma$ point, $k_{BIC}$ tends to zero
and $\omega_{RBIC}$ tends to $\omega_{R0}$ , with
$\frac{\omega_{R0}}{c}L_{opt}=(2p+1)\pi$ ; the curvature is then simply
written as below (for example when the bright branch is air like, that is for
$K$>0):

  * •

For the bright branch:

$\frac{d^{2}\omega_{I}}{dk^{2}}\bigg{|}_{k=k_{BIC}}\cong-2G\left[\frac{\frac{L_{opt}}{n^{2}\frac{\omega_{R0}}{c}}\left(1+\frac{\omega_{R0}v^{2}}{\frac{c^{2}}{n^{2}}}(K+\kappa_{c})\right)}{1+\frac{2L_{opt}}{c}G}\right]^{2}k_{BIC}^{2}.$
(87)

The curvature varies like $k_{BIC}^{2}$ : this behaviour is a manifestation of
the merging of two ordinary transverse BICs (belonging to the bright branch
and occurring at $\pm|k_{BIC}|$ vectors), when $k_{BIC}$ approaches 0. We call
this particular transverse BIC occurring at the $\Gamma$ point, double
transverse BIC, which exhibits a flat imaginary dispersion characteristic. As
a matter of fact, the $k_{BIC}^{2}$ dependence of the curvature expresses that
the imaginary dispersion characteristic is like $k^{4}$ around the $\Gamma$
point.

  * •

For the dark branch:

$\frac{d^{2}\omega_{I}}{dk^{2}}\bigg{|}_{k=k_{BIC}}\cong-\frac{G}{2}\left[\frac{L_{opt}v}{n^{2}\frac{\omega_{R0}}{c}(K+\kappa_{c})}\left(1-\frac{\omega_{R0}v^{2}}{\frac{c^{2}}{n^{2}}}(K+\kappa_{c})\right)\right]^{2}k_{BIC}^{4}.$
(88)

The curvature varies like $k_{BIC}^{4}$ : this behaviour is a manifestation of
the merging of two ordinary transverse BICs (belonging to the dark branch and
occurring at $\pm|k_{BIC}|$ vectors) with the lateral BIC (systematically
occurring at the $\Gamma$ point of a dark branch), when $k_{BIC}$ approaches
0. We call this particular transverse BIC, triple transverse BIC, which
exhibits a very flat imaginary dispersion characteristic. As a matter of fact,
the $k_{BIC}^{4}$ dependence of the curvature expresses that the imaginary
dispersion characteristic is like $k^{6}$ around the $\Gamma$ point.

###  C.4 Double Exceptional point

A double exceptional point is formed when conditions for full degeneracy of
the complex eigenvalues of the dark and bright branches is achieved [27] (see
section III.3.1). In the case of the even branches, Eq. (4) leads to the
following eigenvalues:

$\omega=\omega_{R}+i\omega_{i}=\omega_{0}+\kappa_{d}-iG(1+e^{i\psi})\pm\sqrt{v^{2}k^{2}+\left(K+\kappa_{c}-iG(1+e^{i\psi})\right)^{2}}.$
(89)

Degeneracy of the complex eigenvalues occurs when :

$v^{2}k_{EP}^{2}+\left(K+\kappa_{c}-iG(1+e^{i\psi(k_{EP})})\right)^{2}=0,$
(90)

where $k=k_{EP}$ is the wave-vector at the exceptional points. It results:

$k_{EP}=\pm\frac{G\left(1+\cos{\psi(k_{EP})}\right)}{v},$ (91)

with the condition:

$K+\kappa_{c}+G\sin{\psi(k_{EP})}=0,$ (92)

being to be held in addition. Note that the previous condition can be met
provided that $0\leq|K+\kappa_{c}|\leq G$. This can be adjusted, in practice,
by an appropriate setting of the filling factor of the grating structure.
$\omega_{R}(k_{EP})$, $\omega_{I}(k_{EP})$ and $k_{EP}$ can be derived from
Eqs. (4), (91) and (92).

Finally, it is reminded (see Eq. (7)):

$\psi(k_{EP})=\sqrt{\left[\frac{\omega(k_{EP})}{c}\right]^{2}-\left(\frac{k_{EP}}{n}\right)^{2}}L_{opt},$
(93)

which is an additional condition for the double exceptional to be achieved.
This can be adjusted in practice by an appropriate setting of $L_{opt}$, that
is of the thickness of the grating structure. Interestingly, for a given
structure showing a double exceptional point, the real eigenvalues of the dark
branch at the $\Gamma$ point (lateral BIC) and at the exceptional points
coincide. Indeed, for the lateral BIC:

$\omega_{R}(k=0)=\omega_{0}+\kappa_{d}-K-\kappa_{c}=\omega_{0}+\kappa_{d}+G\sin{\psi(k_{EP})}=\omega_{R}(k_{EP}).$
(94)

Finally, for wave-vector $k$ exceedind $k_{EP}$, the real dispersion
characteristics of the dark and bright branches are linear $(\pm vk)$ and both
branches equally share the loss rate, while, at the $\Gamma$ point, the bright
branch takes the full part of the losses.

###  C.5 Double transverse BIC with flat real dispersion characteristic

It is reminded that the second derivative $\frac{d^{2}\omega_{R}}{dk^{2}}$ or
curvature $C_{R}$ of the real dispersion characteristic at the $\Gamma$ point
can be written as (see Eq. (11)):

$C_{R}=\frac{\frac{1}{2G}\frac{v^{2}}{K+\kappa_{c}}+\frac{L_{opt}}{c}\frac{\left(\frac{c}{n}\right)^{2}}{\omega_{R0}}}{\frac{1}{2G}+\frac{L_{opt}}{c}}=\frac{\tau_{wg}\times
C_{wg}+\tau_{rad}\times C_{rad}}{\tau_{wg}+\tau_{rad}},$ (95)

where $C_{wg}=\frac{v^{2}}{K+\kappa_{c}}$ and
$C_{rad}=\frac{\left(\frac{c}{n}\right)^{2}}{\omega_{R0}}$ are the guided and
radiated curvatures respectively. The radiated curvature is strictly positive,
while the guided curvature may be positive or negative, as the parameter
$K+\kappa_{c}$. These two cases have been illustrated in figure 5, where the
bright branch was shown to be either air like (positive curvature) or
dielectric like (negative curvature), depending on the sign of
$K+\kappa_{c}+G\sin{\psi_{0}}$, where $\psi_{0}=\psi(k=0)$. In the current
discussion, the bright branch turns to be a transverse BIC at the $\Gamma$
point, therefore $\psi_{0}=\pi$ and $K+\kappa_{c}+G\sin{\psi_{0}}\equiv
K+\kappa_{c}$. For $K+\kappa_{c}<0$, or $C_{wg}<0$, it is possible in
principle to meet the condition $C_{R}=0$. Given that $v\cong\frac{c}{n}$, and
that $G$ and $|K+\kappa_{c}|$ are in practice lower than
$\omega_{R0}=\frac{2\pi c}{\lambda}$, the condition for a zero real curvature
implies that $L_{opt}$ is significantly larger than $\lambda$. In other words,
the radiated curvature $C_{rad}$ is rather small as compared to the guided
curvature $C_{wg}$, and the former has to be over-weighted by a large
$\tau_{r}ad$ (that is large $L_{opt}$) with respect to the later in order to
result in a total curvature $C_{R}=0$.

###  C.6 Design rules of a Dirac point at a triple BIC

Two conditions have to be met simultaneously for the formation of a Dirac
point at a triple BIC: first, the condition for degeneracy at the $\Gamma$
point of the frequency of the lateral BIC with the real part of the bright
eigenfrequency at the $\Gamma$ point, which occurs when
$K+\kappa_{c}+G\sin{\psi_{0}}=0$ ; second, the condition for the bright mode
to turn into a double transverse BIC, when $\psi_{0}=\pi(mod2\pi)$ (see
section III.3.2). For a given index contrast between the gratings and the
surrounding medium, the high index material filling factor (FF) and the
effective optical thickness of the coupled grating structure are the two
available “joysticks” to be handled simultaneously to achieve those two
conditions. The degeneracy condition, which corresponds to the cancellation of
the overall diffractive coupling processes between wave-guided resonances, can
be viewed as the physical counterpart of the full transmission condition which
occurs in a half wavelength Bragg mirror stack; this condition is met when the
optical thicknesses of the high / low index layers are an integer of
$\frac{\lambda}{2}$. For example this is simply achieved for a high index
$FF=\frac{n_{low}}{n_{low}+n_{high}}$. For the wave-guided coupled grating
structure, this condition is met for a different value of FF; for example,
$FF<\frac{n_{low}}{n_{low}+n_{high}}$ when the two gratings are in contact.
This is due to the fact that the phase change occurring at the reflection /
transmission of the guided wave impinging the high / low index interface of
the grating is different from 0 or $\pi$ $(mod2\pi)$, unlike the case of plane
waves in a Bragg mirror. For a practical design, it is advisable to start with
the value of FF which applies to a Bragg mirror stack (for example 0.25, if
$n_{high}=3$ and $n_{low}=1$) and then adjust it, to compensate for the phase
change difference and therefore, approach the condition requested for
degeneracy between the lateral BIC and the bright mode frequency. Then, in
order to convert the bright mode into a transverse double BIC, the effective
optical thickness of the coupled grating structure must be adjusted: this can
be achieved by adjusting the thickness $H$ of each grating and / or the
distance D between them. A few iterations of those successive adjustments of
FF and $h$, $D$ may be necessary to get close to the formation of the elusive
perfect Dirac point at a triple BIC.

##  Appendix D Eigenvalues in structures with broken lateral symmetry

Eigenvalues can be obtained from diagonalization of the Hamiltonian given in
II.1 and expressed in the base formed by base vectors
$(a_{1+},a_{1-},a_{2+},a_{2-})$. The Hamiltonian can rewritten in a new base
formed by even and odd (along the transverse direction), forward and backward
base vectors $(a_{e+},a_{e-},a_{o+},a_{o-})$, in the same way as for fully
symmetrical structures (see section C.1), given that the symmetry along the
transverse direction is maintained. The new 4×4 Hamiltonien is as below:

$H=\left(\begin{matrix}\omega_{0}+vk+\kappa_{d}-iG(1+e^{i\psi})&K+\kappa_{c}-iG(1+e^{i\psi})e^{-i\varphi}&0&0\\\
K+\kappa_{c}-iG(1+e^{i\psi})e^{i\varphi}&\omega_{0}-vk-iG(1+e^{i\psi})&0&0\\\
0&0&\omega_{0}+vk-\kappa_{d}-iG(1-e^{i\psi})&K-\kappa_{c}-iG(1-e^{i\psi})e^{-i\varphi}\\\
0&0&K-\kappa_{c}-iG(1-e^{i\psi})e^{i\varphi}&\omega_{0}-vk-\kappa_{d}-iG(1-e^{i\psi})\end{matrix}\right),$
(96)

and can be rewritten as :

$H=\left(\begin{matrix}H_{e}&0\\\ 0&H_{o}\end{matrix}\right),$ (97)

where $H_{e}$, $H_{o}$ are 2×2 Hamiltonians whose diagonalization provides,
separately, the couples of even and odd eigenvalues. The two $H_{e}$, $H_{o}$
are fully bright, and do not result in any dark branch showing a lateral BIC
at the $\Gamma$ point, unlike the case of fully symmetrical structures. This
is a signature of the broken lateral symmetry, which expresses it-self by the
phase $\varphi\neq 0$ (mod$\pi$).

##  Appendix E Quasi transverse BICs at the $\Gamma$ point in transverse
symmetry broken structures: the case of negligible near-field coupling between
the gratings

In section V.1, we show that, at $\Gamma$ point, 2 dark and 2 bright modes are
obtained. We conclude in section V.3 that transverse BIC cannot be formed at
the $\Gamma$ point in transverse symmetry broken structures as a result of the
finite phase $\varphi$. However, if the near field evanescent coupling is
considered as negligible when the two gratings are far apart, then, in Eq.
(23), $K_{+}\cong 0$ and the 2x2 hamiltonian for the bright modes at the
$\Gamma$ point is given by:

$H=\left(\begin{matrix}\omega_{1}+K_{12}-iG_{12}&-i\sqrt{G_{12}G_{21}}e^{i(\psi-\varphi)}\\\
-i\sqrt{G_{12}G_{21}}e^{i(\psi+\varphi)}&\omega_{2}+K_{21}-iG_{21}\end{matrix}\right).$
(98)

It clearly results that the impact of the phase $\varphi$ is also negligible
when it comes to diagonalize the 2×2 Hamiltonian of the bright modes.
Therefore, the two corresponding eigenvalues can be made real when the phase
$\psi$ is set at 0 (mod $\pi$). This is indeed possible provided that
$\omega_{1}=\omega_{2}$. Then the 2 bright modes of the Hamiltonian turn into
2 quasi-BICS, but are not true BICS.

##  Appendix F "Fishbone" structure

###  F.1 Aligned and half-period misalignment structures

The Hamiltonien given in section VI.1, is rewritten along the more compact
version below:

$H=\left(\begin{matrix}A+vk&B_{2}&C_{2}&D\\\ B_{1}&A-vk&D&C_{1}\\\
C_{1}&D&A+vk&B_{1}\\\ D&C_{2}&B_{2}&A-vk\end{matrix}\right).$ (99)

Note that for aligned grating structures ($\phi=0$ (mod$2\pi$)) or with half
period lateral off-set ($\phi=\pi$ (mod$2\pi$)), $B_{1}=B_{2}$ and
$C_{1}=C_{2}$, which means that the compact expressions of the Hamiltonian are
formally identical. Therefore for this two limit cases, the properties of the
eigenvalues are also formally identical. For half-period lateral off-set, we
obtain the two couple of complex eigenvalues below:

_Fundamental modes:_

$\omega=\omega_{R}+i\omega_{i}=\omega_{0}+\kappa_{d}-iG(1-e^{i\psi})\pm\sqrt{v^{2}k^{2}+\left(K+\kappa_{c}+i|G|(1-e^{i\psi})\right)^{2}},$
(100)

_Excited modes:_

$\omega=\omega_{R}+i\omega_{i}=\omega_{0}-\kappa_{d}-iG(1+e^{i\psi})\pm\sqrt{v^{2}k^{2}+\left(K-\kappa_{c}+i|G|(1+e^{i\psi})\right)^{2}},$
(101)

with: $K=-\kappa(1+\beta)$, $G=\gamma(1-\sqrt{\beta})^{2}$.

###  F.2 Structures with arbitrary misalignment: expansion of complex
eigenvalues in the vicinity of the $\Gamma$ point

In general, complex eigenvalues $\omega$ are obtained from the diagonalization
of the Hamiltonian given by Eq. 28. The corresponding secular equation can be
written in the compact form:

$|H-\omega\mathbb{1}_{4}|=\left|\left(\begin{matrix}H_{B+}&H_{V}\\\
H_{V}&H_{B-}\end{matrix}\right)-\omega\mathbb{1}_{4}\right|,$ (102)

where $H_{B\pm}$ and $H_{V}$ are 2x2 matrices. At the $\Gamma$ point, where
$H_{V}=0$, the eigenvalues are the solution of 2 independent equations:

$|H_{B_{\pm}}-\omega\mathbb{1}_{2}|=0.$ (103)

Complex in general, the eigenvalues become real if $\psi=0$ or $\pi$ and are
the signatures of transverse BICs (see section VI.3). From expansion of Eq.
(102), it is possible to derive an analytical expression of the curvature
$C_{R}$ of the dispersion characteristics at the $\Gamma$ point. Considering a
transverse BIC at the $\Gamma$ point in the vicinity of the eigenvalue
numbered "3", corresponding with Eq. (31) (section VI.3), which is appropriate
for the production of flat BIC, we get:

$C_{R}=\frac{\frac{1}{(1+\alpha)|G|}\frac{v^{2}}{K_{eff}}+\frac{L_{opt}}{c}\frac{\left(\frac{c}{n}\right)^{2}}{\omega_{R0}}}{\frac{1}{(1+\alpha)|G|}+\frac{L_{opt}}{c}},$
(104)

where:

$\alpha=\frac{\Re(\frac{\kappa_{d}+K}{G})}{\frac{\kappa_{d}+K}{G}},$ (105)

$\frac{1}{K_{eff}}=2\Re\left(\frac{(|\kappa_{d}+K|-2i|G|-2\kappa_{c})|\kappa_{d}+K|+\kappa_{d}^{2}-2i\kappa_{d}\Re(G)-|K|^{2}-2i\Re(KG^{*})}{|\kappa_{d}+K|(\kappa_{d}-K-2iG)(\kappa_{d}-K^{*}-2iG^{*})-(|\kappa_{d}+K|-2\kappa_{c}-2i|G|^{2})}\right).$
(106)