# PrototypeFormer: Learning to Explore Prototype Relationships for
Few-shot Image Classification

Feihong He1, Gang Li2,3, Lingyu Si2,3, Leilei Yan1 Fanzhang Li†,1, Fuchun Sun4
$\dagger$ Corresponding author

###### Abstract

Few-shot image classification has received considerable attention for
addressing the challenge of poor classification performance with limited
samples in novel classes. However, numerous studies have employed
sophisticated learning strategies and diversified feature extraction methods
to address this issue. In this paper, we propose our method called
PrototypeFormer, which aims to significantly advance traditional few-shot
image classification approaches by exploring prototype relationships.
Specifically, we utilize a transformer architecture to build a prototype
extraction module, aiming to extract class representations that are more
discriminative for few-shot classification. Additionally, during the model
training process, we propose a contrastive learning-based optimization
approach to optimize prototype features in few-shot learning scenarios.
Despite its simplicity, the method performs remarkably well, with no bells and
whistles. We have experimented with our approach on several popular few-shot
image classification benchmark datasets, which shows that our method
outperforms all current state-of-the-art methods. In particular, our method
achieves 97.07% and 90.88% on 5-way 5-shot and 5-way 1-shot tasks of
miniImageNet, which surpasses the state-of-the-art results with accuracy of
7.27% and 8.72%, respectively. The code will be released later.

## Introduction

Neural networks have been remarkably successful in large-scale image
classification. However, the domain of few-shot image classification, where
models must rapidly adapt to new data distributions with limited labeled
samples (e.g., five or one sample for each class), remains a challenge. As a
result of its promising applications in diverse fields such as medical image
analysis and robotics, few-shot learning (Wang et al. 2020) has captivated the
attention of the computer vision and machine learning community.

Recent few-shot learning approaches mainly improve the generalization by
augmenting the samples/features or facilitating feature representation with
novel neural modules. A multitude of methods (Zhang et al. 2018, 2017; Chen et
al. 2019; Schwartz et al. 2018; Yang, Liu, and Xu 2021; Li, Zheng, and Su
2022) utilizes generative models to generate new samples or augment feature
space, aiming to approximate the actual distribution, and thus finally
facilitate the few-shot learning. Devising sophisticated feature
representation modules is also a meaningful way to improve the model
performance on low-shot categories. Specifically, CAN (Hou et al. 2019)
leverages cross-attention mechanisms to acquire enriched sample embeddings
with enhanced class-specific features in a transductive way, while DN4 (Li et
al. 2019), DMN4 (Liu et al. 2022b), and MCL (Liu et al. 2022a) adopt local
feature representations instead of global representations to obtain more
discriminative feature representations. Following the line of feature
representation learning approaches, we introduce a prototype extraction module
to enhance the prototype embeddings. Contrary to earlier feature
representation methodologies, our study delves into the intricate
interconnections both within each class and across the entire task to derive
more discriminative prototype representations.

Learning prototype embedding (Snell, Swersky, and Zemel 2017; Zhang et al.
2021) is useful for few-shot classification. ProtoNet (Snell, Swersky, and
Zemel 2017) introduces a methodology employing prototype points to encapsulate
the feature embeddings of entire categories, and (Zhang et al. 2021) proposes
to enhance the notion of prototype points. However, they significantly ignore
the prototype relationships for learning robust class features. In this paper,
we delve into the interconnections between prototype points, considering both
intra-class and inter-class relationships. We first introduce a novel
prototype extraction module to learn the relationship of intra-class samples
through the self-attention of sub-prototypes, which are obtained by average
subsets of sample features. This module captures more representative category
features through the consideration of feature relationships among sample
subsets of the same class.

To further enhance the robustness of class features in few-shot scenarios, we
introduce the “prototype contrastive loss”—a novel contrastive loss designed
explicitly to capture interactions among inter-class prototypes. Central to
our approach is the concept of “sub-prototypes,” representing both positive
and negative class embeddings in the few-shot context. By employing these sub-
prototypes within a contrastive learning framework, we aim to cultivate more
discriminative representations. Specifically, the contrastive learning
strategy ensures that similar class embeddings are drawn closer in the feature
space, while dissimilar ones are pushed apart, thus enhancing the
discriminative power of our representative prototypes.

Moreover, some works (Zhou et al. 2022; Gao et al. 2021) have demonstrated the
impressive feature extraction capabilities of the CLIP pre-trained model in
few-shot learning. We also integrate the image encoder of CLIP into our
approach as the powerful feature extraction module.

We summarize our contributions as follows:

  * •

Prototype Extraction Module. We introduce a novel and straightforward network
architecture for few-shot learning, employing a learnable prototype extraction
module to extract prototype representations.

  * •

Prototype Contrastive Loss. We form sub-prototypes by employing linear
combinations of the support subsets. Subsequently, we optimize the model using
the prototype contrastive loss based on these sub-prototypes to obtain more
robust prototype representations.

  * •

Achieving State-of-the-Art Performance. We evaluate our method on multiple
publicly few-shot benchmark datasets, and the results demonstrate that the
proposed method outperforms state-of-the-art few-shot learning methods across
three popular datasets. It’s worth noting that we achive a remarkable
improvement of up to 8.72% in miniImageNet dataset.

Figure 1: Exploring the feature relationships among samples of the same class
and different classes to construct task-specific feature representations.
Figure 2: This figure presents the overall process flowchart of the method
proposed in this paper. We linearly combine the support set and obtain sub-
prototypes through the prototype extraction module. The sub-prototypes are
utilized for computing the prototype contrastive loss $L_{prototype}$, while
the prototype is employed for calculating the classification loss
$L_{classifier}$. We sum the $L_{prototype}$ and $L_{classifier}$ to obtain
the final optimization objective.

## Related Work

### Few-shot Learning

The rapid development of deep neural networks in recent years has primarily
benefited from large-scale labeled datasets. However, the high cost of data
collection and manual labeling has brought few-shot learning to the forefront
of widespread interest.

Few-shot learning is usually classified into optimization-based and metric-
based methods. The main idea of metric-based methods is to define specific
metrics to classify samples in a way similar to the nearest neighbor
algorithm. The Siamese Network(Koch et al. 2015) employs shared feature
extractors to derive feature representations from both support sets and query
sets. Subsequently, it computes classification similarity individually for
each pair of support set and query set. The Prototypical Network (Snell,
Swersky, and Zemel 2017) computes prototype points for each class of samples,
and the query samples are categorized by calculating the L2 distance to each
prototype point. In Relation Network (Sung et al. 2018), the incorporation of
learnable nonlinear classifiers for sample classification is done
innovatively. CAN (Hou et al. 2019) has improved model performance by
computing cross-attention on samples to enhance the network’s focus on
classification targets. Also, to reduce sample background interference, local
descriptors that do not contain classification targets are eliminated in DN4
(Li et al. 2019) and DMN4 (Liu et al. 2022b) by comparing the similarity
between local descriptors. COSOC (Luo et al. 2021), as a similar endeavor,
seeks to enhance classification performance by distinguishing between
classification targets and background elements. HCTransformers (He et al.
2022) propose a hierarchical cascading transformer architecture, aiming to
address the overfitting challenges faced by large-scale models in few-shot
learning. Meanwhile, FewTURE (Hiller et al. 2022) similarly employs
transformer architecture to extract key features from the main subjects within
images. In the realm of generalized few-shot learning, a substantial body of
work (Zhou et al. 2022; Zhang et al. 2022b) has already leveraged pre-trained
models to enhance the efficacy of few-shot learning. In our research, we have
also incorporated the pre-trained CLIP (Radford et al. 2021) model to enhance
the feature extraction capabilities of our model. The critical distinction,
however, lies in the fact that our model is trained using a meta-learning
approach.

### Sample Relation

There exist diverse sample relationships among different class samples, and
currently, most models are built upon the foundation of establishing these
sample relationships. Numerous studies aim for models to achieve strong
generalization performance across various class sample relationships, thereby
minimizing vicinal risk. CAN (Hou et al. 2019) and OLTR (Liu et al. 2019)
incorporate sample-specific relationships within the shared context by
leveraging the correlations among individual samples. IEM (Zhu and Yang 2020)
analyzes local correlations among samples and performs memory storage updates
for these correlations. IRM (Arjovsky et al. 2019) achieves a reduced vicinal
risk by exploring the correlation between sample invariant features and
spurious features. In cross-domain tasks, (Huang et al. 2020) explores the
transferability of sample relationships across different domains by discarding
specific sample relationships. Similar to (Huang et al. 2020), (Peng et al.
2019) explores domain-invariant and class-invariant relationships by employing
the deep adversarial disentangled autoencoder to achieve cross-domain
classification tasks. BatchFormer (Hou, Yu, and Tao 2022) has achieved
significant improvements across various data scarcity tasks by implicitly
exploring the relationships among mini-batch samples during training. In mixup
(Zhang et al. 2017), samples are linearly interpolated to capture the class-
invariant relationships between samples. In our work, we perform linear
combinations of samples to explore task-relevant relationships among them.

### Contrative Learning

Contrastive learning has achieved significant success in recent years.
InstDisc (Wu et al. 2018) proposes the utilization of instance discrimination
tasks as an alternative to class-based discrimination tasks within the
framework of unsupervised learning. MOCO (He et al. 2020) achieves favorable
transferability to downstream tasks through the strategy of constructing a
dynamic dictionary and performing momentum-based updates. Contrastive learning
has exhibited its generality and flexibility in time series tasks,
encompassing domains like audio and textual data. An abundance of work (He et
al. 2020; Chen et al. 2020; Radford et al. 2021) has demonstrated the positive
impact of contrastive learning in both unsupervised learning and
generalization research within the realm of computer vision. The objective of
contrastive learning is to bring together samples of the same class while
separating those from different classes, thus constructing suitable patterns
for sample feature extraction. In episodic training, we utilize contrastive
learning methods to extract class relationships within the task, enhancing the
classification performance for few-shot learning.

## Method

In this section, we first describe the problem definition related to few-shot
learning. Subsequently, an exposition of our proposed methodology is
presented. Conclusively, we delve into a comprehensive discussion on the two
important components of our method: Prototype Extraction Module and Prototype
Contrastive Loss.

### Problem Formulation

Episodic training differs from the deep neural networks training approach. In
the traditional training of deep neural networks, we usually train the neural
network on a sample-by-sample basis. In episodic training, we typically train
the neural network on a task-by-task basis. The episodic training mechanism
(Vinyals et al. 2016) has been demonstrated to facilitate the learning of
transferable knowledge across classes.

In few-shot learning, we usually divide the dataset into training, validation,
and test sets. The training set, validation set, and test set have no
overlapping classes. Therefore, we refer to the classes in the training set as
seen classes, while the classes in the validation set and test set are termed
unseen classes. During the training phase, we randomly sample from the
training set to create the support set and the query set. We use $S$ to
represent the support set and $Q$ to define the query set. In the support set
$S$, there are $N$ classes, and each class contains $K$ samples. We treat the
query set $Q$ as unlabeled samples and perform classification on the unlabeled
samples in $Q$ using the labeled samples in the support set $S$, which
contains $N$ classes, each with $K$ samples. During the testing phase, we
follow the same procedure and divide the test set into a support set and a
query set, similar to what we did during the training phase. This allows us to
evaluate the few-shot learning performance of the model on unseen classes in a
manner consistent with the training process. We typically refer to tasks that
satisfy the above settings as N-way K-shot tasks. In our work, we train and
evaluate the model using the aforementioned problem formulation.

Figure 3: The prototype extraction module adopts the transformer structure
(Vaswani et al. 2017), taking the prototype token and embeddings of same-class
images from the support set as inputs to obtain the prototype and sub-
prototype for that class.

### Overview

We linearly combine the support set and apply non-linear mapping through the
prototype extraction module. Furthermore, we optimize the prototype extraction
module using contrastive learning strategies to attain improved prototype
representations. An overview of our approach is depicted in Figure 2.

As illustrated in Figure 2, we process both the support set and query samples
through a frozen CLIP feature extraction network to obtain image embeddings.
Subsequently, we perform linear combinations on the support set samples to
generate $C^{1}_{K}$ sub-support sets. Simultaneously, a prototype token is
added to each support set and sub-support set, derived by computing the
average of the respective embedding collection. Individually, each support set
and sub-support set is fed into the prototype extraction module to obtain
encoded prototypes and sub-prototypes. We retain the prototypes and sub-
prototypes while discarding the sample embeddings from the support sets. We
compute $L_{Prototype}$ using the retained sub-prototypes through contrastive
loss, while $L_{Classifier}$ is obtained by calculating the embeddings of
query samples and prototypes. Finally, we sum up $L_{Prototype}$ and
$L_{Classifier}$ to create the ultimate optimization objective.

### Prototype Extraction Module

In this section, we will provide a comprehensive exposition of our proposed
prototype extraction module. Additionally, we will conduct a comparative
analysis between our method and existing class feature extraction approaches
found in the paper.

First we introduce the prototype representation, the earliest class feature
representation to appear in few-shot learning. In the N-way K-shot task, we
assume the existence of a class C, and in the support set $S$, there exists a
subset $S_{C}=\left\\{x_{1},x_{2},\dots x_{K}\mid y=C\right\\}$. We refer to
the feature extraction network as $f$. In that case, we can express the class
feature representation in the prototypical networks (Snell, Swersky, and Zemel
2017) as follows:

$Prototype\left(C\right)=\frac{1}{K}\sum_{i=1}^{K}f\left(x_{i}\right),x_{i}\subset
S_{C}$

The method of prototype points provides a simple and effective way to express
class features. Absolutely, the global average pooling layer used in the
feature extraction network can introduce noise into the prototype points,
causing them to deviate from their true representation and leading to bias. To
address this issue, DN4 (Li et al. 2019) and DMN4 (Liu et al. 2022b) remove
the global average pooling layer from the feature extraction network. They
employ local descriptors to replace the global feature representation of
images and utilize a discriminative nearest neighbor algorithm to obtain the
most representative local descriptors in the images as the feature
representation for samples.

However, we believe that the image background has a certain influence on the
image classification performance and also provides some category-related
contextual features. Therefore, we propose a novel class feature extraction
module referred to as prototype extraction module to replace the current few-
shot class feature representation. In ViT (Dosovitskiy et al. 2020), the image
is divided into patches, and transformer (Vaswani et al. 2017) is utilized to
compute the correlations between these patches, resulting in the overall
feature representation of the entire image. Inspired by ViT, we simply treat
the image as a set of patches input to the transformer, thereby obtaining the
feature representation for the entire class. The fundamental architecture of
prototype extraction module is illustrated in Figure 3. We use $\phi$ to
represent the prototype extraction module, and we can express it in the
following form:

$Prototype\left(C\right)=$
$\phi\left(x_{token},f\left(x_{1}\right),f\left(x_{2}\right),\dots
f\left(x_{K}\right)\right),x_{i}\subset S_{C}$

In the formula, $x_{token}$ represents the prototype token for that class, and
it can be expressed as:

$x_{token}=\frac{1}{K}\sum_{i=1}^{K}f\left(x_{i}\right),x_{i}\subset S_{C}$

Finally, we use a simple metric learning classification method to classify the
query samples. Specifically, we calculate the distance between the embeddings
of the query samples and the prototype points in the feature space to measure
the similarity between the query samples and each class. This distance metric
is used for classification, where the query sample is assigned to the class
with the closest feature embedding in the feature space. This classification
approach can be formalized with the following formula:

$argmin_{c\subset C}L_{2}\left(x_{query},Prototype\left(c\right)\right)$

The classification loss is optimized using the cross-entropy loss, and the
formula for the classification loss is as follows:

$Loss_{classify}=$
$-\sum_{c=1}^{N}y_{c}log\left(\frac{e^{-L_{2}\left(x_{query},Prototype\left(c\right)\right)}}{\sum_{i=1}^{N}e^{-L_{2}\left(x_{query},Prototype\left(i\right)\right)}}\right)$

The $y_{c}$ is the one-hot encoding of the true class label for the sample.

### Prototype Contrastive Loss

To enhance the generalization capability of the prototype extraction module,
we drew inspiration from contrastive learning and proposed prototype
contrastive loss. The contrastive loss was first introduced by (Hadsell,
Chopra, and LeCun 2006) and laid the foundation for subsequent highly
successful contrastive learning (He et al. 2020; Chen et al. 2020). The main
idea of the contrastive loss is to construct positive and negative sample
pairs, where positive pairs are brought closer together in the feature space,
while negative pairs are pushed further apart.

In few-shot learning, by extracting K-1 samples from the same class in the
support set $S$, we can obtain $K$ different sub-support set of samples
$S_{ci}=\left\\{x_{c1},\dots,x_{ci-1},x_{ci+1}\dots,x_{cK}\right\\},i=1,2\dots
K,c\subset C$. Then, we pass each of these K sub-support sets constructed from
the same class samples through the prototype extraction module to obtain K
sub-prototypes for that class. We use the K sub-prototypes obtained from the
same-class support set samples as positive pairs. At the same time, we use the
sub-prototypes obtained from different-class sub-support sets as negative
pairs. We represent the constructed positive sample pairs as follows:

$Pos_{c}=\left\\{p_{c1},p_{c2},\dots p_{cK}\right\\},C=1,2\dots N$

Thus, we can obtain the prototype contrastive loss using the constructed
positive and negative pairs as follows:

$L_{prototype}=exp\left(\frac{1}{N}\cdot\frac{\sum_{i,j=1}^{K}L_{2}\left(p_{ci},p_{cj}\right)+I}{\sum_{m\neq
n}\sum_{i,j=1}^{K}L_{2}\left(p_{m,i},p_{n,j}\right)+I}\right)$

Because when $K=1$, the support set contains only one sample per class,
leading to $\sum_{i,j=1}^{K}L_{2}\left(p_{ci},p_{cj}\right)=0$. To avoid this
situation, we add the identity element I to prevent it from happening. The
overall loss of the model during the training phase is as follows:

$Loss=Loss_{classifier}+Loss_{prototype}$

Model | miniImageNet | tieredImagenet | CUB-200
---|---|---|---
5-way 5-shot | 5-way 1-shot | 5-way 5-shot | 5-way 1-shot | 5-way 5-shot | 5-way 1-shot
MAML (Finn, Abbeel, and Levine 2017) | 64.31±1.1% | 47.78±1.75% | 71.10±1.67% | 52.07±0.91% | - | -
Prototypical Network (Snell, Swersky, and Zemel 2017) | 78.44±0.21% | 60.76±0.39% | 80.11±0.91% | 66.25±0.34% | - | -
HCTransformers (He et al. 2022) | 89.19 ± 0.13% | 74.62 ± 0.20% | 91.72 ± 0.11% | 79.57 ± 0.20% | - | -
DeepEMD (Zhang et al. 2022a) | 82.41 ± 0.56% | 65.91 ± 0.82% | 86.03 ± 0.58% | 71.16 ± 0.87% | 88.69 ± 0.50% | 75.65 ± 0.83%
MCL (Liu et al. 2022a) | 83.99% | 67.51% | 86.02% | 72.01% | 93.18% | 85.63%
POODLE (Le et al. 2021) | 85.81% | 77.56% | 86.96% | 79.67% | 93.80% | 89.88%
FRN (Wertheimer, Tang, and Hariharan 2021) | 82.83±0.13% | 66.45±0.19% | 86.89±0.14% | 72.06±0.22% | 92.92±0.10% | 83.55±0.19%
PTN (Huang et al. 2021) | 88.43±0.67% | 82.66±0.97% | 89.14±0.71% | 84.70±1.14% | - | -
FewTURE (Hiller et al. 2022) | 86.38±0.49% | 72.40±0.78% | 89.96±0.55% | 76.32±0.87% | - | -
EASY (Bendou et al. 2022) | 89.14 ± 0.1% | 84.04 ± 0.2% | 89.76 ± 0.14% | 84.29 ± 0.24% | 93.79 ± 0.10% | 90.56 ± 0.19%
iLPC (Lazarou, Stathaki, and Avrithis 2021) | 88.82±0.42% | 83.05±0.79% | 92.46±0.42% | 88.50±0.75% | 94.11±0.30 | 91.03±0.63
Simple CNAPS (Bateni et al. 2020) | 89.80% | 82.16% | 89.01% | 78.29% | - | -
Ours | 97.07 ± 0.11% | 90.88 ± 0.31% | 95.00 ± 0.19% | 87.26 ± 0.40% | 94.25 ± 0.16% | 89.04 ± 0.35%

Table 1: Few-shot learning classification accuracies on miniImageNet,
tieredImagenet and CUB-200 under the setting of 5-way 1-shot and 5-way 5-shot
with 95% confidence interval. (‘-’ not reported)

## Experiments

In this section, we will evaluate the proposed method on multiple few-shot
benchmark datasets and compare it with state-of-the-art methods. Additionally,
we will conduct ablation experiments and visualization experiments to further
analyze and validate the effectiveness of PrototypeFormer.

### Datasets

miniImageNet (Vinyals et al. 2016) is a subset of the larger ImageNet dataset
and is widely used in few-shot learning research. It consists of 100 classes,
with each class containing 600 images, resulting in a total of 60,000 images.
The dataset is divided into 64 classes for the training set, 16 classes for
the validation set, and 20 classes for the test set.

tieredImagenet is a larger subset of the ImageNet dataset compared to
miniImagenet. The dataset consists of 608 classes with a total of 779,165
images. For few-shot learning, it is divided into three subsets, with 351
classes used for the training set, 97 classes for the validation set, and 160
classes for the testing set.

Caltech-UCSD Birds-200-2011 (Wah et al. 2011), also known as CUB, is the
benchmark image dataset for current fine-grained classification and
recognition research. The dataset contains 11,788 bird images, encompassing
200 subclasses of bird species. We split it into 100, 50, and 50 classes for
training, validation, and testing, respectively.

### Experimental Settings

To obtain better image features, we use ViT-Large/14 as the backbone for image
feature extraction and pair it with the same CLIP pre-trained model used in
CoOp (Zhou et al. 2022) and Clip-Adapter (Gao et al. 2021). Due to the limited
data in the context of few-shot learning, prototype extraction module adopts a
two-layer transformer architecture without incorporating positional encoding.
During the training phase, we freeze the feature extraction network and only
train the prototype extraction module proposed in this paper to preserve the
image feature extraction capabilities of the pre-trained CLIP model and obtain
a prototype extraction module with excellent class feature representations.

During the training phase, we maintain the traditional episodic training
approach and conduct training on 5-way 5-shot and 5-way 1-shot task settings.
Additionally, we use the Adam (Kingma and Ba 2014) optimizer to optimize the
model. We set the initial learning rate of the optimizer to 0.0001. The
momentum weight coefficients $\beta$1 and $\beta$2, as well as the $\epsilon$
parameter of the optimizer, are set to their default values of 0.9, 0.999, and
1e-8, respectively. In the gradient updating strategy, we adopt the gradient
accumulation algorithm, where we accumulate gradients over every 10 batches
before performing a parameter update. We train the model for 100 epochs, where
each epoch consisted of 500 batches, and each batch represented a task. In
image augmentation, we resize the images and then apply center cropping to
obtain $224\times 224$ pixel image inputs.

In the testing phase, to ensure fairness, we adhere to the evaluation
methodology of few-shot learning without making any changes. We randomly
sample 2000 tasks from the test set. For each task, we extract 15 query
samples per class to evaluate our method. We report the average accuracy with
a 95% confidence interval to ensure the reliability of our results.

Figure 4: We randomly select eight task sets from the test dataset and
visualize their feature embeddings using t-SNE (Van der Maaten and Hinton
2008). In the visualization, circular points represent query samples,
triangles represent prototype points obtained by averaging the support set,
and pentagrams represent class feature embeddings obtained through our
proposed method in this paper.

### Results

Following the few-shot standard experimental settings, we conduct experiments
on both 5-way 1-shot and 5-way 5-shot tasks to evaluate our method. The
experimental results are presented in Table 1.

As shown in the table 1, our method outperforms the current state-of-the-art
results on both 5-way 5-shot and 5-way 1-shot tasks in the miniImageNet
dataset. Excitingly, our method achieve an accuracy improvement of 7.27% over
the current state-of-the-art method in the 5-way 5-shot task on this dataset.
At the same time, our method also achieve a 6.84% accuracy improvement in the
5-way 1-shot task compared to the current state-of-the-art method. Our method
achieve significant improvements in the 5-way 5-shot task on both the
tieredImagenet dataset and the CUB-200 dataset compared to the existing
methods. Observing the table, we can notice that compared to the 5-way 5-shot
tasks, our method’s performance is slightly inferior in the 5-way 1-shot
tasks. We believe that this is due to the lack of positive pairs in the 5-way
1-shot task, which hinders the prototype extraction module’s ability to
represent class features accurately.

### Ablation Study

To validate the effectiveness of our method, we conduct ablation experiments
from various perspectives on the proposed approach.

Model | miniImageNet | tieredImagenet | CUB-200
---|---|---|---
5-way 5-shot | 5-way 1-shot | 5-way 5-shot | 5-way 1-shot | 5-way 5-shot | 5-way 1-shot
CLIP | 95.13 ± 0.14% | 83.86 ± 0.40% | 92.25 ± 0.24% | 79.24 ± 0.46% | 89.20 ± 0.24% | 72.51 ± 0.51%
Ours | 97.07 ± 0.11% | 90.88 ± 0.31% | 95.00 ± 0.19% | 87.26 ± 0.40% | 94.25 ± 0.16% | 89.04 ± 0.35%
Table 2: This ablation experiment aims to validate the effectiveness of the
prototype extraction module.

To validate the effectiveness of prototype extraction module, we conduct
ablation experiments under two conditions: removing the prototype extraction
module and retaining the prototype extraction module as part of our method.
The experimental results are shown in Table 2, where “CLIP” represents the
condition where we remove the prototype extraction module and retain only the
CLIP pre-trained model. From the Table 2, we can observe that the CLIP pre-
trained model itself exhibits good few-shot image classification performance
due to its strong zero-shot knowledge transfer ability in few-shot learning.
Furthermore, our proposed method shows significant performance improvement
compared to the comparative methods in the ablation experiments.

As shown in Table 3, we conduct experiments on the miniImageNet dataset in
both 5-way 5-shot and 5-way 1-shot settings with and without the inclusion of
the prototype contrastive loss. The experimental results indicate that the
prototype loss has a positive impact on model optimization. Additionally,in
Table 4, we conducte ablation experiments on prototype extraction modules with
2, 4 and 6 layers of transformer blocks.

Model | miniImageNet
---|---
5-Way 5-Shot | 5-Way 1-Shot
L_classifier | 96.24 ± 0.11% | 89.13 ± 0.32%
L_classifier+L_prototype | 97.07 ± 0.11% | 90.88 ± 0.31%
Table 3: The table presents a comparative experiment on whether to include the
prototype contrastive loss in the model. Figure 5: We randomly select eight
task sets from the test dataset and visualize their feature embeddings using
t-SNE (Van der Maaten and Hinton 2008). In the visualization, circular points
represent query samples, triangles represent prototypes obtained by averaging
the support sets, and pentagrams represent prototypes obtained through our
proposed method in this paper.

## Visualization

In this section, we conduct visualization analysis using the model trained on
the 5-way 5-shot task of the miniImageNet dataset. As shown in Figure 4, we
randomly extract samples from 8 tasks in the test set and visualize them using
t-SNE. In the visualization, we only show the 15 query set samples using
circular symbols. We use triangular symbols to represent the prototype points
obtained by averaging the embeddings of support set samples. Additionally, we
use pentagram symbols to represent the prototype obtained using the prototype
extraction module proposed in this paper.

By observing Figure 4, we can notice that the class embeddings obtained using
the prototype point calculation method in the prototypical network (Snell,
Swersky, and Zemel 2017) are relatively positioned closer to the center of the
respective classes. On the other hand, the class embeddings obtained using the
method proposed in this paper are positioned toward the edges of the
respective classes. We believe that the class embeddings obtained through the
prototype point calculation method are driven by representing the
characteristics of that class. The class embeddings obtained using the method
proposed in this paper are driven by classification purposes. As a result, the
prototype points are always positioned at the center of the class to describe
the distribution of that class in the feature space. On the contrary, the
class embeddings obtained through our method are positioned towards the edges
of the class, aiming to be far away from other class samples and closer to
samples of their own class for effective classification.

Finally, we conduct a matrix similarity visualization comparing our method
with the prototype point approach. As illustrated in Figure 5, where “CLIP”
denotes the traditional prototype point representation using the CLIP pre-
trained model as the backbone. We conduct experiments separately on
miniImageNet, tieredImagenet, and CUB-200 datasets. The results depicted in
Figure 5 clearly demonstrate a significant enhancement achieved by our method
in the few-shot classification.

L$\times$block | miniImageNet
---|---
5-way 5-shot | 5-way 1-shot
2 | 97.07 ± 0.11% | 90.88 ± 0.31%
4 | 95.96 ± 0.13% | 90.03 ± 0.33%
6 | 94.44 ± 0.17% | 88.33 ± 0.35%
Table 4: Ablation experiments of prototype extraction module with 2, 4 and 6
transformer blocks on miniImageNet dataset.

## Conclusions

We propose a transformer-based module for extracting class feature embeddings,
which is applied to traditional few-shot learning tasks to achieve improved
classification performance. Moreover, inspired by contrastive learning, we
introduce an optimization strategy that relies on this module, aiming to
obtain prototype representations that are more finely adapted to the
classification task. By capitalizing on the inherent structure and
interconnections among the data instances, our approach reinforces the
discriminative capacity of the acquired embeddings, consequently yielding
superior effectiveness and precision in few-shot classification tasks. We
evaluated our proposed method on several popular few-shot image classification
benchmark datasets and conduct comprehensive analyses through ablation
experiments and visualization techniques. The experimental results demonstrate
that our approach significantly outperforms the current state-of-the-art
methods.

## References

  * Arjovsky et al. (2019) Arjovsky, M.; Bottou, L.; Gulrajani, I.; and Lopez-Paz, D. 2019.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_.
  * Bateni et al. (2020) Bateni, P.; Goyal, R.; Masrani, V.; Wood, F.; and Sigal, L. 2020.  Improved few-shot visual classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 14493–14502.
  * Bendou et al. (2022) Bendou, Y.; Hu, Y.; Lafargue, R.; Lioi, G.; Pasdeloup, B.; Pateux, S.; and Gripon, V. 2022.  Easy—ensemble augmented-shot-y-shaped learning: State-of-the-art few-shot classification with simple components.  _Journal of Imaging_ , 8(7): 179.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607. PMLR.
  * Chen et al. (2019) Chen, Z.; Fu, Y.; Zhang, Y.; Jiang, Y.-G.; Xue, X.; and Sigal, L. 2019.  Multi-level semantic feature augmentation for one-shot learning.  _IEEE Transactions on Image Processing_ , 28(9): 4594–4605.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Finn, Abbeel, and Levine (2017) Finn, C.; Abbeel, P.; and Levine, S. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , 1126–1135. PMLR.
  * Gao et al. (2021) Gao, P.; Geng, S.; Zhang, R.; Ma, T.; Fang, R.; Zhang, Y.; Li, H.; and Qiao, Y. 2021\.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv preprint arXiv:2110.04544_.
  * Hadsell, Chopra, and LeCun (2006) Hadsell, R.; Chopra, S.; and LeCun, Y. 2006.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06)_ , volume 2, 1735–1742. IEEE.
  * He et al. (2020) He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 9729–9738.
  * He et al. (2022) He, Y.; Liang, W.; Zhao, D.; Zhou, H.-Y.; Ge, W.; Yu, Y.; and Zhang, W. 2022.  Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9119–9129.
  * Hiller et al. (2022) Hiller, M.; Ma, R.; Harandi, M.; and Drummond, T. 2022.  Rethinking generalization in few-shot classification.  _Advances in Neural Information Processing Systems_ , 35: 3582–3595.
  * Hou et al. (2019) Hou, R.; Chang, H.; Ma, B.; Shan, S.; and Chen, X. 2019.  Cross attention network for few-shot classification.  _Advances in neural information processing systems_ , 32.
  * Hou, Yu, and Tao (2022) Hou, Z.; Yu, B.; and Tao, D. 2022.  Batchformer: Learning to explore sample relationships for robust representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7256–7266.
  * Huang et al. (2021) Huang, H.; Zhang, J.; Zhang, J.; Wu, Q.; and Xu, C. 2021.  Ptn: A poisson transfer network for semi-supervised few-shot learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 1602–1609.
  * Huang et al. (2020) Huang, Z.; Wang, H.; Xing, E. P.; and Huang, D. 2020.  Self-challenging improves cross-domain generalization.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , 124–140. Springer.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Koch et al. (2015) Koch, G.; Zemel, R.; Salakhutdinov, R.; et al. 2015.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2. Lille.
  * Lazarou, Stathaki, and Avrithis (2021) Lazarou, M.; Stathaki, T.; and Avrithis, Y. 2021.  Iterative label cleaning for transductive and semi-supervised few-shot learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 8751–8760.
  * Le et al. (2021) Le, D.; Nguyen, K. D.; Nguyen, K.; Tran, Q.-H.; Nguyen, R.; and Hua, B.-S. 2021\.  Poodle: Improving few-shot learning via penalizing out-of-distribution samples.  _Advances in Neural Information Processing Systems_ , 34: 23942–23955.
  * Li, Zheng, and Su (2022) Li, G.; Zheng, C.; and Su, B. 2022.  Transductive distribution calibration for few-shot learning.  _Neurocomputing_ , 500: 604–615.
  * Li et al. (2019) Li, W.; Wang, L.; Xu, J.; Huo, J.; Gao, Y.; and Luo, J. 2019.  Revisiting local descriptor based image-to-class measure for few-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 7260–7268.
  * Liu et al. (2022a) Liu, Y.; Zhang, W.; Xiang, C.; Zheng, T.; Cai, D.; and He, X. 2022a.  Learning to affiliate: Mutual centralized learning for few-shot classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 14411–14420.
  * Liu et al. (2022b) Liu, Y.; Zheng, T.; Song, J.; Cai, D.; and He, X. 2022b.  Dmn4: Few-shot learning via discriminative mutual nearest neighbor neural network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 1828–1836.
  * Liu et al. (2019) Liu, Z.; Miao, Z.; Zhan, X.; Wang, J.; Gong, B.; and Yu, S. X. 2019.  Large-scale long-tailed recognition in an open world.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2537–2546.
  * Luo et al. (2021) Luo, X.; Wei, L.; Wen, L.; Yang, J.; Xie, L.; Xu, Z.; and Tian, Q. 2021.  Rectifying the shortcut learning of background for few-shot learning.  _Advances in Neural Information Processing Systems_ , 34: 13073–13085.
  * Peng et al. (2019) Peng, X.; Huang, Z.; Sun, X.; and Saenko, K. 2019.  Domain agnostic learning with disentangled representations.  In _International Conference on Machine Learning_ , 5102–5112. PMLR.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763. PMLR.
  * Schwartz et al. (2018) Schwartz, E.; Karlinsky, L.; Shtok, J.; Harary, S.; Marder, M.; Kumar, A.; Feris, R.; Giryes, R.; and Bronstein, A. 2018.  Delta-encoder: an effective sample synthesis method for few-shot object recognition.  _Advances in neural information processing systems_ , 31.
  * Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017.  Prototypical networks for few-shot learning.  _Advances in neural information processing systems_ , 30.
  * Sung et al. (2018) Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P. H.; and Hospedales, T. M. 2018\.  Learning to compare: Relation network for few-shot learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1199–1208.
  * Van der Maaten and Hinton (2008) Van der Maaten, L.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ , 9(11).
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29.
  * Wah et al. (2011) Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011.  The caltech-ucsd birds-200-2011 dataset.
  * Wang et al. (2020) Wang, Y.; Yao, Q.; Kwok, J. T.; and Ni, L. M. 2020.  Generalizing from a few examples: A survey on few-shot learning.  _ACM computing surveys (csur)_ , 53(3): 1–34.
  * Wertheimer, Tang, and Hariharan (2021) Wertheimer, D.; Tang, L.; and Hariharan, B. 2021.  Few-shot classification with feature map reconstruction networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8012–8021.
  * Wu et al. (2018) Wu, Z.; Xiong, Y.; Yu, S. X.; and Lin, D. 2018.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3733–3742.
  * Yang, Liu, and Xu (2021) Yang, S.; Liu, L.; and Xu, M. 2021.  Free lunch for few-shot learning: Distribution calibration.  _arXiv preprint arXiv:2101.06395_.
  * Zhang et al. (2021) Zhang, B.; Li, X.; Ye, Y.; Huang, Z.; and Zhang, L. 2021.  Prototype completion with primitive knowledge for few-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3754–3762.
  * Zhang et al. (2022a) Zhang, C.; Cai, Y.; Lin, G.; and Shen, C. 2022a.  Deepemd: Differentiable earth mover’s distance for few-shot learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5): 5632–5648.
  * Zhang et al. (2017) Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2017.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_.
  * Zhang et al. (2018) Zhang, R.; Che, T.; Ghahramani, Z.; Bengio, Y.; and Song, Y. 2018.  Metagan: An adversarial approach to few-shot learning.  _Advances in neural information processing systems_ , 31.
  * Zhang et al. (2022b) Zhang, R.; Zhang, W.; Fang, R.; Gao, P.; Li, K.; Dai, J.; Qiao, Y.; and Li, H. 2022b.  Tip-adapter: Training-free adaption of clip for few-shot classification.  In _European Conference on Computer Vision_ , 493–510. Springer.
  * Zhou et al. (2022) Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9): 2337–2348.
  * Zhu and Yang (2020) Zhu, L.; and Yang, Y. 2020.  Inflated episodic memory with region self-attention for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4344–4353.


# Immersive Video Compression using Implicit Neural Representations ††thanks:
This work was funded by UK EPSRC (iCASE Awards), BT and the UKRI MyWorld
Strength in Places Programme. High performance computational facilities were
provided by the Advanced Computing Research Centre, University of Bristol.

Ho Man Kwan  {hm.kwan, fan.zhang<EMAIL_ADDRESS>Fan Zhang
{hm.kwan, fan.zhang<EMAIL_ADDRESS>Andrew Gower
<EMAIL_ADDRESS>David Bull  {hm.kwan, fan.zhang,
<EMAIL_ADDRESS>
###### Abstract

Recent work on implicit neural representations (INRs) has evidenced their
potential for efficiently representing and encoding conventional video
content. In this paper we, for the first time, extend their application to
immersive (multi-view) videos, by proposing MV-HiNeRV, a new INR-based
immersive video codec. MV-HiNeRV is an enhanced version of a state-of-the-art
INR-based video codec, HiNeRV, which was developed for single-view video
compression. We have modified the model to learn a different group of feature
grids for each view, and share the learnt network parameters among all views.
This enables the model to effectively exploit the spatio-temporal and the
inter-view redundancy that exists within multi-view videos. The proposed codec
was used to compress multi-view texture and depth video sequences in the MPEG
Immersive Video (MIV) Common Test Conditions, and tested against the MIV Test
model (TMIV) that uses the VVenC video codec. The results demonstrate the
superior performance of MV-HiNeRV, with significant coding gains (up to
72.33%) over TMIV. The implementation of MV-HiNeRV is published for further
development and evaluation111https://hmkx.github.io/mv-hinerv/.

###### Index Terms:

Video Compression, Immersive video, Multi-view video, Implicit neural
representation, MV-HiNeRV

##  I Introduction

As part of an extended video parameter space, including higher spatial
resolution, greater dynamic range, higher frame rate and wider colour gamut,
new video formats have emerged to enable more immersive viewing experience
with three or six degrees of freedom (3DoF or 6DoF). These underpin the
development of virtual reality (VR), augmented reality (AR) or mixed reality
(MR) systems [1]. Raw immersive video data is typically generated based on
computer-generated imagery models or captured using multiple-camera systems
and dedicated geometry sensors. It is then converted into different data
formats, such as point clouds, multi-view texture+depth or equirectangular
video, which are compressed using different video/data codecs for transmission
or storage. To present or display immersive video content, the compressed data
is decoded and synthesised/rendered to enable 3DoF/6DoF viewing capabilities
on VR, AR or MR devices.

To standardise immersive video production and streaming, the Moving Picture
Experts Group (MPEG) has developed various solutions, which support current
immersive video formats including multi-view video with depth and point cloud
(PC). The former is commonly used in video-based production workflows (e.g. 3D
film) and the latter is mainly employed for 3D graphics-based production (e.g.
3D games). For both MVD and PC, two standards have been recently developed in
the framework of MPEG-I, referred to as MIV (MPEG Immersive Video) [2] and
V-PCC (Video-based Point Cloud Compression) [3]. In this paper, we focus
solely on video compression for MVD.

In the MIV pipeline, the input comprises multiple source views for both
texture and depth data, together with associated source camera parameters.
Both texture and depth frames are first processed to identify basic and
additional views. The redundancies in the additional views are then removed,
while non-redundant information is combined with the basic video and
compressed using a standard video encoder (e.g., HEVC HM [4] or VVC VVenC
[5]). It is noted that, while it is convenient to employ a conventional video
codec for texture and depth video compression, such codecs are not designed to
exploit the redundancy within multiple views, and MIV achieve this by
extracting basic views and removing redundant information in additional views.

In the realm of learning-based video compression, deep neural networks have
been deployed to enhance the traditional coding pipeline [6, 7, 8, 9, 10] or
build a fully learnable framework to achieve end-to-end optimisation [11, 12,
13, 14]. More recently, Implicit Neural Representations (INRs) [15, 16, 17,
18, 19] have shown great potential to achieve comparable compression
performance to both standard and other learning-based video codecs,
importantly also achieving relatively fast decoding speeds. An INR is a neural
network that is optimised to map coordinates to pixels during video encoding,
reconstructing the entire video at the decoder by performing inference.
However, existing INR-based codecs were designed for encoding single-view
videos; the application to multi-view video coding has not previously been
investigated.

To this end, we investigate the use of INRs for immersive video compression,
and propose a new INR-based multi-view video codec, MV-HiNeRV. The proposed
approach is an extension of a state-of-the-art INR-based video codec, HiNeRV
[18], which was developed for conventional video compression. Specifically,
for multi-view video coding, MV-HiNeRV learns a unique set of feature grids
for each view, with shared network parameters across views. The sharing of
network parameters effectively exploits the spatio-temporal redundancy among
different views.

To evaluate its coding efficiency, MV-HiNeRV has been tested on MIV CTC test
sequences [20] and compared against the Test Model of MIV (TMIV) [21]. The
results show its significant performance improvement over the original TMIV,
with an average bit rate saving of 46.92%. To the best of our knowledge, MV-
HiNeRV is the first INR-based immersive video codec, and the results
demonstrate the promise of using INR-based models for immersive video
compression.

##  II Method

Implicit Neural Representation (INR)-based video compression has been shown to
outperform or compete with various advanced standard codecs in terms of coding
efficiency [15, 16, 17, 22, 18, 19]. In the related works, a single instance
of INR has shown capability of encoding a large number of video frames, for
example, up-to 600 frames for the sequences in the UVG dataset[23]. Due to its
ability to exploit redundancies between frames, this capability is likely to
benefit the compression of immersive videos which exhibit additional
redundancies through multiple views. Based on this observation, we propose an
INR-based codec, MV-HiNeRV (a multi-view version of HiNeRV [18]) for encoding
multi-view videos with the aim of achieving improved coding performance.

###  II-A MV-HiNeRV

(a)

(b)

Figure 1: (a). HiNeRV. In HiNeRV, an input patch can be obtained by
interpolation from the feature grids, and the network layers output the
corresponding video patch. (b). The proposed MV-HiNeRV. In MV-HiNeRV, each
view is represented by a dedicated set of feature grids, and the network
layers are shared across views. This allow efficient multi-view video coding
as it exploiting the spatial, temporal and view redundancy simultaneously.

Existing multi-view video coding methods either compress multiple views by
leveraging inter-view redundancy [24, 25], or apply view pruning [2]. As
mentioned above, by utilising INR models, it is possible to directly compress
a large number of video views with a high coding efficiency. Moreover, using
INR models for exploiting redundancy is potentially time efficient, since the
encoding process is the model training, which its length has a sub-linear
relationship with the number of input frames or videos; this differs from
conventional and other learning-based coding methods.

In this work, to adapt HiNeRV for multi-view video compression, we modified
the original INR model by storing a separated set of feature grids for each
view, where the network parameters, e.g., those for the convolutional and
fully connected layers, are shared among views, in addition to the spatial and
temporal dimensions. This allows simultaneous exploitation of redundancy in
all three dimensions. Fig. 1(a) and 1(b) show the difference between the
original HiNeRV and the proposed multi-view variant.

Following the approach in [18], assuming that we encode a multi-view video
with $K$ views, we denote the $k$-th view as $V_{k}$, such that $0\leq k<K$.
For simplicity, we assume that the video size is the same for
$V_{0}$…$V_{K-1}$, i.e., $T\times H\times W\times C$, where $T$, $H$, $W$ and
$C$ are the temporal resolution, spatial resolution and the number of
channels, respectively.

With a patch size $M\times M$, MV-HiNeRV represents each video in multiple
patches, where each patch is denoted by its patch index $(i,j,t)$, where
$0\leq t<T$, $0\leq j<\frac{H}{M}$ and $0\leq i<\frac{W}{M}$. To compute the
output of the $k$-th view, MV-HiNeRV first computes the input feature map,
i.e., the encoding obtained from interpolation with the multi-temporal
resolution grids [17] from the $k$-th view, $\gamma_{k,base}$; it then applies
a stem convolutional layer $F_{stem}$, to obtain the first stage feature map:

$X_{0}=F_{stem}(\gamma_{k,base}(i,j,t)).$ (1)

Subsequently, $N$ HiNeRV blocks are progressively processed using the feature
maps for both upscaling and transformation, where we denote the output of the
$n$-th HiNeRV block as $X_{n}$, $0<n\leq N$:

$\begin{split}X_{n}=&F_{n}(U_{n}(X_{n-1})+F_{enc}(\gamma_{k,n}(i,j,t))),0<n\leq
N\end{split}$ (2)

It should be noted that the hierarchical encoding [18] in MV-HiNeRV is also
view dependent, as the grids are compact and only need a small number of
parameters. The hierarchical encoding at the $k$-th view of the $n$-th block
is denoted as $\gamma_{k,n}$. In MV-HiNeRV, we simply follow HiNeRV, where we
use the ConvNeXt [26] block as the internal layer in the HiNeRV blocks.

Finally, a linear head layer $F_{head}$ is applied to transform the feature
maps $X_{N}$ into the target output space:

$Y=F_{head}(X_{N}),$ (3)

Unlike HiNeRV, MV-HiNeRV encodes both the texture and depth maps at the same
time, hence we produce a four channel (RGB and D) output with the linear head
layer.

During model training, we randomly sample video patches from all views and
frames [18]. We also perform training with overlapped patches in the feature
space; this supports training with patches while enhancing the encoding
quality.

###  II-B Weight Quantisation and Entropy Regularisation

Weight quantisation. Following [19], we applied the learned quantisation and
entropy regularisation to reduce the model size. Specifically, for a parameter
vector $\boldsymbol{\theta}=\\{\theta_{i}\\}$ with a trainable quantisation
step vector $\boldsymbol{\delta}=\\{\delta_{i}\\}$, the quantised parameter
$\boldsymbol{\hat{\theta}}=\\{\hat{\theta_{i}}\\}$ is computed as follows:

$\hat{\theta_{i}}=\delta\times\lfloor\frac{\theta_{i}}{\delta_{i}}\rceil.$ (4)

We follow the model compression implementation in [27], where the logarithm of
$\delta$ is learned instead of $\delta$. The actual step sizes are also shared
between a subset of the parameters from the same layer, in order to reduce the
overhead. For instance, the same step size is employed for a row or a column
in the weight of the linear layer. Since quantisation is a non-differentiable
operation, we use Quant-Noise [28] as in HiNeRV [18] in the training and
employ the quantisation during evaluation.

Entropy regularisation. To train the model parameters and the quantisation
step sizes at the same time, we follow the common practice in lossy
compression where entropy regularisation is used to optimise both the rate and
distortion jointly, incorporating a Lagrangian multiplier, $\lambda$:

$L=R+\lambda D,$ (5)

Here, the rate term $R$ is defined as the sum of the negative log likelihood
of the quantised weights, i.e., the lower bound of the total code length, and
the distortion term $D$ can be calculated using different loss functions such
as the MSE loss. For a model with a quantised parameter vector
$\boldsymbol{\hat{\theta}}=\\{\hat{\theta_{i}}\\}$, we compute the rate term
by

$R=\sum_{i=0}^{|\boldsymbol{\hat{\theta}}|}-\mathrm{log}_{2}p(\hat{\theta_{i}})$
(6)

In particular, we use a multivariate Gaussian [29, 30] for estimating the
weight distribution and thus the rate term. Unlike other learning-based video
codecs [11, 12], the distribution of the symbols, can be computed easily
because the symbols are simply the quantised network parameters. Thus, we
compute the Gaussian parameters directly in each training step, and encode
them along with with the model parameter bitstream. To estimate the rate of
each symbol, we also use the continuous approximation of the quantised
variables in [31]. We only apply regularisation to the feature grids and the
weights of the linear and convolutional layers.

Previous work [19] employed a non-parametric model [29] for estimating the
distribution, introducing additional parameters that require training. This
may increase complexity, so in this work we simply assume Gaussian
distribution which is effective in practice.

###  II-C Model Training Pipeline

Regarding model training, previous work in [19] adopted a two stage approach,
in which two models are trained without regularisation in Stage 1, while
multiple models are trained in Stage 2 with regularisation for different rate
points, achieved by adjusting the weight between the distortion and rate
terms. However, we found that the above approach will lead to sub-optimal
rate-distortion performance when the weight of the rate term is heavier, i.e.,
for obtaining the more compact models. Hence, in this paper, we modified this
two-stage approach [19], but using a different value for $\lambda$, and a
different set of the model hyper-parameters for each target rate point model,
as in some other INR-based works [15, 16, 17, 22, 18]. In Stage 1, we train
one MV-HiNeRV model without any regularisation which allows faster
convergence. In Stage 2 we manually initialise the step sizes to a value such
that the quantisation width to 7 at the beginning. Then, we train the model
with entropy regularisation; where we found that using a smaller learning rate
for network parameters and a larger one for the step size is actually more
effective for preserving the model quality while reducing the size. We also
apply a linear scheduling for the quantisation noise rate [28]. In our
approach, only 300 epochs are used for Stage 1 training without
regularisation, and 60 epochs for Stage 2, while 1200 and 300 epochs are used
in [19], respectively. Most of the training configurations for MV-HiNeRV are
the same as those in [18].

##  III Experiments

We employed six mandatory test sequences in the MIV Common Test Conditions
(CTC) [20] for testing. We compared MV-HiNeRV with the MPEG Test Model (TMIV)
[21] using the main anchor configuration. TMIV employs VVenC [5] in Random
Access mode to encode both texture and depth information. Following the CTC,
we performed experiments with the specified start frames and encoded 65 frames
in each sequence. For MV-HiNeRV, we converted the original YUV 4:2:0 texture
frames into the RGB colour space, then concatenated them with the depth frames
as the input.

After decoding using MV-HiNeRV, we convert the output of MV-HiNeRV back to the
original YUV 4:2:0 format, and apply the same view synthesizer in TMIV to
generate both source and pose trace views for evaluation, as described in the
MIV CTC [20]. We use both PSNR and IV-PSNR [32] to evaluate video quality, and
employ range coding for entropy coding to obtain the actual rate. Metrics are
computed with regard to the best reference [20].

For MV-HiNeRV training, we use MSE loss in the RGBD space as the distortion
term $D$. It should be noted that the reconstruction quality is highly
dependent on the accuracy of the depth map. Computing MSE loss in the depth
channel may result in insufficient precision due to the varying range of depth
values across different scenarios. As a result, we normalise the depth map of
each sequence to [0, 1]. Detailed network and training configurations are
provided in the code repository.

###  III-A Quantitative and Qualitative Results

TABLE I summarises compression results for MV-HiNeRV compared to TMIV in terms
of BD-rate values measured in PSNR/IV-PSNR [32] (the average among both
reconstructed source and pose trace views). MV-HiNeRV performs significantly
better than TMIV on all six test sequences, achieving up to 72.33% bit rate
savings (measured by Bjøntegaard Delta Rate, BD-rate [33] using IV-PSNR). This
performance is also confirmed by Fig. 3, which plots the rate-PSNR curves for
all sequences (including both the reconstructed views at the source and the
pose trace views).

We also measured the bit rate distribution, and we found that the shared
parameters accounted for more than 60% of the total rate on average. Given the
high quantitative performance, this suggest that in MV-HiNeRV, the shared
parameters are an effective shared representation between views.

TABLE I: BD-Rate (%, measured in PSNR/IV-PSNR) results of MV-HiNeRV on the MIV
CTC test sequences (for both source and pose trace views). The anchor is TMIV.

Metric | B02 | D01 | E01 | J02 | J04 | W01 | Overall
---|---|---|---|---|---|---|---
PSNR | -17.60 | -65.93 | -36.59 | -59.96 | -80.03 | -35.48 | -49.27
IV-PSNR | -38.11 | -61.08 | -6.28 | -70.21 | -72.33 | -33.50 | -46.92

For perceptual quality assessment, we provide visual examples of the
synthesised views from MIV and MV-HiNeRV (using the same viewer synthesiser)
in Fig. 2. It can be observed that, in general, the MV-HiNeRV produces high-
quality frames with rich spatial details. More importantly, the outputs
contain fewer visual artefacts as in TMIV, which may be due to the MIV
preprocessing. In these cases, the bit rates for MV-HiNeRV examples are
similar or even smaller than that for MIV. The only issue we noticed is that
MV-HiNeRV perform worse in cases where high depth map precision is required,
such as for the edges of the foreground objects in sequence B02.

In terms of complexity, the encoding process with MV-HiNeRV is relatively
slow, similar to existing INR-based methods [15, 16, 17, 22, 18, 19]. For
example, encoding the sequence W01, which comprises 21 full HD views, each
consisting of 65 frames, can take up to 26 hours on a computer with a V100
GPU. While decoding with INRs is typically fast, in our case, the entire
decoding process is limited by the TMIV view synthesizer. For the same
sequence, decoding all source views with MV-HiNeRV can be completed in less
than 2 minutes, but rendering a synthesised view with the TMIV view
synthesizer takes approximately 12 minutes.

(a) B02 - p03

(b) D01 - v13

(c) J02 - p02

(d) J04 - v07

Figure 2: Example of synthesised views (cropped) of the CTC test sequence
[20]. pXX: pose trace views. vXX: source views.

##  IV Conclusion

In this paper, we proposed an INR-based codec, MV-HiNeRV, for encoding multi-
view videos. MV-HiNeRV is an extension of HiNeRV, which was developed for
single-view video compression. The proposed approach learns a different set of
feature grids for each view and shares the model parameters among all views,
effectively exploiting the inter-view redundancy. MV-HiNeRV has been evaluated
against the MPEG MIV Test Model, TMIV, and achieved significant performance
improvement, with 46.92% average coding gain. The results demonstrate the
significant potential of INR-based video codecs for the compression of
immersive video formats. Future work should focus on better integration of
viewer synthesiser with INR-based immersive video codecs.

Figure 3: IV-PSNR results of TMIV and MV-HiNeRV on the MIV CTC test sequences.
pose: Pose trace views. src: Source views. all: The average of pose
trace/source views.

## References

  * [1] D. Bull and F. Zhang, _Intelligent image and video compression: communicating pictures_. Academic Press, 2021.
  * [2] J. M. Boyce, R. Doré, A. Dziembowski, J. Fleureau, J. Jung, B. Kroon, B. Salahieh, V. K. M. Vadakital, and L. Yu, “MPEG immersive video coding standard,” _Proc. IEEE_ , vol. 109, no. 9, pp. 1521–1536, 2021.
  * [3] D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, and A. Tabatabai, “An overview of ongoing point cloud compression standardization activities: Video-based (v-pcc) and geometry-based (g-pcc),” _APSIPA Transactions on Signal and Information Processing_ , vol. 9, p. e13, 2020.
  * [4] G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 22, no. 12, pp. 1649–1668, 2012. [Online]. Available: https://doi.org/10.1109/TCSVT.2012.2221191
  * [5] A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe, “VVenC: An Open And Optimized VVC Encoder Implementation,” in _Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW)_ , pp. 1–2.
  * [6] R. Song, D. Liu, H. Li, and F. Wu, “Neural network-based arithmetic coding of intra prediction modes in HEVC,” in _2017 IEEE Visual Communications and Image Processing, VCIP 2017, St. Petersburg, FL, USA, December 10-13, 2017_. IEEE, 2017, pp. 1–4. [Online]. Available: https://doi.org/10.1109/VCIP.2017.8305104
  * [7] M. Afonso, F. Zhang, and D. R. Bull, “Video compression based on spatio-temporal resolution adaptation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 1, pp. 275–280, 2018.
  * [8] F. Zhang, D. Ma, C. Feng, and D. R. Bull, “Video compression with CNN-based postprocessing,” _IEEE MultiMedia_ , vol. 28, no. 4, pp. 74–83, 2021.
  * [9] D. Ma, F. Zhang, and D. R. Bull, “MFRNet: a new cnn architecture for post-processing and in-loop filtering,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 15, no. 2, pp. 378–387, 2020.
  * [10] F. Zhang, M. Afonso, and D. R. Bull, “ViSTRA2: Video coding using spatial resolution and effective bit depth adaptation,” _Signal Processing: Image Communication_ , vol. 97, p. 116355, 2021.
  * [11] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “DVC: An End-To-End Deep Video Compression Framework,” in _CVPR_. Computer Vision Foundation / IEEE, 2019, pp. 11 006–11 015.
  * [12] J. Li, B. Li, and Y. Lu, “Deep Contextual Video Compression,” in _NeurIPS_ , 2021, pp. 18 114–18 125.
  * [13] E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici, “Scale-Space Flow for End-to-End Optimized Video Compression,” in _CVPR_. Computer Vision Foundation / IEEE, 2020, pp. 8500–8509.
  * [14] J. Li, B. Li, and Y. Lu, “Neural Video Compression with Diverse Contexts,” in _CVPR_. IEEE, 2023, pp. 22 616–22 626.
  * [15] H. Chen, B. He, H. Wang, Y. Ren, S. Lim, and A. Shrivastava, “NeRV: Neural Representations for Videos,” in _NeurIPS_ , 2021, pp. 21 557–21 568.
  * [16] H. Chen, M. Gwilliam, S. Lim, and A. Shrivastava, “HNeRV: A Hybrid Neural Representation for Videos,” in _CVPR_. IEEE, 2023, pp. 10 270–10 279.
  * [17] J. C. Lee, D. Rho, J. H. Ko, and E. Park, “FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos,” in _ACM Multimedia_. ACM, 2023.
  * [18] H. M. Kwan, G. Gao, F. Zhang, A. Gower, and D. Bull, “HiNeRV: Video Compression with Hierarchical Encoding based Neural Representation,” 2023.
  * [19] C. Gomes, R. Azevedo, and C. Schroers, “Video Compression with Entropy-Constrained Neural Representations,” in _CVPR_. IEEE, 2023, pp. 18 497–18 506.
  * [20] A. Dziembowski, B. Kroon, and J. Jung, “Common test conditions for MPEG immersive video.” ISO/IEC JTC 1/SC 29/WG 04, 2023.
  * [21] G. L. Adrian Dziembowski, “Test model 17 for MPEG immersive video.” ISO/IEC JTC 1/SC 29/WG 4, 2023.
  * [22] B. He, X. Yang, H. Wang, Z. Wu, H. Chen, S. Huang, Y. Ren, S. Lim, and A. Shrivastava, “Towards scalable neural representation for diverse videos,” in _CVPR_. IEEE, 2023, pp. 6132–6142.
  * [23] A. Mercat, M. Viitanen, and J. Vanne, “UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development,” in _MMSys_. ACM, 2020, pp. 297–302.
  * [24] A. Vetro, T. Wiegand, and G. J. Sullivan, “Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard,” _Proc. IEEE_ , vol. 99, no. 4, pp. 626–642, 2011.
  * [25] G. Tech, Y. Chen, K. Müller, J. Ohm, A. Vetro, and Y. Wang, “Overview of the Multiview and 3D Extensions of High Efficiency Video Coding,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 26, no. 1, pp. 35–49, 2016.
  * [26] Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” in _CVPR_. IEEE, 2022, pp. 11 966–11 976.
  * [27] J. Ballé, S. J. Hwang, and E. Agustsson, “TensorFlow Compression: Learned data compression,” 2023. [Online]. Available: http://github.com/tensorflow/compression
  * [28] P. Stock, A. Fan, B. Graham, E. Grave, R. Gribonval, H. Jégou, and A. Joulin, “Training with Quantization Noise for Extreme Model Compression,” in _ICLR_. OpenReview.net, 2021.
  * [29] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in _ICLR_. OpenReview.net, 2018.
  * [30] D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in _NeurIPS_ , 2018, pp. 10 794–10 803.
  * [31] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end Optimized Image Compression,” in _ICLR_. OpenReview.net, 2017.
  * [32] A. Dziembowski, D. Mieloch, J. Stankowski, and A. Grzelka, “IV-PSNR - the objective quality metric for immersive video applications,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 32, no. 11, pp. 7575–7591, 2022.
  * [33] G. Bjøntegaard, “Calculation of average PSNR differences between RD-curves,” in _13th VCEG Meeting, no. VCEG-M33,Austin, Texas_ , 2001, pp. USA: ITU–T.


# PER Measurement of BLE in RF Interference and Harsh Electromagnetic
Environment

Mir Lodro, Gabriele Gradoni , Ana Vukovic, David Thomas and Steve Greedy Mir
Lodro, Steve Greedy, Ana Vukovic, David Thomas and Gabriele Gradoni are with
George Green Institute for Electromagnetic Research-GGIEMR, the University of
Nottingham, UK. Gabriele Gradoni is also with British Telecommunications and
the University of Cambridge, UK.

###### Abstract

Bluetooth Low Energy (BLE) is a short-range data transmission technology that
is used for multimedia file sharing, home automation, and internet-of-things
application. In this work, we perform packet error rate (PER) measurement and
RF testing of BLE receiver in the harsh electromagnetic environment and in
presence of RF interference. We check the PER performance in the line-of-sight
(LOS) and non-line-of-sight (NLOS) scenario in absence of any interfering
signal and in presence of wideband WLAN interference. The BLE PER measurements
are conducted in a large reverberation chamber which is a rich scattering
environment. Software-defined-radio has been used to create BLE communication
link for PER measurement in LOS and NLOS configuration. The BLE PER is
measured both in the presence and in absence of WLAN interference. Our
measurement results show a higher PER for uncoded BLE PHY modes in NLOS
channel condition and in presence of wideband interference. Whereas coded BLE
PHY modes i.e. LE500K and LE125K are robust to interference with lower PER
measurements.

###### Index Terms:

PlutoSDR, EVM, BLE Receiver, Internet of Things, PER Measurement, EM
interference, Channel Characterization.

##  I Introduction

Bluetooth low energy (BLE) has become critically important as it has been
demonstrated by authors in [1][2][3] where it is shown that BLE can
potentially contain and delay the outbreak of infectious diseases such as the
ongoing severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic.
BLE is a wireless personal area network (WPAN) technology created by Bluetooth
special interest group (SIG) as BLE 4.0. The latest BLE 5.0 can attain a data
rate of 2 Mbps with two additional coded PHY modes supporting a data rate of
500 kbps and 125 kbps respectively. In contrast to classical Bluetooth, BLE is
targeted for short-range power-constraint internet of things (IoT)
applications such as medical internet of things, smart homes, indoor
positioning, and localization [4][5]. BLE operates in two modes connected
communication and non-connected communication. Non-connection mode is used for
device discovery to make a connection for data transfer. It’s also known as
the neighbor discovery process (NDP). Once the link is established using the
NDP process, the two devices can perform data transfer using connected
communication mode. BLE is a low-power short-range communication system. Apart
from multimedia file transmission between mobile phones its usage in IoT
applications and home automation is on the constant rise. BLE backscatter
communication is presented in [6][7]. Work in [8] shows smart city soil health
monitoring for healthy plants. Authors in [9] have addressed dense deployment
of BLE for the body are networks. BLE neighbor discovery issue is addressed in
[10]. The rapid influx of internet of things devices per square kilometer has
increased the risk of interference in an already spectrally contested EM
environment. IoT devices operate using WiFi, Zigbee, BLE, and narrowband-IoT
(NB-IoT) PHY layers. Almost all IoT devices use the most sought-after 2.4 GHz
ISM band. This paper is organized into five sections. Section I is the
introduction, Section II is the background, Section III is about measurement
setup. Experimental results are presented in Section IV. Conclusion and future
aspect of the work has been given in conclusion in Section V.

##  II Background

BLE uses GMSK for LE1M and LE2M PHY modes that support uncoded 1 Mbps and 2
Mbps data rates respectively. BLE also uses coded LE500K and LE125K PHY layer
modes that support data rates of 500 kbps and 125 kbps respectively. BLE has
four maximum transmit power classes. Class 1, class 1.5, class 2 and class 3
allow maximum transmit power of 20 dBm (100 mW), 10 dBm (10 mW), 4 dBm (4 mW)
and 0 dBm (1 mW) respectively. BLE uses GMSK modulation in comparison to basic
rate/enhanced data rate (BR/EDR) Bluetooth which used GFSK, $\pi/4$-DQPSK, and
8-DPSK modulation techniques. Additionally, BLE uses broadcast, mesh, and
point-to-point network topologies.

###  II-A BLE Frequency Plan

The BLE system uses forty RF channels separated by 2MHz, unlike Zigbee which
uses sixteen RF channels separated by 5 MHz. BLE uses 2.4 GHz ISM band at
2400-2483.5 MHz. Figure 1 shows BLE RF channels where each channel is
allocated a unique channel index. The BLE frequency plan shows advertising and
data channels. BLE has two main channels advertising channels and data
channels. Advertising channels are unidirectional or broadcast channels for
device discovery and beacon signals. Out of forty BLE, RF channels three
channels (channel number 37, 38, and 39) are advertising channels and thirty-
seven channels (channel 0-36) are data channels. Advertising channels are
spread across BLE 2.4 GHz ISM band in order to avoid interference. The purpose
of advertising channels is to transmit advertising packets, scan packets, and
connection packets.

Figure 1: BLE frequency plan showing advertising channels and data channels.

###  II-B BLE Channel Selection Algorithms

BLE uses channel hopping to avoid interference and yield maximum throughput.
2.4 GHz ISM band is used by several wireless standards such as WLAN, therefore
BLE is susceptible to interference. In presence of interference, many
retransmission may be requested for successful data delivery. BLE advertising
device transmits advertising packets on three advertising channels in a cyclic
manner. The scanning device listens on three advertising channels in a cyclic
manner. There are two BLE channel selection algorithms whose purpose is to
avoid channels that are susceptible to interference. Good and bad data channel
map is selected based on key performance indicators (KPIs), for example, SNR
and PER. The channel map is exchanged between the master and the slave.
Algorithm1 is governed by hope count, current channel, and a list of used good
channels. Algorithm2 uses access address and a list of used good channels as
main parameters.

###  II-C Packet Format

BLE has two packet formats for both uncoded and coded PHY layers. The packet
format for uncoded LE1M and LE2M PHY modes of BLE is shown in Fig.2(a). The
preamble is a fixed sequence of alternating 1 and 0 bit, and it is part of all
link-layer BLE packets. The preamble is used in the receiver to perform
functions like automatic gain control (AGC) function, frequency
synchronization, and timing recovery. The preamble length for LE1M PHY and
LE2M PHY modes is 8 bits and 16 bits respectively. It consists of a 32-bit
access address, 16-2056 bit PDU, and 24-bit CRC. The packet format for coded
PHY LE125K and LE500K is shown in Fig.2(b). It has an 80-bit preamble, 32-bit
access address, 16-2056 PDU bits, and 24-bit CRC. It has also three more
fields 2-bit coding indicator that indicates coding level hence coded BLE PHY
layer and 3-bit term1 and term 2 fields.

((a))

((b))

Figure 2: BLE packet format structure for uncoded and coded PHY modes. TABLE I: BLE SPECIFICATIONS Parameters | Value
---|---
Frequency Range | 2.4-2.4835 GHz
Modulation | GMSK
PHY layer modes | LE1M,LE2M,LE500K,LE125K
Number of Channels | 40
Frame Length | 16832
Total number of frames | 10,000

###  II-D BLE Waveform Generation

BLE link layer PDUs both advertising PDUs and data channel PDUs can be
generated and can be converted into BLE waveforms as shown in Fig.3. PDUs are
converted to standard-compliant baseband waveforms which are transmitted over
the air using a software-defined-radio platform. Fig. 3(a) and Fig.3(b) shows
procedure for BLE waveform generation of uncoded and coded PHY modes
respectively.

((a))

((b))

Figure 3: BLE uncoded and coded waveform generation.

###  II-E BLE Receiver

BLE receiver performs RF impairment compensation such as RF nonlinearity
effect, carrier frequency offset, carrier phase offset, and symbol timing
synchronization. Standard-compliant BLE receivers consist of AGC stage, DC
compensation stage, matched filtering, carrier frequency offset (CFO)
estimation, and timing synchronization stage. After the initial receiver
operations demodulation of GMSK symbols is accomplished and a CRC check is
performed. The AGC component in the digital receiver is vital for the correct
recovery of information and for end-to-end PER performance measurement. The
AGC is the first stage where a weak signal is received and stabilized for
further downstream processing. The AGC can be configured in slow-attack mode
or fast-attack mode to track the received signal. Fast attack AGC is used when
the environment is non-stationary. Previously we have recorded our
experimental experience with AGC in the metal enclosure [11]. The AGC helps
for coarse and fine frequency correction in the baseband receiver. After the
AGC stage, DC removal in the received waveform is performed using a notch
filter. The existence of the DC component is a common problem in radio
receivers and it is particularly common in direct-conversion receivers such as
software-defined-radios. The next two stages are frequency offset correction
and matched filtering. Frequency offset estimation and correction must be
performed before received samples are processed by a matched filter. The
matched filter is implemented half at the transmitter and a half at the
receiver and it increases the SNR i.e. it limits the amount of noise that is
passed down to subsequent stages[12] in the receiver. Coarse frequency
compensation uses an FFT-based or correlation-based algorithm to measure
frequency offset. Fine frequency and demodulation of GMSK packets are
performed. After these critical design stages, data decoding and de-mapping
are performed. Long series of zeros and ones such as 000000 or 1111111 may
occur in the data bitstream. Data whitening is used to avoid such a long
sequence of ones and zeros. Data de-whitening and CRC are performed to measure
the BLE PER. For the packet error measurement, access address is checked first
and then CRC is checked, If the access address or CRC is incorrect then the
packet shall be rejected. The received BLE packets is considered only valid
when both access address and CRC check is correct.

##  III Measurement Setup

We generated a baseband BLE waveform using MATLAB based BLE transmitter. The
generated BLE waveforms were transmitted using PlutoSDR connected to the host
PC. The transmitted BLE signals were received by another PlutoSDR connected
with the same host PC running the BLE receiver in another MATLAB session.
PlutoSDR is a full-duplex transceiver. PlutoSDR uses Xilinx Zynq Z-7010 FPGA
and a highly integrated RF agile transceiver chip AD9363 from Analog Devices.
It has RF coverage from 325 MHz to 3.8 GHz which is software upgradeable from
70 MHz to 6 GHz. It can provide instantaneous bandwidth of 20 MHz and it
offers 12-bit ADC/DAC flexible rate. It has support from MATLAB, Simulink, and
GNU Radio and it has C++ and Python API. We performed BLE PER measurement in
NDP mode in LOS and NLOS scenario in reverberation chamber which is a highly
reflecting environment. In RC the scattering is very rich, therefore, a higher
number of multipath components can be expected similar to a factory
environment where dense metal bars, enclosures, ducts, and other
instrumentation are used. We recorded higher PER in the reverberation chamber
for Tx gain of 0 dB and Rx gain of 25 dB. We repeated the same with Tx and Rx
gain values in an indoor laboratory environment where PER of $1e^{-2}$ was
measured with repeatable measurements. The PER measurements within RC were
around $0.5$ with repeatable measurements, We also ruled out any possibility
of cables connecting PlutoSDRs with host PC were faulty.

Figure 4: PlutoSDR based NLOS measurement setup in presence of RF absorbers
and interferer. Figure 5: PlutoSDR based LOS measurement setup in presence of
wideband RF interference.

Fig. 4 shows measurement setup where Tx and Rx PlutoSDRs are separated by a
NLOS metal sheet. Third PlutoSDR is used as source of narrowband and wideband
interference, grid of RF abosrber cones is placed closer to the Tx. We
performed measurements in master and scanner configuration. We conducted PER
measurement in perfectly metal conducting environment. We observed higher PER
values at 2.402 GHz for various Tx and Rx gain values configuration.

Figure 6: BLE measurement flowchart.

Fig.6 shows a flowchart of the measurement process and the Tx and Rx
processing steps where link-layer PDUs of the desired BLE PHY modes converted
into IQ baseband waveform. The baseband IQ waveforms for a given channel
configuration are upconverted and transmitted over-the-air using PlutoSDR. BLE
waveforms are captured using Rx PlutoSDRs which sends downconverted samples to
host PC for receiver processing. After performing receiver operations number
of packets and CRC are counted and BLE PER is measured.

###  III-A LOS and NLOS Scenario

In this scenario, Tx and Rx PlutoSDRs are placed in LOS and NLOS of each other
at a distance of x m. The interferer was inactive in both the LOS and NLOS
scenario. We checked the PER measurement at 2.402 GHz, we noticed frequency-
selectivity in the baseband spectrum. We conducted PER measurement in a
perfectly metal conducting environment. We observed higher PER values at 2.402
GHz for various Tx and Rx gain values configurations. Later on, we introduced
RF absorbers to reduce the effect of fading and repeated PER measurements in
NLOS for various Tx and Rx gain values. After the PER measurements in NLOS, we
repeated the same measurements without the NLOS metal sheet. We observed that
no packets were detected, it happened because of frequency selectivity in the
spectrum. We changed the Tx and Rx roles of SDR and performed PER measurements
in LOS. We noticed that there was no frequency selectivity in the baseband
spectrum and the packets were received for all PHY modes of BLE. The addition
of absorbers reduced the effect of frequency-selectivity which was observed
from the received baseband spectrum. Fig. (7(a) shows clean Tx baseband
spectrum of BLE waveform of coded LE125K BLE PHY. At the receiver side, BLE
waveform and the packets are captured as shown in Fig.7(b).

((a))

((b))

Figure 7: Baseband spectrum of transmit and received GMSK modulated BLE500K
waveform in LOS case for Tx Gain=-10 dB and Rx gain=25 dB.

((a))

((b))

Figure 8: PER measurement of four BLE PHY in NLOS and LOS channel condition.

Figure 8 shows PER of BLE PHY modes in NLOS and LOS channel conditions. It can
be seen from Fig.8(a) that the PER is highest for uncoded PHY modes of BLE. A
large PER measurement sample was collected for all BLE PHY modes. The PER
reduced when we removed the NLOS metal sheet. The reason for reduced PER is
the presence of increased forward transmission and dominant LOS component. The
existence of a dominant LOS component increased the SNR of the BLE link. In
the case of the NLOS metal sheet, there was no forward propagation from Tx to
Rx in LOS. PER of coded PHY is lower than the uncoded PHY counterparts in the
same channel propagation condition. This is because coded PHY modes are more
robust to MPCs. PER for LE125K is the lowest of all BLE PHY modes in both NLOS
and LOS scenarios. In the NLOS scenario, the maximum PER has reduced from 0.34
to 0.05 which is an $85.20\%$ reduction in the maximum PER. In contrast to the
NLOS scenario, the maximum PER in the LOS scenario has reduced from 0.48 to
0.03 which is a $93\%$ reduction in the maximum PER. It is clear from these
PER values in both scenarios that the coded PHY modes are robust to fading and
offers the lowest PER values in a harsh EM environment.

###  III-B LOS and NLOS in Presence of Interference

This section addresses the PER of BLE in both LOS and NLOS in presence of a
WLAN signal operating at a frequency of 2.402 GHz. We generated a 20 MHz wide
WLAN interference signal using B210 USRP. B210 USRP was configured to transmit
standard-compliant IEEE 802.11a frames [13]. We checked the performance in
presence of 20 MHz wide WLAN interfering signals. B210 USRP was mounted at
around 12 ft from the main BLE communication link. We checked the BLE PER
performance for various Tx gain values of B210 USRP. B210 USRP was set to
operate with Tx gain values of 65 dB, 70 dB, and 75 dB respectively. For each
of the Tx gain values, we checked the PER performance of all BLE PHY modes. We
noticed that uncoded BLE PHY modes i.e. LE1M and LE2M exhibited higher PER for
all the Tx gain values of the main communication link and at all interference
power levels. In contrast to uncoded BLE PHY modes, coded BLE PHY modes LE500K
and LE125K were robust to RF interference at all interference power
levels.LE1M and LE2M BLE modes were susceptible to interference and the PER
was 1 or no packets were detected. In contrast LE1M and LE2M, the LE125K and
LE500K were robust to RF interference and have shown low PER in presence of
in-band interference (see Fig.9). The worst PER for LE500K was 0.333 and the
best was no PERs, where LE125K was highly robust to interference because for
most of the runs it has shown no packets received in error.

((a))

((b))

Figure 9: PER measurement of four BLE PHY in LOS and in presence of WLAN
interference.

##  IV Conclusion and Future Work

We checked the performance of the BLE receiver in the LOS and NLOS scenario in
the presence and absence of WLAN interference in a highly reflecting metal
environment. Initially, we observed frequency selectivity in the baseband
spectrum of BLE received signal both in LOS and NLOS environment. In this
configuration, high PER was observed in all BLE PHY modes. Then, we introduced
RF absorbers to reduce the effect of frequency selectivity and later performed
PER in both LOS and NLOS environments. The addition of RF absorbers removed
the frequency selectivity and the PER for BLE PHY modes was improved. After
the addition of RF absorbers, we conducted PER measurements in presence of
wideband WLAN interference. Higher PER measurements are observed in presence
of RF interference for LE1M and LE2M, however, coded LE500K and LE125K are
more robust to RF interference.

## References

  * [1] F. Sattler, J. Ma, P. Wagner, D. Neumann, M. Wenzel, R. Schäfer, W. Samek, K.-R. Müller, and T. Wiegand, “Risk estimation of sars-cov-2 transmission from bluetooth low energy measurements,” _NPJ digital medicine_ , vol. 3, no. 1, pp. 1–4, 2020.
  * [2] M. Cunche, A. Boutet, C. Castelluccia, C. Lauradoux, D. Le Métayer, and V. Roca, “On using bluetooth-low-energy for contact tracing,” Ph.D. dissertation, Inria Grenoble Rhône-Alpes; INSA de Lyon, 2020.
  * [3] D. J. Leith and S. Farrell, “Coronavirus contact tracing: Evaluating the potential of using bluetooth received signal strength for proximity detection,” _ACM SIGCOMM Computer Communication Review_ , vol. 50, no. 4, pp. 66–74, 2020.
  * [4] R. Montoliu, E. Sansano, A. Gascó, O. Belmonte, and A. Caballer, “Indoor positioning for monitoring older adults at home: Wi-fi and ble technologies in real scenarios,” _Electronics_ , vol. 9, no. 5, p. 728, 2020.
  * [5] P. Spachos and K. Plataniotis, “Ble beacons in the smart city: Applications, challenges, and research opportunities,” _IEEE Internet of Things Magazine_ , vol. 3, no. 1, pp. 14–18, 2020.
  * [6] J. F. Ensworth and M. S. Reynolds, “Ble-backscatter: Ultralow-power iot nodes compatible with bluetooth 4.0 low energy (ble) smartphones and tablets,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 65, no. 9, pp. 3360–3368, 2017.
  * [7] J. F. Ensworth, A. T. Hoang, T. Q. Phu, and M. S. Reynolds, “Full-duplex bluetooth low energy (ble) compatible backscatter communication system for mobile devices,” in _2017 IEEE Topical Conference on Wireless Sensors and Sensor Networks (WiSNet)_. IEEE, 2017, pp. 45–48.
  * [8] C. Yu, K. Kam, Y. Xu, Z. Cui, D. Steingart, M. Gorlatova, P. Culligan, and I. Kymissis, “Plant spike: A low-cost, low-power beacon for smart city soil health monitoring,” _IEEE Internet of Things Journal_ , vol. 7, no. 9, pp. 9080–9090, 2020.
  * [9] Q. D. La, D. Nguyen-Nam, M. V. Ngo, H. T. Hoang, and T. Q. Quek, “Dense deployment of ble-based body area networks: A coexistence study,” _IEEE Transactions on Green Communications and Networking_ , vol. 2, no. 4, pp. 972–981, 2018.
  * [10] B. Luo, F. Xiang, Z. Sun, and Y. Yao, “Ble neighbor discovery parameter configuration for iot applications,” _IEEE Access_ , vol. 7, pp. 54 097–54 105, 2019.
  * [11] M. Lodro, C. Smart, G. Gradoni, A. Vukovic, D. Thomas, and S. Greedy, “Near-field ber and evm measurement at 5.8 ghz in mode-stirred metal enclosure.” _Applied Computational Electromagnetics Society Journal_ , vol. 35, no. 9, 2020.
  * [12] L. Litwin, “Matched filtering and timing recovery in digital receivers,” _RF design_ , vol. 24, no. 9, pp. 32–49, 2001.
  * [13] M. Lodro, G. Gradoni, C. Smartt, A. Vukovic, D. Thomas, and S. Greedy, “Near-field image transmission and evm measurements in rich scattering environment in metal enclosure,” _Progress In Electromagnetics Research M_ , vol. 101, pp. 139–147, 2021.