These authors contributed equally to this work.

These authors contributed equally to this work.

1]Department of Electrical and Computer Engineering, University of Patras,
26504, Patras, Greece

2]Department of Electronics and Informatics, Vrije Universiteit Brussel,
B-1050, Brussels, Belgium

# ExpPoint-MAE: Better interpretability and performance for self-supervised
point cloud transformers

Ioannis Romanelis  0000-0002-2917-8705<EMAIL_ADDRESS>Vlassis Fotis
0000-0002-1212-5500<EMAIL_ADDRESS>Konstantinos Moustakas
0000-0001-7617-227X<EMAIL_ADDRESS>Adrian Munteanu
0000-0001-7290-0428<EMAIL_ADDRESS>[  [

###### Abstract

In this paper we delve into the properties of transformers, attained through
self-supervision, in the point cloud domain. Specifically, we evaluate the
effectiveness of Masked Autoencoding as a pretraining scheme, and explore
Momentum Contrast as an alternative. In our study we investigate the impact of
data quantity on the learned features, and uncover similarities in the
transformer’s behavior across domains. Through comprehensive visualizations,
we observe that the transformer learns to attend to semantically meaningful
regions, indicating that pretraining leads to a better understanding of the
underlying geometry. Moreover, we examine the finetuning process and its
effect on the learned representations. Based on that, we devise an unfreezing
strategy which consistently outperforms our baseline without introducing any
other modifications to the model or the training pipeline, and achieve state-
of-the-art results in the classification task among transformer models.

###### keywords:

Deep Learning, Self-Supervision, Transformers, Point Clouds, Explainability

##  1 Introduction

Deep learning models at large scale require adequately large labeled datasets
to be able to learn. This is clearly a limitation for deep learning in
general, since manual annotation is a very time-consuming and costly task. As
a result, it was a great breakthrough when it was discovered that models can
actually benefit from unlabeled data, by using them to design and solve
pretext tasks, in which the sample itself is the label.

This concept appears to have features resembling a real-world analog, that is,
the training happening inside a small child’s/infant’s brain. Supervision from
experts (adults) is actually responsible for a small percentage of knowledge
acquired throughout a human’s lifespan, and it mostly tailors to specific
material rather than the perception of the real world at large. Simply by
observing the behavior of other people and surrounding objects, they gain
intuitive understanding of their environment. While we have fairly
successfully replicated a form of supervised learning, self-supervision is an
area that still lags behind, despite theoretically offering the most
advantages.

Naturally, the data needs to be diverse and of sufficient size for this
process to actually yield any benefits. Fortunately, by leveraging the ocean
of unlabeled data that is freely available on the web, researchers have
managed to significantly boost the performance of their models. Great examples
of this concept can be found in both language [1] and in image [2, 3] domains,
showing great promise for a variety of potential applications in other
domains.

The most intuitive and (arguably) popular pretext task applied both in vision
and language is the so-called “fill in the gaps”. It refers to tasks where
input samples are truncated to occlude pieces of information or corrupted by
applying various types of perturbations and noise. The deep learning model is
given the damaged sample as input and is tasked to reconstruct/complete the
original. During this process, the model learns features related to the
sample’s class. These features are incredibly useful and can be leveraged to
improve the performance of the model in downstream supervised tasks, involving
potentially much smaller datasets.

As this approach became more common, more sophisticated techniques of self-
supervision were devised [4, 5, 6]. However, few studies have actually delved
into what it is that the models actually learn [7, 8, 9] and even fewer have
examined point clouds in particular [10, 11]. Coupled with several challenges
that arise in the domain shift from language/image to point clouds, we find
that this particular area is lacking.

In this study we adopt the standard transformer as our baseline model. By
”standard transformers” we refer to models that follow the architecture
presented in ViT [12], as opposed to architectures that use transformer-like
blocks. It is a highly versatile, strong architecture that has demonstrated
incredible results in both language and vision, overtaking previous state-of-
the-art models [13]. However, transformer-based works on point clouds do not
reflect this success. In this work we explore various aspects of this
architecture, in order to better understand its inner workings and find ways
to improve its performance. We argue that our findings will be of further use
in future works, since they are targeted at a widely used, general-purpose
architecture rather than a specialized one.

Figure 1: Graphical description of the two pretraining pipelines studied in
this paper, namely Masked-AutoEncoding (MAE) [14] and Momentum Contrast (MoCo)
[4]. In simple terms, MAE trains an autoencoder to reconstruct a shape with
missing parts, whereas MoCo trains two networks (Student/Teacher) to generate
approximately equal predictions for different augmentations of a data sample.

To summarize, our contributions are threefold:

  * •

We propose strategic unfreezing, a finetuning strategy that retains the
properties of the backbone, learned through pretraining, while increasing the
accuracy of our baseline both in ModelNet40 (+0.5%) and ScanObjectNN (+0.86%,
+1.73%, +0.07%), and achieving state-of-the-art results among transformer
models. (Sec. 4)

  * •

We adjust explainability tools from the image and NLP domain to work with
point clouds, with the aim of understanding the inner workings of the
transformer and the effect of pretraining. Interestingly, we uncover that with
more data, the transformer seems to learn the inductive bias of convolution,
to attend locally. (Sec. 5)

  * •

We perform comparative evaluations of the baseline Masked AutoEncoding
pretraining scheme to its main contender, Contrastive Learning, and discover
several key differences in their learned representations, such as a
complementary reliance on feature and positional embeddings respectively.
(Sec. 6)

##  2 Related Work

###  2.1 Deep Learning on Point Clouds

The domain shift from 2D to 3D came with several extra challenges, due to the
lack of grid structure and uneven density of point clouds. Early attempts
involved the application of image methods as-is, using multi-view images [15,
16] or voxels [17, 18]. Not long after that, specialized architectures
emerged. PointNet [19] pioneered point-wise MLPs and pooling for extracting
global features. PointNet++ [20] later followed a multi-scale approach by
incorporating neighborhood information. Following this paradigm, other works
create more complex kernels, taking advantage of geometric priors [21, 22, 23,
24, 25].

More recently, transformer variants have been successfully applied to the
point cloud domain. Originally introduced in [26], transformers were quickly
established as the go-to choice in NLP. Dosovitskiy et al.[12], later
introduced the vision transformer, operating on image data. This was
accomplished by tokenizing the input into patches and incorporating spatial
information through a positional embedding. This approach also allowed data
from different modalities to be used jointly [27, 28], making the transformer
the most prominent architecture in that field. Point Transformer [29] is among
the first works to apply a transformer-like architecture to point cloud data.
They apply intra-neighborhood attention to create a patch feature vector and
use pooling operations to downsample the point cloud.

In PCT [30], they extract point embeddings through a downsampling network and
apply a series of single attention transformer layers to the remaining points.
The feature maps of these layers are concatenated, and global features are
extracted through mean and max pooling. In PVT [31], the authors create
specialized attention modules for both points and voxels, utilizing the
complementary information they contain to extract better features.

Point-BERT [32] tries to generalize the plain transformer, as used both in
Image and NLP. They train a ’tokenizer’ autoencoding network, based on DGCNN
[33], to map point neighborhoods into feature vectors. During pretraining, a
transformer encoder-decoder architecture receives masked point clouds as input
and tries to reconstruct the embeddings of the ’tokenizer’. Additionally, the
classification token is trained through a contrastive loss like in [4]. Point-
MAE [14] and Point-Multiscale MAE (M2AE) [34] both utilize masked autoencoding
to pretrain their transformer backbones, by reconstructing the actual points
of the masked neighborhoods directly. Point-M2AE uses a pyramid-like backbone
that gradually downsamples the input point cloud, obtaining multi-scale
features. On the other hand, Point-MAE uses the same architecture as Point-
BERT, that is, the plain transformer.

While hierarchical architectures such as M2AE are typically associated with
slightly better performance, they require parameter tuning when transferred to
other domains or scaled to larger datasets. Since the main scope of this work
is explainability, the simplicity and cross-domain applicability of MAE are
more valuable. It can be trivially scaled by adding or removing blocks and
provides a fair ground for comparisons with image and NLP. Therefore, for our
baseline model, we choose to adopt the transformer presented in Point-MAE
[14].

###  2.2 Self supervised Learning

The idea of making use of large amounts of unlabeled data to improve a
learning model is not new, rather it has been around for decades [35, 36, 37,
38]. Its popularity did not surge until the mid-2010s [39, 2, 3] however, when
the availability of unlabeled data at scale started to increase and the
potency of hardware could finally keep up to the challenge. Of particular
importance is masked autoencoding, pioneered by [1], which revolutionized
language modeling. This paradigm was later followed by other works, such as
[40, 41, 42], that further scaled up in terms of parameter and dataset size,
achieving state-of-the-art results in the few-shot and zero-shot settings. In
[43] they experiment with pretraining by using synthetic data, artificially
injected with desirable properties.

Another line of work that has been amassing popularity recently is contrastive
learning [4, 5, 44, 45, 46, 47, 48] as a way to learn meaningful data
representations. It is accomplished by generating positive and negative sample
pairs and training the model to pull together or push them away, respectively.
The pairs are generated through data augmentations; perturbations, crops, and
transformations. A similar idea, dubbed contrastive clustering, is applied to
groups of samples instead of pairs [6, 49]. [6] in particular, eases up the
computational load by introducing learnable cluster centers. In [50], the
authors eliminate the need for negative pairs by training two networks to
produce matching feature vectors for two different views of the same sample.

Despite the scarcity of point cloud data compared to image and language data,
and the corresponding need for good self-supervision techniques in this
domain, the topic has only recently begun attracting attention. [51] proposes
to split point clouds into parts and use these parts to parameterize gaussian
mixture models. The model is then trained by using a loss that resembles
likelihood maximization. In [52] the authors perform patch-level contrastive
learning. By rotating a query patch they form positive pairs, while negative
pairs are formed by taking into account any other patch of the shape. A
similar approach is followed in [53], finding positive and negative pairs by
using an inductive model to perform pseudolabeling. In [54], a similar
approach to masked autoencoding is followed, that is, completion of occluded
point clouds acquired by taking custom viewpoints through virtual cameras.
[55] argues that performing pretext tasks on single object point clouds might
have limited benefits in real world applications. They instead sample positive
and negative pairs from complex scene scans, hoping to get a better estimate
of the target distributions, and apply a contrastive loss to train their
model.

###  2.3 Explainability in deep learning

An extremely important field in deep learning research is explainability. Deep
models are generally seen as black boxes and any attempt at designing new or
improving existing models is usually empirical. Nevertheless, there are
several tools one can use in order to better understand a model’s behavior. In
[56, 57] the authors present CKA, a similarity measure between feature
representations of two arbitrary neural networks. [7] utilizes this tool and
provides valuable insights into the differences between how CNNs and
transformers learn, while [58] tries to compare the representations of
networks with different depths and feature map sizes. Other works [11, 9]
utilize gradient-based methods in order to visualize the receptive fields or
the relevancy of input patches towards the model’s decision. We use a
combination of all the above tools to attain a complete picture of our model
and accompanying pretraining scheme.

##  3 Baseline Method, Tools, and Dataset

In this section, we briefly overview our baseline model and pretraining setup.
After training and finetuning, we assess the quality of the learned features
both qualitatively and quantitatively by using explainability tools and
measuring the accuracy in the classification task. Further details regarding
the model and the training process can be found in the appendix. We utilize
pytorch lightning111https://www.pytorchlightning.ai/ for our codebase, in
favor of clarity, transparency, and reproducibility. The code is publicly
available on github222https://github.com/VVRPanda/ExpPoint-MAE.git.

###  3.1 Masked AutoEncoders

For our baseline model, we adopt the vision transformer trained with masked
autoencoding, as presented by Pang et al. [14], which we restate here for the
sake of completeness. The transformer is comprised of 12 blocks with 6
attention heads per block. The input point cloud is split into N, possibly
overlapping patches, 60% of which are masked out. The remaining patches are
embedded into feature space via a small PointNet-like [19] network, and the
patch centroids are used to generate positional encodings. The two are added
to form the input to our transformer, which outputs a feature vector for each
patch. At this stage, masked patches are assigned a mask token and a
positional encoding and are concatenated with the rest before being fed into
the decoder. The decoder reconstructs the masked patches and a reconstruction
loss based on Chamfer distance is applied patch-wise.

Figure 2: Comparison of different unfreezing points for the backbone of the
transformer model. Unfreezing the model too early or too late can result in
suboptimal results, as the network may either ’forget’ the features learned
through pretraining or fail to acquire task-specific knowledge, respectively.

###  3.2 Explainability Tools

For explainability, one of the main points of interest in this work, we
utilize a variety of tools. The Centered Kernel Alignment criterion offers a
versatile way of comparing feature representations between two models across a
dataset, and its importance to our work cannot be overstated. It provides
insights into the differences between models that have been pretrained &
finetuned vs just finetuned, as well as models that have been pretrained using
varying datasets and or strategies. We omit the math behind CKA and refer the
interested reader to [56, 57]. Attention visualization is another commonly
used tool that helps to understand how the model correlates the point patches
based on the extracted geometric features. Finally, we compute and visualize
the receptive fields to better understand the information flow within the
network.

###  3.3 Dataset

As opposed to their image counterparts, point cloud datasets are generally
fewer and significantly smaller in size. Additionally, there are a lot of
impactful differences between point cloud datasets, including sampling
density, scanning device-specific artifacts, and shape variations in general
(real-world objects, CAD models). As a result, a model trained on a specific
dataset may not generalize to other data distributions. Our first goal towards
explainability is to see how the amount of training data might affect the
network’s performance. To this end, we concatenate the commonly used ShapeNet
[59] with the more obscure CC3D [60] (41k and 43k training samples,
respectively). CC3D contains high resolution, single-object, fine-grained CAD
shapes from arbitrary categories. To study the effects of the increased volume
of data, we conduct pretraining experiments using only ShapeNet (S) and the
concatenated dataset (C+S). We compare the two by finetuning for
classification in ModelNet40 [61] and ScanObjectNN [62] and report the results
in Tables 1, 2. In both cases, our proposed dataset helps achieve better
accuracy. For the rest of the paper, we will be using C+S for pretraining,
unless otherwise specified.

##  4 Strategic Unfreezing

Figure 3: CKA comparison of the pretrained backbone (y-axis) with versions
that have been finetuned for 300 epochs, unfreezing the backbone on various
epochs (x-axis). The first and second blocks indicate the positional and
feature embedding extractors, while the rest of the blocks correspond to the
outputs of the attention layers. High values indicate high similarity between
feature representations. The later the network is unfrozen, the higher the
similarity with the pretrained backbone, retaining the properties learned
through pretraining. In the case of (a), as done in [14], the final network
has very little similarity with the pretrained backbone, thereby nullifying
the effectiveness of pretraining.

There are several ways to approach the finetuning process of a pretrained
model. Most commonly, the pretrained backbone is frozen, that is, gradient
propagation is allowed but no weight updates happen, and a smaller
classification (or other) head is trained to map the learned features to
logits. This method is based on the assumption that the backbone has already
learned a robust representation of the data and is capable of separating the
class clusters effectively. However, the learned data representation often
differs significantly from the data used in the downstream tasks. A great
example of this is pretraining on synthetic data and using the model in real-
world applications.

Another approach is to perform task-specific adjustments to the model by
training both the backbone and the added head end-to-end using a small
learning rate. However this contains an important caveat, as the random
initialization of the classification head’s weights can cause the weights of
the backbone to be perturbed in an unintended direction, effectively
destroying the learned features and representations.

In order to properly evaluate these approaches we perform comparative the
following experiments.

  * •

We test the first method by measuring the accuracies in ModelNet40
classification (Figure 2). We see that when keeping the backbone frozen and
only relying on the classification head, the accuracy is vastly inferior
compared to the backbone being unfrozen at various intervals. This is because
no task-specific knowledge has been incorporated into the backbone.

  * •

In Figure 3(a), we test the second method. We compare the pretrained backbone
that has been finetuned without freezing and the pretrained backbone without
finetuning. It is evident that there is hardly any similarity between them,
except for the very early layers. This arguably defeats the purpose of
pretraining, since very few of the valuable properties are retained in the
final network. This deficiency is also reflected in the final accuracy score,
as seen in Figure 2.

  * •

Finally, in Figure 2 we note the accuracy of the model that has not been
pretrained. It exhibits the lowest accuracy among all models that have been
pretrained, proving the effectiveness of pretraining.

Before diving into what the model actually learns (section 6), we first
propose a two-stage training scheme that retains the pretrained network’s
characteristics and obtains superior performance. First, we train the
classification head for the majority of epochs, ensuring that it has learned
to separate the clusters formed via the backbone. Finally, we jointly train
both of them for the remainder of the epochs. In order to figure out the
appropriate step to unfreeze the backbone, we perform several experiments and
compare the feature representations with the pretrained network each time
(Figure 3). Additionally, we measure the best and the convergence accuracy for
each of these models in the classification task on ModelNet40 (Figure 2). It
is evident that the earlier the backbone is unfrozen, the more the pretrained
backbone’s features are distorted, and the final accuracy is compromised.

In order to identify the proper unfreezing epoch, one must consider the data
and the task. For ModelNet40 classification, as a subset of ShapeNet, similar
samples were seen by the model during pretraining. As a result, unfreezing
during the late stages is appropriate. On the other hand, when finetuning on
ScanObjectNN, that consists of real scanned objects, it is wiser to unfreeze
earlier so that the backbone can become accustomed to the new data
distribution. Finally, when finetuning on the hardest variant of the
ScanObjectNN which includes other modifications, such as rotations, we found
that the most effective way is to perform an extra domain adapaptation
pretraining step, before finetuning. The results of this step can be seen in
Table 2 in the SoNN entry. More details are presented in the Appendix.

Designing and training a model is a difficult and iterative process, that is
based heavily on parameter tuning. We believe that with our method researchers
will be able to make more informed decisions that will allow them to utilize
their models to their maximum capacity.

Table 1: Comparison of different transformer based models and pretraining
schemes with and without using voting on the ModelNet40 test set.

Model | w/o voting | voting
---|---|---
Point-BERT [32] | 92.7 | 93.2
MaskPoint [63] | - | 93.8
Point-MAE [14] | 93.2 | 93.8
PCT-2L [30] | 93.2 | -
PCT-3L [30] | 93.4 | -
Point-M2AE [64] | 93.4 | 94.0
Ours (S) | 93.6 | -
Ours (C+S) | 93.7 | 94.2

††footnotetext: Bold and underlined denotes 1st and 2nd highest

Table 2: Classification results on ScanObjectNN. The HARD variant refers to
PB-T50-RS that is the most challenging one.

Model | OBJ-BG | OBJ-ONLY | HARD
---|---|---|---
Point-BERT | 87.43 | 88.12 | 83.07
MaskPoint | 89.3 | 89.7 | 84.6
Point-MAE | 90.02 | 88.29 | 85.18
Point-M2AE | 91.22 | 88.81 | 86.43
Ours (unfr 200) | 90.88 | 89.33 | 84.98
Ours (unfr 250) | 87.44 | 90.02 | 83.80
Ours (SoNN111SoNN indicates that the model has been pretrained for a few epochs in the ScanObjectNN-hard dataset.) | 90.36 | 89.67 | 85.25

††footnotetext: Bold and underlined denotes 1st and 2nd highest

##  5 Explainability Study

Figure 4: Attention Visualization of the classification token for each block
(1-12), averaged across heads (brighter = higher score). Although the
classification token’s attention score towards itself cannot be visualized, it
has the highest value in all cases. This score is included in the
normalization process, so that the relative scale between them is visible.
Figure 5: Attention Visualization of the classification token for each block
(1-12), averaged across heads for a finetuned model without pretraining. It is
evident the locations it attends to do not follow any recognizable pattern. In
each layer, the attention is focused on a specific location, which is often
the same between layers (e.g. head in layers 0,5,7,10).

###  5.1 Attention Visualization

In this section, we will use the finetuned backbone, pretrained with MAE, and
finetuned with the strategy outlined in the previous section. This is because
it retains the useful properties acquired through pretraining, but also has a
trained classification token, as opposed to [14], which uses max-average
pooling. By visualizing the attention matrices at each block we can obtain
meaningful information regarding how the model correlates various parts of the
shape. This information can help identify patterns that indicate whether the
pretraining procedure was successful, such as semantic correlation, symmetry,
and locality.

We observe that in all input shapes, the first attention block is always
global, while the rest gradually shift from attending to specific regions to
attending globally. Sharp geometric features in particular attract high
attention scores in most layers (Figure 4). This behavior is quantitatively
verified through attention distances in the following section.

It is important to note that finetuned models that have not been pretrained do
not share this trait. As can be seen in figure 5, although the classification
token attends to locations with valid semantic meaning, the behavior is
erradic, repetitive and there is no clear transition from specific areas to
the whole shape.

###  5.2 Attention Distance

Following the paradigm of [7], we study the sorted attention distance. The
mean Attention Distance (mAD) of layer $l$ for a head $h$ is calculated as
follows:

$mAD_{l,h}=\frac{1}{N^{2}}\sum_{ij}A_{l}\odot D$

Where $A_{l}^{N\times N}$ is the attention matrix of layer $i$ and $D^{N\times
N}$ is the distance matrix, where each entry $d_{i,j}$ is the Euclidean
distance between centers $i$ and $j$ and the $\odot$ symbol denotes the
Hadamard product. The product is averaged across all entries to obtain a mean
attention-distance value, which is also averaged across the validation set of
ModelNet40.

Based on the graphs in Figure 6, it can be inferred that with the increase in
the amount of training data, the earlier layers tend to focus more locally
with certain attention heads, while others still maintain a global focus. This
observation suggests that the transformer network may begin to learn the
inductive bias of the convolution, that is, attending to local features, while
still retaining the capability to aggregate global information, thereby
resulting in more comprehensive representations. This mitigates the general
struggle of self-attention to model local relationships [65], without
requiring any modifications to the architecture.

Interestingly, our findings are consistent with those of [7, 66, 67],
indicating that transformers may exhibit akin behaviors across domains. A
noticeable difference is that in the case of ShapeNet (Figure 6b) the first
two layers seem attend globally with all of their attention heads, whereas
when CC3D is added to the pretraining data, this phenomenon is only observed
in the first layer (Figure 6c). We assume that the network requires a rough
understanding of the entire shape before opting to extract local features.
This demeanor should be studied further if point cloud datasets at large scale
become available in the future.

Figure 6: Attention Distance of the attention heads (distances are sorted in
ascending order for clarity). The distances are averaged across the entire
validation set of ModelNet40. We see that as the amount of data increases,
early layers of the network begins to attend locally, while higher layers
incorporate global information, exhibiting similar behaviour to ViT in the
image domain[12] Notably, a key difference is observed in the very first
layer, where all heads consistently attend globally.

###  5.3 Effective Receptive Fields

We have established that by using our unfreezing strategy, the final network
shares many similarities with the pretrained network. In order to understand
their differences (trained for pretext versus downstream tasks), we visualize
the effective receptive fields [7]. We accomplish this by selecting a patch
and propagating the gradients backwards from its feature vector to its
embedding, including both feature and positional information. We then
visualize the norm of said gradients and present the results in figure 7.

We notice that the pretrained model, trained explicitly for reconstruction,
looks strictly at neighboring as well as semantically similar patches, since
they contain the most relevant information. These areas have the highest
impact on the finetuned model as well, but its receptive field is noticeably
wider. We believe this is because the classification task requires global
features.

Figure 7: Effective receptive fields of the patch of interest (colored in
red). We compute the gradient of the feature representations generated by the
last layer with respect to the input patches. Each point is colored based on
the magnitude of the gradient of its corresponding patch. In the right
variant, that is finetuned for classification, the model retrieves
infrormation from the entire shape, as it tries to incorporate class
information. Conversly, the left variant, that is the pretrained backbone,
focuses mostly to local and symmetric parts, aligning with the task it was
trained for, that is reconstruction.

##  6 Contrastive Learning

Having delved into the inner workings of masked autoencoding and the
properties associated with it as a pretraining scheme, we need to evaluate its
effectiveness against other pretraining methods in the literature. We choose
what we believe to be the next most popular one, contrastive learning.
Impressed by its performance in the image domain, we follow the approach of
Momentum Contrast (MoCo) [4], which we extend to work with 3D data.

In MoCo, two copies of the network are trained simultaneously, one through
backpropagation and the other using a momentum update rule. Positive pairs are
created by feeding crops of the same shape with different augmentations to
these networks, while previous activations of the momentum network act as
negative pairs. We follow the same strategy and create our crops by using a
mix of gaussian noise, anisotropic scaling, rotations, and random point
dropouts.

By finetuning the models pretrained with MoCo, we observe that, although it
improves the original model without pretraining, there is a significant gap in
accuracy compared to its MAE counterpart (Table 3). This, however, does not
rule out the possibility that other variants of contrastive learning might be
more effective. Instead, it suggests that a well-designed pretraining scheme
will most likely offer a significant performance boost to any baseline model.

Figure 8: Attention scores of the classification token, averaged across all heads, pretrained using the MoCo pipeline. We visualize a subset of the network layers, [1, 3, 7, 8] left to right. While in the early layers the CLS token attends to meaningful parts of the shape, exhibiting a tendency to attend at symmetric parts, from layer 8 and onwards it attends uniformly to the whole shape. Figure 9: Comparison of Attention Distances achieved through Masked AutoEncoding (MAE) [14] and Momentum Contrast (MoCo) [4] pretraining schemes. We observe that the setup of MoCo is insufficient to enable the transformer to learn to attend locally. Layers 10 and 11 in MoCo have the exact same attention distance.  Figure 10: CKA comparisons between the two pretrained backbones (MAE and MoCo pipelines) and themselves. The first two entries correspond to the positional and feature embedding respectively while the rest to the outputs of the attention blocks. We observe little resemblance between the representations learned through the different pretraining pipelines. Interestingly, the final feature representations obtained through the MAE pipeline appear to rely more on the features learned within the layers of the network, showing little similarity with the positional embeddings. Conversely, the MoCo pipeline demonstrates a higher reliance on the positional embedding.  Table 3:  Network Accuracy Achieved with different pretraining schemes  Model | No Pre. | MAE | MoCo | MAE & MoCo
---|---|---|---|---
Accuracy | 91.1 | 93.7 | 92.2 | 91.8

As opposed to masked autoencoding, an interesting property of contrastive
learning is that it allows the classification token to be trained during the
pretraining stage. In fact, by examining the attention scores, we notice that
the classification token attends to characteristic areas of the input shape,
and exhibits symmetry, by attending to shape parts with similar semantics
(Figure 8). We also observe that from layers 8 and onward, the CLS token
attends uniformly to the whole shape. This is verified by Figure 9b where
layers 10 and 11 have the exact same attention distance across all heads.
Finally, we see that through MoCo pretraining the backbone does not learn to
attend to neighboring patches, indicating that the training scheme might
require more data or alterations to converge. For the sake of completeness, we
also tested a MoCo and MAE hybrid (details in the supplementary material).

Furthermore, useful insights can be extracted by comparing the two pretraining
schemes against each other, as well as themselves, to get an idea of how the
information is distributed throughout the layers. The results can be seen in
Figure 10.

We observe that middle layers in MAE seem to have similar feature
representations, while there is some similarity between input and output
layers. However, the same does not seem to apply to MoCo, where each layer has
common representations only with a few of its preceding and succeeding layers,
creating a chain-like representation. Astoundingly, the layers seem to
gradually lose their similarity with the early feature embedding and rely more
and more on the positional embedding, whereas in MAE, the feature embedding is
the most dominant in all layers. Naturally, MAE and Moco are based on
fundamentally different principles, so they bear very little resemblance and
only in the middle layers.

##  7 Ablation Studies

In the ablation study section we focus on two separate topics that are the
masking ratio and the intermediate representations of the transformer layers.

Firstly, we focus of the reconstruction abilities of the backbone, trained
through the MAE pipeline. Point-MAE has already studied the effect of masking
in the final accuracy, concluding that a 60% masking is the optimal choice. In
our experiments we want to test the reconstruction ability of the network when
masking the input with a higher ratio. As depicted in Figure 11, even in the
extreme case when 90% of the model is masked out, the network can provide a
decent reconstruction. We believe that this is because positional embeddings
of the masked patches are given to the decoder, providing useful hints
regarding the position of individual parts.

Original 60% 80% 90% 95%

Figure 11: Reconstruction results using different masking ratios. The network
is pretrained, using MAE, with a 60% masking ratio. We test the reconstruction
ability of the system when we provide as input a point cloud with higher
masking ratios. Input and reconstructed patches are shown in red and cyan,
respectively.

The classification token in the transformer model progressively accumulates
information in order to get a high-level understanding of the input shape. In
our second experiment we try to evaluate the representation attained through
intermediate layers. In Figure 12 we use the same pretrained backbone, but use
only a subset of its transformer blocks. Numeric results are presented in
Table 4 . When removing up to 5 layers, the model manages to retrain an
accuracy above the baseline. Interestingly, that’s the same layer in which the
transition from location-specific to uniform attention becomes clearer. This
indicates that the class cluster separation is possible even in earlier
layers, and any additional information simply contributes to refining these
clusters.

Figure 12: Accuracy graph of truncated copies of the same model. Each entry
specifies the layer from which the feature vector is derived (all succeeding
layers are discarded for that run). We observe that the model maintains higher
accuracy than the baseline, even with a fraction of the original model’s
parameters. Table 4: Classification results on ModelNet40 when using the CLS
token from previous layers as global descriptor of the 3D shapes. The network
is trained with our unfreezing strategy, that is training only the cls head
for the first 250 epochs.

Model | Point-BERT | Point-MAE | Ours(full depth)
---|---|---|---
Acc. | 92.70 | 93.19 | 93.70

Without Layers | -5 | -4 | -3 | -2 | -1
---|---|---|---|---|---
Acc. | 93.23 | 93.56 | 93.48 | 93.40 | 93.52

##  8 Discussion

Using our proposed unfreezing strategy, we manage to outperform our baseline
model, as well as models that use transformers in a similar way. We do not use
voting unless explicitly mentioned for fair comparisons, nor do we include
random transforms in the validation set. The method we suggest combined with
our learning rate scheduling policy yield stable training with very close
convergence points and weaker fluctuations.

Given sufficient training time, task-specific knowledge must be incorporated
into the backbone in order to improve performance, but only when the head
network has been trained enough for the weights to move in the appropriate
direction. The specific intervals can vary depending on the task and dataset.
For instance, data with background obtained from scans (in ScanObjectNN)
require more train time for the backbone, so the optimal unfreezing point is
around 200 epochs instead of 250. We also find that when the data distribution
is fairly different from the pretraining data, it is more beneficial to
pretrain our network for a few epochs in the new distribution instead of
unfreezing the backbone earlier (Table 2).

In our explainability experiments, we observe that point neighborhoods with
sharp geometric features are generally more likely to be attended to by the
classification token. This is an indicator of good pretraining, as sharp
features are more relevant descriptors of the sample’s class. We also notice
that the transition from local to global seems characteristic of transformers.
This property is experimentally backed in the image domain [7], and our own
experiments confirm that this holds true in point clouds as well.

Finally, we remark that doubling the volume of data causes more attention
heads to attend locally and introduces accuracy boosts in the classification
task, albeit small. In [7] they observe similar behavior but with even greater
impact, which is likely attributed to the significantly larger scale of both
the datasets and models. Nevertheless, in both cases it suggests that local
information aggregation is a property linked with good performance.

##  9 Future Work

Reasoning about the learned representations requires depth and repeating all
finetuning experiments for each new pretrained network. In future work, we
plan on experimenting and associating the properties of pretrained networks
with specific downstream tasks. We theorize that due to their distinct
characteristics, some pretraining schemes may offer greater advantages for a
specific downstream task than others. Sample body text. Sample body text.
Sample body text. Sample body text. Sample body text. Sample body text. Sample
body text. Sample body text.

## Appendix A

###  A.1 MoCo & MAE Hybrid Pretraining

We devise a pretraing sceme that combines both core concepts from MoCo and
MAE. The main idea is that the reconstruction loss from MAE will help the
backbone converge faster, being a more straightforward task, and the
contrastive loss will help in training the CLS token and slightly contribute
to the backbone features as well. The scheme is realized through a weighted
sum of both losses, appropriately scaled so that they are in the same order of
magnitude.

Specifically, the standard reconstruction loss presented in [33] is applied
($L_{rec}$) and a contrastive loss, $L_{con}$, as in [20] is added, using a
queue size of $4096$ encoded samples. The final loss function is realized as:

$L=L_{rec}+w_{c}\cdot L_{con}$

Where $w_{c}$ was set to $10^{-2}$ to bring the two losses to the same order
of magnitude. We also experimented with a smaller weight($10^{-3}$), as the
MoCo loss is responsible mainly for the CLS token. Both approaches, however,
produced similar results.

Although the hybrid model’s accuracy reaches an unimpressive $91.8$% score, we
believe that with further research it is possible to create a model that makes
the best of both worlds (masked autoencoding and contrastive learning).

Figure 13: Confusion matrix visualization for the validation set of
ModelNet40. Each row has been normalized through division by the number of
samples of that class.

###  A.2 Strategic unfreezing

As briefly discussed in the main paper (due to space constraints), the
finetuning data from ModelNet40 have already been seen during the pretraining
stage and therefore the distribution of said data has been captured by the
backbone. Therefore, it is sufficient to only train the backbone for the last
50 epochs, where the learning rate is small enough so that no overblown weight
updates happen.

The ScanObjectNN is probably an even better indicator of this strategy. We
observe that in table 2, when training on objects without background the same
unfreezing point yields the best results. On the other hand, when background
is included, unfreezing at 200 epochs is optimal and obtains even better
accuracy. While the higher accuracy can easily be attributed to the extra
information contained in the background itself, it is important to notice that
the background changes the data distribution significantly. This change,
coupled with the fact that the data has been obtained through scanning, leads
us to think that 50 epochs is no longer enough for the backbone to adjust to
the new distribution.

This argument is further verified by finetuning on the hard variant of
ScanObjectNN. Sure enough, 100 epochs of training with the unfrozen backbone
makes a staggering $+1.18\%$ difference in accuracy compared to 50 epochs,
however it is still far from the optimal, which is to pretrain the network for
a few epochs on this dataset. In our experiments, 40 epochs of pretraining are
adequate to reach an accuracy of $85.25\%$, that surpasses our baseline Point-
MAE.

###  A.3 Experimental Setup

###  A.4 Optimization and learning rate policy

For both our optimizer and learning rate we follow one of [33]’s suggestions.
For our finetuning we use the AdamW optimizer with a peak learning rate of
$\lambda=5\cdot 10^{-4}$ and a weight decay of $W_{d}=0.05$. We utilize the
cosine annealing learning rate scheduler, updating during every step for a
total training of 300 epochs. A linear warmup is also performed, with an
initial learning rate $\lambda_{i}=10^{-6}$ for a duration of 10 epochs in
total. The exact same setup is also used for pretraining, with the exception
of peak learning rate, being $\lambda=10^{-3}$. The batch size is 128 for
pretraining and 32 for finetuning.

###  A.5 Data Preprocessing

All data samples are normalized to be inside the unit sphere. For data
augmentations we apply random sampling of the point clouds and anisotropic
scale, following the paradigm of our baseline.

## Appendix B

###  B.1 Class Clusters

Figure 14: Visualization of samples from problematic classes of ModelNet40.
First, second and third rows correspond to ”flower pot”, ”plant” and ”vase”
respectively.

In order to get a better idea of the causes behind misclassified objects we
present a confusion matrix (figure 13) generated from samples of ModelNet40
validation set. The most striking entry is the 15th category, ”flower pot”,
being mistaken for classes 26, ”plant”, and 37, ”vase”. These categories
contain an unreasonably large amount of overlap, e.g. pots have plants, plants
have pots and a lot of pots have a vase-like shape (figure 14). Therefore, it
is hardly a surprising result that these objects are being misclassified, and
this particular error is not a good descriptor of the model’s performance.

Figure 15: Clustering performed by the model in the feature space for 10
arbitrary classes. The model has been pretrained using the MAE pipeline. T-SNE
is used to project the models two the 2-dimensional space. (A) A sample from
the airplane class is placed near the guitar cluster. An ”excusable” choice as
both share a lengthy shape and sharp protruding parts. (B) Two shape from the
person and plant classes are placed further from their clusters, and near each
other, as they have a near non-existent depth. (C) Two uniquely shaped lamps
have been assigned to a cluster of their own.

Figure 16: Clustering performed by the model in the feature space for 10
arbitrary classes. The model has been pretrained using the MoCo pipeline.
T-SNE is used to project the models two the 2-dimensional space. (A) A tall
cactus is clustered together with the people class. (B1), (B2), Two lamps with
distinct characteristics have been assigned to a separate cluster. (C) A plant
with hanging leaves is placed along with the table cluster, most likely
because the leaves share a similar structure with table legs.

In figures 16 and 16 (MAE and MoCo respectively) we visualize the feature
vectors of 15 shapes from 10 arbitrary classes randomly sampled from the
validation set of ModelNet40 and projected onto the 2d space by using T-SNE.
Naturally, the test accuracy is the sole indicator of how well the clustering
is performed. However, by using this technique we are hoping to understand
whether the clustering/classification operates within the realms of human
logic. For this purpose, we are mostly interested in the misclustered samples.

All in all, these mistakes seem plausible from a human perspective. In
particular, the model seems to be ”fooled” by objects that have similar
height/width or are composed by roughly equivalent parts. This is hardly a
fault however, as it only decreases the score on the ModelNet40 benchmark,
whilst achieving greater generalization ability.

## References

  *   * Devlin et al. [2019] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019)
  * Krizhevsky et al. [2017] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Communications of the ACM 60, 84–90 (2017)
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * He et al. [2020] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * Chen et al. [2020] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607 (2020)
  * Caron et al. [2020] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: Conference on Neural Information Processing Systems (NIPS), vol. 33, pp. 9912–9924 (2020)
  * Raghu et al. [2021] Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? In: Conference on Neural Information Processing Systems (NIPS), vol. 34, pp. 12116–12128 (2021)
  * Nguyen et al. [2021] Nguyen, T., Raghu, M., Kornblith, S.: Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In: International Conference on Learning Representations (ICLR) (2021)
  * Chefer et al. [2021] Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visualization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791 (2021)
  * Zhang et al. [2019] Zhang, B., Huang, S., Shen, W., Wei, Z.: Explaining the pointnet: What has been learned inside the pointnet? In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 71–74 (2019)
  * Tayyub et al. [2022] Tayyub, J., Sarmad, M., Schönborn, N.: Explaining deep neural networks for point clouds using gradient-based visualisations. In: Proceedings of the Asian Conference on Computer Vision (ACCV), pp. 2123–2138 (2022)
  * Dosovitskiy et al. [2021] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR (2021)
  * Chen et al. [2021] Chen, X., Hsieh, C.-J., Gong, B.: When vision transformers outperform resnets without pretraining or strong data augmentations. arXiv preprint arXiv:2106.01548 (2021)
  * Pang et al. [2022] Pang, Y., Wang, W., Tay, F.E.H., Liu, W., Tian, Y., Yuan, L.: Masked autoencoders for point cloud self-supervised learning. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 604–621 (2022)
  * Su et al. [2015] Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3d shape recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 945–953 (2015). https://doi.org/10.1109/ICCV.2015.114
  * Qi et al. [2016] Qi, C.R., Su, H., Nießner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view cnns for object classification on 3d data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5648–5656 (2016). https://doi.org/10.1109/CVPR.2016.609
  * You et al. [2018] You, H., Feng, Y., Ji, R., Gao, Y.: Pvnet: A joint convolutional network of point cloud and multi-view for 3d shape recognition. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1310–1318 (2018). https://doi.org/10.1145/3240508.3240702
  * Maturana and Scherer [2015] Maturana, D., Scherer, S.: Voxnet: A 3d convolutional neural network for real-time object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922–928 (2015). https://doi.org/10.1109/IROS.2015.7353481
  * Qi et al. [2017a] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * Qi et al. [2017b] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: Conference on Neural Information Processing Systems (NIPS), pp. 5105–5114 (2017)
  * Thomas et al. [2019] Thomas, H., Qi, C.R., Deschaud, J.-E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: Flexible and deformable convolution for point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
  * Liu et al. [2019] Liu, Y., Fan, B., Xiang, S., Pan, C.: Relation-shape convolutional neural network for point cloud analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * Ma et al. [2022] Ma, X., Qin, C., You, H., Ran, H., Fu, Y.: Rethinking network design and local geometry in point cloud: A simple residual mlp framework. International Conference on Learning Representations (ICLR) (2022)
  * Xiang et al. [2021] Xiang, T., Zhang, C., Song, Y., Yu, J., Cai, W.: Walk in the cloud: Learning curves for point clouds shape analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 915–924 (2021)
  * Xu et al. [2018] Xu, Y., Fan, T., Xu, M., Zeng, L., Qiao, Y.: Spidercnn: Deep learning on point sets with parameterized convolutional filters. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 90–105 (2018)
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Conference on Neural Information Processing Systems (NIPS), vol. 30 (2017)
  * Shin et al. [2022] Shin, A., Ishii, M., Narihira, T.: Perspectives and prospects on transformer architecture for cross-modal tasks with language and vision. International Journal of Computer Vision (IJCV) (2022)
  * Wu and Li [2023] Wu, X., Li, T.: Sentimental visual captioning using multimodal transformer. International Journal of Computer Vision (IJCV) (2023)
  * Zhao et al. [2021] Zhao, H., Jiang, L., Jia, J., Torr, P.H.S., Koltun, V.: Point transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16259–16268 (2021)
  * Guo et al. [2021] Guo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R.R., Hu, S.-M.: Pct: Point cloud transformer. Computational Visual Media 7, 187–199 (2021)
  * Zhang et al. [2022] Zhang, C., Wan, H., Shen, X., Wu, Z.: Pvt: Point-voxel transformer for point cloud learning. International Journal of Intelligent Systems (2022)
  * Yu et al. [2022] Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., Lu, J.: Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19313–19322 (2022)
  * Wang et al. [2019] Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG) (2019)
  * He et al. [2022] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000–16009 (2022)
  * LeCun et al. [1989] LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., Jackel, L.: Handwritten digit recognition with a back-propagation network. Conference on Neural Information Processing Systems (NIPS) (1989)
  * Ng and Jordan [2002] Ng, A., Jordan, M.: On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Conference on Neural Information Processing Systems (NIPS) (2002)
  * Raina et al. [2007] Raina, R., Battle, A., Lee, H., Packer, B., Ng, A.Y.: Self-taught learning: Transfer learning from unlabeled data. International Conference on Machine Learning (ICML) (2007)
  * Vincent et al. [2008] Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: International Conference on Machine Learning (ICML), pp. 1096–1103 (2008)
  * LeCun et al. [2015] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
  * Brown et al. [2020] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Conference on Neural Information Processing Systems (NIPS), vol. 33, pp. 1877–1901 (2020)
  * Radford et al. [2018] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2018)
  * Zhai et al. [2022] Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: Lit: Zero-shot transfer with locked-image text tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * Kataoka et al. [2022] Kataoka, H., Okayasu, K., Matsumoto, A., Yamagata, E., Yamada, R., Nakamura, N.I.A., Satoh, Y.: Pre-training without natural images. International Journal of Computer Vision (IJCV) (2022)
  * Chen et al. [2021] Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9640–9649 (2021)
  * Chen et al. [2020] Chen, X., Fan, H., Girshick, R., He, K.: Improved Baselines with Momentum Contrastive Learning
  * Caron et al. [2021] Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
  * Jiang et al. [2021] Jiang, L., Shi, S., Tian, Z., Lai, X., Liu, S., Fu, C.-W., Jia, J.: Guided point contrastive learning for semi-supervised point cloud semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6423–6432 (2021)
  * Oord et al. [2018] Oord, A.v.d., Li, Y., Vinyals, O.: Representation Learning with Contrastive Predictive Coding. arXiv (2018)
  * Caron et al. [2018] Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
  * Grill et al. [2020] Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent - a new approach to self-supervised learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Conference on Neural Information Processing Systems (NIPS), vol. 33, pp. 21271–21284 (2020)
  * Eckart et al. [2021] Eckart, B., Yuan, W., Liu, C., Kautz, J.: Self-supervised learning on 3d point clouds by learning discrete generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8248–8257 (2021)
  * Du et al. [2021] Du, B., Gao, X., Hu, W., Li, X.: Self-contrastive learning with hard negative sampling for self-supervised point cloud learning. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 3133–3142 (2021)
  * Cheraghian et al. [2022] Cheraghian, A., Rahman, S., Chowdhury, T.F., Campbell, D., Petersson, L.: Zero-shot learning on 3d point cloud objects and beyond. International Journal of Computer Vision (IJCV) (2022)
  * Wang et al. [2021] Wang, H., Liu, Q., Yue, X., Lasenby, J., Kusner, M.J.: Unsupervised point cloud pre-training via occlusion completion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9782–9792 (2021)
  * Xie et al. [2020] Xie, S., Gu, J., Guo, D., Qi, C.R., Guibas, L., Litany, O.: Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 574–591 (2020)
  * Cortes et al. [2012] Cortes, C., Mohri, M., Rostamizadeh, A.: Algorithms for learning kernels based on centered alignment. Proceedings of the 36th International Conference on Machine Learning 13, 795–828 (2012)
  * Kornblith et al. [2019] Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. Journal of Machine Learning Research 97, 3519–3529 (2019)
  * Nguyen et al. [2021] Nguyen, T., Raghu, M., Kornblith, S.: Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In: International Conference on Learning Representations (ICLR) (2021)
  * Chang et al. [2015] Chang, A.X., Funkhouser, T.A., Guibas, L.J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: Shapenet: An information-rich 3d model repository. CoRR (2015)
  * Cherenkova et al. [2020] Cherenkova, K., Aouada, D., Gusev, G.: Pvdeconv: Point-voxel deconvolution for autoencoding CAD construction in 3d. In: 2020 IEEE International Conference on Image Processing (ICIP) (2020)
  * Wu et al. [2015] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes, pp. 1912–1920 (2015). https://doi.org/10.1109/CVPR.2015.7298801
  * Uy et al. [2019] Uy, M.A., Pham, Q.-H., Hua, B.-S., Nguyen, D.T., Yeung, S.-K.: Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In: International Conference on Computer Vision (ICCV) (2019)
  * Liu et al. [2022] Liu, H., Cai, M., Lee, Y.J.: Masked discrimination for self-supervised learning on point clouds. Proceedings of the European Conference on Computer Vision (ECCV) (2022)
  * Zhang et al. [2022] Zhang, R., Guo, Z., Gao, P., Fang, R., Zhao, B., Wang, D., Qiao, Y., Li, H.: Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint arXiv:2205.14401 (2022)
  * Zho et al. [2023] Zho, J., Wang, P., Tang, J., Wang, F., Liu, Q., Li, H., Jin, R.: What limits the performance of local self-attention? International Journal of Computer Vision (IJCV) (2023)
  * Steiner et al. [2021] Steiner, A., Kolesnikov, A., , Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 (2021)
  * Zhang et al. [2023] Zhang, Q., Xu, Y., Zhang, J., Tao, D.: Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision (IJCV) (2023)


# Quantum Secrecy in Thermal States III

Anne Ghesquière  Benjamin T. H. Varcoe  Quantum Experimental Group, School of
Physics and Astronomy, University of Leeds, Leeds LS2 9JT, United Kingdom
<EMAIL_ADDRESS>
###### Abstract

In this paper we expanded the security of a central broadcast protocol using
thermal states to the case in which the eavesdropper controls the source.
Quantum secrecy in a continuous variable central broadcast scheme is
guaranteed by the quantum correlations present in thermal states arising from
the Hanbury Brown and Twiss effect. This work allows for a method of key
exchange in which two parties can agree a key as long as both can detect the
same source and they are within the spatial coherence length of the source.
This is important because it allows quantum secure key exchange with only
minimal changes to existing infrastructure.

Expanding Quantum Key Distribution outside of the normal paradigm of single
photons sent from Alice to Bob is a current challenge that is particularly
relevant to traditional networks where upgrading to single photons optical
fibers is impractical. In a series of papers sakuya:2017_1 ; sakuya:2018_1 ,
we have been developping a model for thermal states key distribution that is
especially relevant to this challenge because it would allow QKD in a
microwave network. In these papers we demonstrate that, rather than being a
source of ‘noise’Weedbrookpra:2012 , thermal states are a source of discord
that can be used as a resource for quantum computingPirandola:2014 .

In particular the model that we consider is a central broadcast Maurer:1999
model rather than a point-to-point model BB:84 . In central broadcasting, a
single source transmits a signal to several receivers, which allows the
receivers to detect Hanbury Brown and Twiss noise correlations. A common
example of a central broadcast channel is that of a satellite beaming in free-
space to several receiving antennas and the information received depends on
the channel noise and the reliability of the detectors.

This set-up is easy to implement in the laboratory, using the Hanbury Brown
and Twiss (HBT) interferometer HBT:1956_1 ; HBT:1956_2 . A source shines onto
a beamsplitter, which divides the signal into two beams, one detected by
Alice, the other by Bob. When the radiation is thermal, the signal is composed
of bunched pairs. The action of the beamsplitter is to split the pairs, to
exploit the correlations within. This is known as the Hanbury Brown and Twiss
effect, and the correlations are known to exhibit discord ragy:2013 , which is
a necessary condition for QKD Pirandola:2014 .

Therefore, far from being dismissable as mere noise Weedbrookpra:2012 , the
discord present in thermal photons serves as a resource in QKD. In ref.
sakuya:2017_1 ; sakuya:2018_1  we have described a central broadcast scheme
(CBS) where a source (controlled by Alice) emits a signal which is divided and
shared between the two legal parties. There are two channels open to the
eavesdropper in such a protocol, the lower channel between the beamsplitter
and Bob, and the higher channel between the source and the beamsplitter. In
sakuya:2017_1 , we let Eve access the lower channel; in sakuya:2018_1 , we
gave her access to the higher channel. In both cases, the secrecy arises from
the correlations present in the bunched pairs.

Only one step remains to fully establish our CBS as a valid quantum key
distribution, and that is the requirement of trust in the source. The security
of the source is generally considered paramount to quantum key distribution
protocols. Eavesdropping is allowed on the way from source to the legal
parties, but the source cannot be untrusted.

In a CBS, the source can be either under (for instance) Alice’s control, or
out of either legal parties’ control. So far, we have considered that it was
under Alice’s control. This has allowed us to make the assumption that the
radiation transmitted to Alice and Bob is thermal. Technically, no such
assumption can be made if the source is out of Alice’s or Bob’s control. It is
natural to wonder then, if the protocol remains secure in this case.

Let us recall the protocol briefly, as illustrated on Figure 1. A source
produces states which are sent onto a beamsplitter $\eta_{ab}$, which splits
it into one part which goes to Alice and one part that goes to Bob. Alice and
Bob measures their signal separately. They perform a $g^{(2)}(0)>1$ check on
their data, to ensure its thermality and therefore ascertain the presence of
correlations. Once the presence of correlations is confirmed, they proceed as
per usual, with reconciliation and privacy amplification.

The $g^{(2)}(0)$-check is the pivot here, because it is when Alice and Bob
verify that the signal they have detected is indeed, correlated. If it is not,
they must begin anew. This limits the eavesdropper in her actions, and forces
her to emit a thermal signal. If she wants to beat the $g^{(2)}(0)$-check and
retain some information about the signal which Alice and Bob will detect, Eve
must create a signal which will correlate her to both. Thankfully, the
likelyhood of this is very remote for three reasons, firstly because three-way
correlations, where the bunched radiation is not a pair but a triplet, is a
statistically rare occurence. Secondly, Eve would face the time limitation
introduced by the coherence length of the radiation. Simply put, Eve must
somehow find herself in a physical position where she can force the satellite
to produce a sufficient number of triplets for her to be correlated to Alice
and to Bob, and as well, detect her share of the signal within the coherence
time. Lastly, the splitting at the beamsplitter is stochastic and therefore
Eve cannot predict it.

This is not without caveats for Alice and Bob. Since they can have no trust in
the source, they must now reconcile against her, at the cost of a lower key
rates. We are considering here, a retrofit to a broadcast channel. Our aim is
to build trust in existing infrastructures, where the origin of the data is
uncertain, by establishing that it can be used to distribute a quantum secure
secret key, on the condition that the $g^{(2)}(0)$ requirement is satisfied.

In the following, we model the eavesdropping and establish the security of the
protocol for correlated and un-correlated noise.

Figure 1: In this situation, a satellite beams down a signal, which is
received by Alice and Bob. We assume that Eve has control of the satellite,
and so beams up the signal she wants transmitted to Alice and Bob. Figure 2:
Schematics of the set-up. A thermal source shines onto a beamsplitter
$\eta_{ab}$ which splits the signal into a part going to Alice and the other
going to Bob. We assume that Eve has control of the source. The EPR is split
into one mode which is the source signal for Alice and Bob, the other remains
Eve’s.

##  I Formal description

####

To build our model, we follow Figure 2 and consider that Eve is sending one
mode of an EPR state down. This is the equivalent of a prepare-and-send
scheme, but instantly it puts Eve at a disadvantage since upon her measurement
of her mode, the mode sent to the legal parties is modelled by a thermal state
Weedbrookrmp:2012 . That mode falls onto a beamsplitter with transmittance
$\eta_{ab}$, which divides it between Alice and Bob. Eve’s input state is

$\Gamma_{EPR}=\left(\begin{array}[]{cc}\nu\boldsymbol{I}_{2}&\sqrt{\nu^{2}-1}\boldsymbol{Z}_{2}\\\
\sqrt{\nu^{2}-1}\boldsymbol{Z}_{2}&\nu\boldsymbol{I}_{2}\end{array}\right)\,,$

where $\boldsymbol{I}_{2}$ is the $2\times 2$ identity matrix and
$\boldsymbol{Z}_{2}=\left(\begin{array}[]{cc}1&0\\\ 0&-1\end{array}\right)$.

We assume a vacuum at the other input of the beamsplitter. The outcome is
(with $\zeta=\sqrt{\nu^{2}-1}$ and $\mu_{ab}=\sqrt{1-\eta_{ab}}$ for clarity)

$\Gamma=\left(\begin{array}[]{cccccc}\nu&0&\sqrt{\eta_{ab}}\zeta&0&-\mu_{ab}\zeta&0\\\
0&\nu&0&-\sqrt{\eta_{ab}}\zeta&0&\mu_{ab}\zeta\\\
\sqrt{\eta_{ab}}\zeta&0&\eta_{ab}\nu+\mu_{ab}^{2}&0&\mu_{ab}\sqrt{\eta_{ab}}(1-\nu)&0\\\
0&-\sqrt{\eta_{ab}}\zeta&0&\eta_{ab}\nu+\mu_{ab}^{2}&0&\mu_{ab}\sqrt{\eta_{ab}}(1-\nu)\\\
-\mu_{ab}\zeta&0&\mu_{ab}\sqrt{\eta_{ab}}(1-\nu)&0&\mu_{ab}^{2}\nu+\eta_{ab}&0\\\
0&\mu_{ab}\zeta&0&\mu_{ab}\sqrt{\eta_{ab}}(1-\nu)&0&\mu_{ab}^{2}\nu+\eta_{ab}\end{array}\right)\,.$

We identify the blocks as

$\Gamma=\left(\begin{array}[]{ccc}\gamma_{e}&\gamma_{eb}&\gamma_{ea}\\\
\gamma_{eb}&\gamma_{b}&\gamma_{ab}\\\
\gamma_{ea}&\gamma_{ab}&\gamma_{a}\end{array}\right)\,.$

As we mentioned before, since the eavesdropper controls the source, the
secrecy is determined by the information the legal parties share independently
of the source, $I(A:B|S)$, which is defined as

$I(A:B|S)=H(a,s)+H(b,s)-H(s)-H(a,b,s)\,.$

To calculate $H(a,s)$, $H(b,s)$, $H(s)$ and $H(a,b,s)$, we need

$\displaystyle\Gamma_{as}=$
$\displaystyle\left(\begin{array}[]{cc}\gamma_{e}&\gamma_{ea}\\\
\gamma_{ea}&\gamma_{a}\end{array}\right)$ (3) $\displaystyle\Gamma_{bs}=$
$\displaystyle\left(\begin{array}[]{cc}\gamma_{e}&\gamma_{eb}\\\
\gamma_{eb}&\gamma_{b}\end{array}\right)$ (6) $\displaystyle\Gamma_{s}=$
$\displaystyle\gamma_{e}$ $\displaystyle\Gamma_{abs}=$
$\displaystyle\left(\begin{array}[]{ccc}\gamma_{e}&\gamma_{eb}&\gamma_{ea}\\\
\gamma_{eb}&\gamma_{e}&\gamma_{ab}\\\
\gamma_{ea}&\gamma_{ab}&\gamma_{a}\end{array}\right)$ (10)

respectively. Then Shannon:1948_3 ; GarciaPatron:07 ,

$I(A:B|S)=\frac{1}{2}\log(2\pi e)^{2}\det(\Gamma_{as})+\frac{1}{2}\log(2\pi
e)^{2}\det(\Gamma_{bs})-\frac{1}{2}\log(2\pi
e)\det(\Gamma_{s})-\frac{1}{2}\log(2\pi e)^{3}\det(\Gamma_{abs})\,.$

Alice and Bob expect correlations in their respective signals. These
correlations are quantified using the quantum discord, defined explicitly as

$D(B|A)=S(\Gamma_{a})-S(\Gamma_{ab})+\min_{\Gamma_{0}}S(\Gamma_{b|x_{A}})\,$

where $\Gamma_{b|x_{A}}$ is the covariance matrix of B conditioned by a
homodyne measurement on A Weedbrookrmp:2012

$\Gamma_{b|x_{A}}=\,\Gamma_{b}-\Gamma_{ab}(X\Gamma_{a}X)^{-1}\Gamma_{ab}^{T}\,,$

with $X=\left(\begin{array}[]{cc}1&0\\\ 0&0\end{array}\right)$ and $()^{-1}$
the pseudo-inverse. The Von Neumann entropy is given by

$S(x)=\sum_{i=1}^{N}\left(\frac{x_{i}+1}{2}\right)\log\left(\frac{x_{i}+1}{2}\right)-\left(\frac{x_{i}-1}{2}\right)\log\left(\frac{x_{i}-1}{2}\right)$

where $x_{i}$ are the symplectic eigenvalues of $\Gamma$.

##  II Influence of correlated noise in Eve’s channel

###  II.1 Coherent state

####

$V_{e}=1SNU$ (Shot Noise Unit) represents a vacuum state, which is the minimum
uncertainty state, centred at the origin. Because a coherent state is a
displaced vacuum state, its variance is also $V_{e}=1SNU$. We have established
before in sakuya:2017_1 ; sakuya:2018_1  that there are no correlations, nor
information in a coherent state when it is shared by central broadcast. Figure
3 confirms this.

Figure 3: We plot the discord (red) and conditional mutual information (blue)
against $\eta_{ab}$ for $V_{e}=1SNU$.

It would be incorrect to assume that this is a general model for a system
running on empty as it were. We consider here loss-less and noise-less
channels between Eve and $\eta_{ab}$, as well as between $\eta_{ab}$ and Alice
and Bob. Therefore, we can establish the amount of correlations which Alice
and Bob can expect, should they share a coherent state. For this reason, we
have also elected to plot not only $I(A:B|S)$ but also $I(A:B)$; this shows
that there is no mutual information between Alice and Bob for them to
reconcile against the source. This is confirmed by the nullity of the discord,
which demonstrates that there are no correlations.

###  II.2 Thermal state

Eve inputs a state $V_{e}=\nu+1$, where $\nu$ is the variance of the thermal
state she sends through the satellite.

Figure 4: We plot the discord (red) and conditional mutual information (blue)
against $\eta_{ab}$ for $V_{e}=2SNU$.

This situation from Eve’s point of view, should be ideal. The state she is
sending is a thermal state of variance 1SNU, the rest being made up of shot
noise. This shot noise does not contain any correlation, as we have seen on
Figure 3. Yet, it brings fluctuations in the signal, so the potential for
photon pairs. This is enough to provide Alice and Bob with discord and
information they can share independently of her, as we see on Figure 4. This
means that as soon as the eavesdropper sends a state to the legal parties, she
has given them enough information that they can build a quantum secure secret
key.

Figure 5: We plot the discord (red) and conditional mutual information (blue)
against $\eta_{ab}$ for $V_{e}=1040SNU$. This value is consistent with an
input at microwave frequencies.

Figure 5 illustrates how thermal preparation noise influences $I(A:B|S)$ and
$D(B|A)$. As we would naturally expect, Alice and Bob’s situation is much
helped by Eve’s high preparation noise. Indeed, the higher Eve’s noise at
preparation, the more photon pairs arrive at $\eta_{ab}$ and become available
to Alice and Bob to build a key from.

It would then be easy to conclude that a high thermal input onto $\eta_{ab}$
helps Alice and Bob, although this would perhaps be hasty. A thermal input
onto $\eta_{ab}$ can happen in essentially two ways: Eve’s preparation noise
is thermal (Figure 5), or the channel between Eve and $\eta_{ab}$ introduces
the noise. Commonly, the thermal channel is modelled as a beamsplitter where
the “free” input receives a thermal state Weedbrookrmp:2012 ; Eisert:2005 .

###  II.3 Thermal channel

To model a thermal channel, let us add a beamsplitter of transmittance
$\eta_{th}$ between the source and $\eta_{ab}$. At one input, we feed $V_{e}$
and at the other, a thermal state of variance $V_{th}$.

The input covariance matrix to $\eta_{ab}$ is then
$\Gamma\bigoplus\boldsymbol{I}_{2}$ where

$\Gamma=\left(\begin{array}[]{cccccc}\nu&0&\sqrt{\eta_{th}}\zeta&0&-\mu_{th}\zeta&0\\\
0&\nu&0&-\sqrt{\eta_{th}}\zeta&0&\mu_{th}\zeta\\\
\sqrt{\eta_{th}}\zeta&0&\eta_{th}\nu+\mu_{th}^{2}V_{th}&0&\mu_{th}\sqrt{\eta_{th}}(V_{th}-\nu)&0\\\
0&-\sqrt{\eta_{th}}\zeta&0&\eta_{th}\nu+\mu_{th}^{2}V_{th}&0&\mu_{th}\sqrt{\eta_{th}}(V_{th}-\nu)\\\
-\mu_{th}\zeta&0&\mu_{th}\sqrt{\eta_{th}}(V_{th}-\nu)&0&\mu_{th}^{2}\nu+\eta_{th}V_{th}&0\\\
0&\mu_{th}\zeta&0&\mu_{th}\sqrt{\eta_{th}}(V_{th}-\nu)&0&\mu_{th}^{2}\nu+\eta_{th}V_{th}\end{array}\right)\,.$

The output matrix becomes

$\Gamma_{out}=\left(\begin{array}[]{cccc}\gamma_{e}&\gamma_{ev}&\gamma_{eb}&\gamma_{ea}\\\
\gamma_{ev}&\gamma_{v}&\gamma_{bv}&\gamma_{av}\\\
\gamma_{eb}&\gamma_{bv}&\gamma_{b}&\gamma_{ab}\\\
\gamma_{ea}&\gamma_{av}&\gamma_{ab}&\gamma_{a}\end{array}\right)\,,$

with

$\displaystyle\gamma_{e}=$ $\displaystyle\left(\begin{array}[]{cc}\nu&0\\\
0&\nu\end{array}\right)$ (13) $\displaystyle\gamma_{v}=$
$\displaystyle\left(\begin{array}[]{cc}V_{x}^{v}&0\\\
0&V_{p}^{v}\end{array}\right)\quad\text{with}\quad
V_{x}^{v}=\mu_{th}^{2}\nu+\eta_{th}V_{th}\quad\text{and}\quad
V_{p}^{v}=\mu_{th}^{2}\nu+\eta_{th}V_{th}$ (16) $\displaystyle\gamma_{a}=$
$\displaystyle\left(\begin{array}[]{cc}\mu_{ab}^{2}V_{x}^{ab}+\eta_{ab}&0\\\
0&\mu_{ab}^{2}V_{p}^{ab}+\eta_{ab}\end{array}\right)\quad\text{with}\quad
V_{x}^{ab}=\eta_{th}\nu+\mu_{th}^{2}V_{th}\quad\text{and}\quad
V_{p}^{ab}=\eta_{th}\nu+\mu_{th}^{2}V_{th}$ (19) $\displaystyle\gamma_{b}=$
$\displaystyle\left(\begin{array}[]{cc}\eta_{ab}V_{x}^{ab}+\mu_{ab}^{2}&0\\\
0&\eta_{ab}V_{p}^{ab}+\mu_{ab}^{2}\end{array}\right)$ (22)
$\displaystyle\gamma_{ev}=$
$\displaystyle\left(\begin{array}[]{cc}-\mu_{th}\zeta&0\\\
0&\mu_{th}\zeta\end{array}\right)$ (25) $\displaystyle\gamma_{eb}=$
$\displaystyle\left(\begin{array}[]{cc}\sqrt{\eta_{ab}}\sqrt{\eta_{th}}\zeta&0\\\
0&-\sqrt{\eta_{ab}}\sqrt{\eta_{th}}\zeta\end{array}\right)$ (28)
$\displaystyle\gamma_{ea}=$
$\displaystyle\left(\begin{array}[]{cc}-\mu_{ab}\sqrt{\eta_{th}}\zeta&0\\\
0&\mu_{ab}\sqrt{\eta_{th}}\zeta\end{array}\right)$ (31)
$\displaystyle\gamma_{bv}=$
$\displaystyle\left(\begin{array}[]{cc}\sqrt{\eta_{ab}}V_{x}^{abv}&0\\\
0&\sqrt{\eta_{ab}}V_{p}^{abv}\end{array}\right)\quad\text{with}\quad
V_{x}^{abv}=\mu_{th}\sqrt{\eta_{th}}(V_{th}-\nu)\quad\text{and}\quad
V_{p}^{abv}=\mu_{th}\sqrt{\eta_{th}}(V_{th}-\nu)$ (34)
$\displaystyle\gamma_{av}=$
$\displaystyle\left(\begin{array}[]{cc}-\mu_{ab}V_{x}^{abv}&0\\\
0&-\mu_{ab}V_{p}^{abv}\end{array}\right)$ (37) $\displaystyle\gamma_{ab}=$
$\displaystyle\left(\begin{array}[]{cc}\mu_{ab}\sqrt{\eta_{ab}}(1-V_{x}^{ab})&0\\\
0&\mu_{ab}\sqrt{\eta_{ab}}(1-V_{p}^{ab})\end{array}\right)$ (40)

We can see the effect of an increasing thermal noise input on Figure 6. When
$\eta_{th}=0$, the state which is transmitted is $V_{th}$. As a result, when
the thermal noise is very high, eg. $V_{th}=500$SNU, $I(A:B|S)$ and $D(B|A)$
are high as $\eta_{th}\rightarrow 0$, since a large thermal state has many
photon pairs to contribute to the secrecy sakuya:2018_1 . As $\eta_{th}$
increases, Eve’s contribution to the signal increases. As $\eta_{th}$
approaches unity, $I(A:B|S)$ and $D(B|A)$ are determined by the correlations
present in Eve’s signal.

The case $V_{th}=1$SNU shows us the effect of loss in the channel.
$\eta_{th}\rightarrow 0$ models high loss and therefore, $I(A:B|S)$ and
$D(B|A)$ are minimal at that point. This merely indicates that most of the
signal is lost on the way from the satellite. This makes the input into
$\eta_{ab}$ essentially shot noise, which we have already seen, by itself has
no secrecy potential.

As $V_{th}$ increase, $I(A:B|S)$ and $D(B|A)$ increase as well. This reflects
what we saw on Figure 5; a higher thermal input means more photon pairs so
more available information.

Naturally, all plots converge to the same value when $\eta_{th}=1$, since that
reflects a noiseless channel. At that point, only the source signal provides
correlations.

(a)

(b)

Figure 6: Conditional mutual information and discord for several values of a
thermal input at $\eta_{th}$. Eve’s input is $V_{e}=2SNU$ and Alice and Bob
get an equal share of the signal, $\eta_{ab}=0.5$.

##  III Influence of Alice’s and Bob’s channel noise

####

It may well seem like noise is a good thing, however, the noise considered in
the previous section is noise in Eve’s channel, before $\eta_{ab}$. Our
results so far exclude channel noise in either Alice’s or Bob’s channels. We
now include a thermal channel between $\eta_{ab}$ and Alice, and one between
$\eta_{ab}$ and Bob, in the same way as we did between the source and
$\eta_{ab}$, by means of beamsplitters $\eta_{th_{a}}$ and $\eta_{th_{b}}$
with secondary thermal inputs $V_{\alpha}$ and $V_{\beta}$.

The now-gigantic covariance matrix becomes

$\Gamma=\left(\begin{array}[]{cccccc}\gamma_{e}&\gamma_{ev}&\gamma_{eb}&\gamma_{ea}&\gamma_{ev_{a}}&\gamma_{ev_{b}}\\\
\gamma_{ev}&\gamma_{v}&\gamma_{bv}&\gamma_{av}&\gamma_{vv_{a}}&\gamma_{vv_{b}}\\\
\gamma_{eb}&\gamma_{bv}&\gamma_{b}&\gamma_{ab}&\gamma_{bv_{a}}&\gamma_{bv_{b}}\\\
\gamma_{ea}&\gamma_{av}&\gamma_{ab}&\gamma_{a}&\gamma_{av_{a}}&\gamma_{av_{b}}\\\
\gamma_{ev_{a}}&\gamma_{vv_{a}}&\gamma_{bv_{a}}&\gamma_{av_{a}}&\gamma_{v_{a}}&\gamma_{v_{a}v_{b}}\\\
\gamma_{ev_{b}}&\gamma_{vv_{b}}&\gamma_{bv_{b}}&\gamma_{a}&\gamma_{v_{a}v_{b}}&\gamma_{v_{b}}\end{array}\right)\,,$

with relevant sub-matrices

$\displaystyle\gamma_{e}=$ $\displaystyle\left(\begin{array}[]{cc}\nu&0\\\
0&\nu\end{array}\right)$ (43) $\displaystyle\gamma_{a}=$
$\displaystyle\left(\begin{array}[]{cc}\eta_{th_{a}}(\mu_{ab}^{2}V_{x}^{ab}+\eta_{ab})+\mu_{th_{a}}V_{\alpha}^{x}&0\\\
0&\eta_{th_{a}}(\mu_{ab}^{2}V_{p}^{ab}+\eta_{ab})+\mu_{th_{a}}V_{\alpha}^{p}\end{array}\right)$
(46) $\displaystyle\gamma_{b}=$
$\displaystyle\left(\begin{array}[]{cc}\eta_{th_{b}}(\eta_{ab}V_{x}^{ab}+\mu_{ab}^{2})+\mu_{th_{b}}V_{\beta}^{x}&0\\\
0&\eta_{th_{b}}(\eta_{ab}V_{p}^{ab}+\mu_{ab}^{2})+\mu_{th_{b}}V_{\beta}^{p}\end{array}\right)$
(49) $\displaystyle\gamma_{eb}=$
$\displaystyle\left(\begin{array}[]{cc}\sqrt{\eta_{th_{b}}}\sqrt{\eta_{ab}}\sqrt{\eta_{th}}\zeta&0\\\
0&-\sqrt{\eta_{th_{b}}}\sqrt{\eta_{ab}}\sqrt{\eta_{th}}\zeta\end{array}\right)$
(52) $\displaystyle\gamma_{ea}=$
$\displaystyle\left(\begin{array}[]{cc}-\sqrt{\eta_{th_{a}}}\mu_{ab}\sqrt{\eta_{th}}\zeta&0\\\
0&\sqrt{\eta_{th_{a}}}\mu_{ab}\sqrt{\eta_{th}}\zeta\end{array}\right)$ (55)
$\displaystyle\gamma_{ab}=$
$\displaystyle\left(\begin{array}[]{cc}\sqrt{\eta_{th_{a}}}\sqrt{\eta_{th_{b}}}\mu_{ab}\sqrt{\eta_{ab}}(1-V_{x}^{ab})&0\\\
0&\sqrt{\eta_{th_{a}}}\sqrt{\eta_{th_{b}}}\mu_{ab}\sqrt{\eta_{ab}}(1-V_{p}^{ab})\end{array}\right)$
(58)

Figure 7: We compare the influence of various noises on the conditional mutual
information. The parameters are as follows. Blue: $V_{e}=1SNU$,
$V_{\alpha}=V_{\beta}=1SNU$. Red: $V_{e}=2SNU$, $V_{\alpha}=V_{\beta}=1SNU$.
Yellow: $V_{e}=1SNU$ $V_{\alpha}=V_{\beta}=10SNU$. Purple: $V_{e}=2SNU$,
$V_{\alpha}=V_{\beta}=10SNU$. For all plots, we consider that there is no
thermal noise between the source and $\eta_{ab}$. $\eta_{a}=\eta_{b}=0.3$. (a)
Figure 8: We compare the influence of various noises on the discord(right).
The parameters are the same as for $I(A:B|S)$.

Figures 7 and 8 allow us to see the influence the thermal noise in Alice’s and
Bob’s channels has on the conditional mutual information and on the discord.

At first glance, this seems to contradict our earlier propos, namely, that
noise is our friend. Indeed, we can see straight away that when Alice’s and
Bob’s channels are noiseless and $V_{e}=2SNU$ (red plot), the conditional
mutual information and the discord are highest. This highlights that thermal
noise in Alice’s and Bob’s channels are deleterious to secrecy. That is fair
enough; after all, this particular noise is uncorrelated and so genuinely
pollutes Alice’s and Bob’s respective signals, independently.

####

The yellow plot shows $I(A:B|S)$ and $D(B|A)$ when there is thermal noise in
the legal channels (so after $\eta_{ab}$) and no source input. In this case,
there is no secrecy (and no discord) either. This shows the need for
correlated noise, so before $\eta_{ab}$. Indeed, the noises in Alice’s and
Bob’s channel are not correlated. Their actions merely degrades the signal
coming through $\eta_{2}$ as demonstrated by the purple curve.

##  IV Closing remarks

In this paper we have demonstrated, that any thermal source can be used for
secure key exchange even if that source is held by a third party. Alice and
Bob only need verify that the statistical properties of the source are
consistent with a thermal source. Moreover we have demonstrated throughout
this paper that any noise suffered by the eavesdropper, be it preparation
noise or channel noise, will benefit the legal parties. Furthermore, we have
demonstrated this with $I(A:B|S)$, not $I(A:B)$. This is significant. Not only
is the mutual information between Alice and Bob positive, but the information
they share independently of Eve is positive, which is a much more restrictive
condition.

Hanbury Brown and Twiss correlations can be preserved over astronomical
distances and, indeed the measurement of HBT correlations forms a core
component of radio astronomy. Microwave sources are particularly convenient,
because the thermal component is so high. Indeed existing infrastructure from
mobile phone masts to satellites could all be used as potential sources.

The ease of finding a source is counter balanced by the substantially reduced
key rates over other methods such as CVQKD Lance:2005  or BB84 protocols
Toshiba:2019 . However, there are many low level consumer applications that do
not require frequent key updates or high volumes of secret key. It is worth
noting that the aim here is not to replace or displace other methods of high
speed key exchange, rather the aim is to find a protocol that would allow
existing secure key exchange using existing communication mechanisms. Indeed
most modern communication systems rely on displaced thermal states (see for
example Saha:1989 ) and in a future paper we will aim to demonstrate key
exchange using data obtained from HBT measurements in a communications system.

####

The authors are grateful to network collaborators J. Rarity, S. Pirandola, C.
Ottaviani, T. Spiller, N. Luktenhaus and W. Munro for very fruitful
discussions, and to M. Carney and J. Wilson for help with the manuscript. This
work was supported by funding through the EPSRC Quantum Communications Hub
EP/M013472/1.

## References

  * [1] Elizabeth Newton, Anne Ghesquière, Freya L. Wilson, Benjamin T. H. Varcoe, and Martin Moseley.  Journal Physics B, 52(12):125501, 2019.
  * [2] Elizabeth Newton, Anne Ghesquière, Freya L. Wilson, Raoul F. Guiazon, Benjamin T. H. Varcoe, and Martin Moseley.  accepted for publication in Journal Physics B, arXiv:1912.04833, 2020.
  * [3] Christian Weedbrook, Stefano Pirandola, and Timothy C. Ralph.  Physical Review A, 86(2):022318(12), 2012.
  * [4] Stefano Pirandola.  Scientific Reports, 4:6956, 2014.
  * [5] Ueli M. Maurer and Stefan Wolf.  IEEE Transactions on Information Theory, 45(2):499–514, 1999.
  * [6] C. H. Bennett and G. Brassard.  Quantum cryptography: Public key distribution and coin tossing.  Proceedings of the IEEE International Conference on Computers, Systems and Signal Processing, pages 175–179, 1984.
  * [7] R. Hanbury Brown and R. Q. Twiss.  Nature, 177:27–29, 1956.
  * [8] R. Hanbury Brown and R. Q. Twiss.  Nature, 178:1046–1048, 1956.
  * [9] Sammy Ragy and Gerardo Adesso.  Physica Scripta, 2013(T153):014052, 2013.
  * [10] Christian Weedbrook, Stefano Pirandola, Raúl García-Patrón, Nicolas J. Cerf, Timothy C. Ralph, Jeffrey H. Shapiro, and Seth Lloyd.  Review of Modern Physics, 84(2):621(49), 2012.
  * [11] C. E. Shannon.  A mathematical theory of communication.  Bell System Technical Journal, pages 623–656, 1948.
  * [12] Raúl García-Patrón Sanchez.  PhD Thesis Université Libre de Bruxelles, 2007.
  * [13] J. Eisert and M. M. Wolf.  Gaussian quantum cchannels.  arXiv:quant-ph/0505151, 2005.
  * [14] Andrew M. Lance, Thomas Symul, Vikram Sharma, Christian Weddbrook, Timothy C. Ralph, and Ping Koy Lam.  No-switching quantum key distribution using broadband modulated coherent light.  Physical Review Letters, 95:180503, 2005.
  * [15] J. F. Dynes, A. Wonfor, W. W. Tam, and et al.  Cambridge quantum network.  npj Quantum Information, 5(101), 2019.
  * [16] D. Saha and T. G. Birdsall.  Quadrature-quadrature phase-shift keying.  IEEE Transactions on Communications, 37(5):437–448, 1989.