# BASICS: Broad quality Assessment of Static point clouds In Compression
Scenarios

Ali Ak, Emin Zerman, Maurice Quach, Aladine Chetouani, Aljosa Smolic, Giuseppe
Valenzise, Patrick Le Callet Manuscript is under preparation.

###### Abstract

Point clouds are now commonly used to represent 3D scenes in virtual world, in
addition to 3D meshes. Their ease of capture enable various applications on
mobile devices, such as smartphones or other microcontrollers. Point cloud
compression is now at an advanced level and being standardized. Nevertheless,
quality assessment databases, which is needed to develop better objective
quality metrics, are still limited. In this work, we create a broad quality
assessment database for static point clouds, mainly for telepresence scenario.
For the sake of completeness, the created database is analyzed using the mean
opinion scores, and it is used to benchmark several state-of-the-art quality
estimators. The generated database is named Broad quality Assessment of Static
point clouds In Compression Scenario (BASICS). Currently, the BASICS database
is used as part of the ICIP 2023 Grand Challenge on Point Cloud Quality
Assessment, and therefore only a part of the database has been made publicly
available at the challenge website. The rest of the database will be made
available once the challenge is over.

###### Index Terms:

Point cloud quality, 3D models, point cloud compression, subjective quality
assessment, database.

##  I Introduction

Digital imaging technologies enable capturing the real world and recreating it
in other times or other places. This has been extended to 3D scenes with the
help of computer graphics and photogrammetry techniques. We can now capture 3D
objects and scenes using purely RGB cameras [2] or RGB cameras with additional
sensors [3]. Two representations are commonly used representation for 3D
models: coloured point clouds and textured 3D meshes [4]. In this paper, we
focus on point clouds which are crucial for many applications such as
augmented and virtual reality.

Point clouds are typically captured via camera arrays, LiDAR sensors, and
cameras. The resulting volume of data is extremely large and compression
becomes essential for transmission and storage. However, evaluating
compression algorithms requires assessing the quality of point clouds
distorted by compression relative to the original point clouds. While research
into this area has been recently expanding [5], there are still many open
questions and problems. Specifically, we address the issues of enabling
evaluation of learning-based compression methods and creation of learning-
based quality metrics by providing a subjective dataset of sufficient scale,
consistency and diversity. Existing datasets until now have been either too
small, inconsistent (with respect to content normalization, rendering
conditions, etc.) or too lacking in diversity (in terms of semantics or the
number of source contents) for research into learning-based approaches.

The contributions of this work are threefold and can be listed as follows:

  * •

We present, and make publicly available (partly, during the ICIP 2023 Point
Cloud Visual Quality Assessment Grand Challenge), a broad point cloud quality
assessment database comprising 75 unique contents that are semantically
meaningful for a telepresence scenario, described in Section III,

  * •

We compare the performances of various state-of-the-art methods for point
cloud compression (Section V), and

  * •

We provide a complete benchmark of the state-of-the-art point cloud quality
metrics, including both point-based and rendering-based assessment (Section
VI).

The created BASICS database is made publicly available under Creative Commons
Attribution (CC BY-NC-SA 4.0) license to support further research in the
field. During the ICIP 2023 Grand Challenge, part of the dataset will be
accessible to the grand challenge participants via CodaLab registration
(Please see the grand challenge website [1]).

##  II Why is There a Need for Another Database?

In this section, we delve deeper into the following question ”Why is there a
need for another point cloud quality dataset?”. First, it is important to note
that learning-based point cloud compressionhas been a strong focus of recent
research among researchers in the last few years. It it thus important to
accurately assess the quality of point clouds distorted by such learning-based
compression methods. In addition, learning-based point cloud quality
assessment have also been explored recently. However, existing datasets lack
several qualities in order to enable evaluation of learning-based methods and
research into learning-based quality assessment approaches.

The existing datasets lack diversity, specifically they present one or more of
the following issues:

  * •

Use of the same point clouds across datasets.

  * •

Lacking variety in terms of geometric complexity and semantic categories.

  * •

Use of the same compression algorithms and consequently metrics over perform
on these distortions while failing on novel distortions (e.g. learning based
coding distortions).

This lack of diversity combined with the small scale of the existing datasets
make them unsuitable for learning-based quality assessment. In addition, the
lack of learning-based compression approaches in these datasets leads to novel
metrics failing for these methods.

The existing datasets also exhibit critical issues relating to data
availability:

  * •

Source point clouds unavailable due to copyright issues (Table II).

  * •

Unavailable raw scores, standard deviation and/or confidence intervals (Table
II).

The missing data impedes research into more sophisticated metric development
methods.

In addition, the existing datasets lack strongly in consistency. Specifically:

  * •

Point clouds are not normalized which impedes analysis and rendering. In this
dataset, all point clouds are voxelized with 10 bit quantization. As a result,
all coordinates are integers ranging between 0 and 1023.

  * •

Point cloud rendering is inconsistent which distorts results depending on the
compression method. In this dataset, all points are rendered using cubes
spanning the volume of their associated voxel. This has an especially
noticeable effect for octree compression (GPCC) as the resulting rendering is
watertight at all compression levels.

  * •

Rendering is stable which is not the case with most other datasets. Rendering
voxelized point clouds with cubes guarantees rendering without flickers as the
volumes are non overlapping.

The lack of consistency in existing datasets make them unpractical for
analysis and research.

TABLE I: Statistical summary of existing PC quality assessment datasets

Dataset | nb SRC | nb PPC |  | nb obs
---
per PPC
| total
---
nb obs
Display | Visualization |  | Subj. test
---
method
| Temporal
---
dimension
Distortions
BASICS (proposed) | 75 | 1494 | 60 | 3600 | 2D | Passive | DSIS (side-by-side) | Static | Compression: GPCC, VPCC, GeoCNN
vsenseVVDB2[4] | 8 | 136 | 23 | 23 | 2D | Passive | ACR | Dynamic | Compression: GPCC, VPCC
Perry et al. [6] | 6 | 90 | - | 73 | 2D | Passive | DSIS (side-by-side) | Static | Compression: GPCC, VPCC
da Silva Cruz et al. [7] | 8 | 48 | - | 50 | 2D | Passive | DSIS | Static |  | Octree pruning
---
Projection-based compression from 3DTK
Yang et al. [8] | 10 | 420 | 16 | 64 | 2D | Passive | ACR | Static |  | Octree pruning, random point down-sample
---
Color noise, geometric gaussian noise
Su et al. [9] | 20 | 740 | - | 60 | 2D | Passive | DSIS (side-by-side) | Static |  | Octree pruning, geometric gaussian noise
---
Compression: SPCC, VPCC, LPCC
NBU-PCD1.0 [10] | 10 | 160 | - | - | 2D | - | - | Static | Octree pruning
SIAT-PCQD [11] | 20 | 340 | 38 | 76 | HMD | Interactive | DSIS | Static | Compression: VPCC
Subramanyam et al. [12] | 8 | 64 | - | 52 | HMD | Interactive | ACR-HR | Dynamic | Compression: VPCC
PointXR [13] | 5 | 40 | 20 | 40 | HMD | Interactive | DSIS | Static | Compression: GPCC

TABLE II: Public availability of the existing datasets

Dataset | Point Clouds | Subjective Annotations
---|---|---
BASICS (proposed) | ✓ | Individual Scores
vsenseVVDB2 [4] | ✓ | Individual Scores
Perry et al. [6] | Broken URL | ✗
Su et al. [9] | ✓ | (D)MOS only
da Silva Cruz et al. [7] | ✗ | ✗
Yang et al. [8] | ✓ | (D)MOS only
NBU-PCD1.0 [10] | ✗ | ✗
SIAT-PCQD [11] | Broken URL | ✗
Subramanyam et al. [12] | ✗ | Individual Scores
PointXR [13] | ✓ | Individual Scores

In summary, a new dataset is required as existing datasets are lacking in one
or more aspects. Namely, in terms of diversity, scale and consistency. We
propose a new dataset that fulfills all of these characteristics which are
essential for learning-based use cases. Specifically, this dataset enables
better research into learning-based quality assessment. In addition, it also
provides crucial insight on the behavior of point cloud quality metrics when
applied to learning-based compression algorithms.

In particular, the proposed BASICS database is aimed to provide a foundation
for research that supports the telepresence applications, in terms of
compression and quality assessment.

##  III The BASICS Database

Imagine that you are a researcher or a developer who wishes to develop a
quality metric using learning-based approaches. Or, you might be seeking to
validate your quality metric. In order to provide means for both of these
scenarios, and to remedy the limitations discussed in the previous section, we
generate the BASICS database. In the following, we describe various stages of
the database generation procedure.

###  III-A Material selection

Our motivation in this work, as previously mentioned, is to develop a
semantically diverse point cloud quality assessment database for a
telepresence application in which a real scene is captured, compressed,
transmitted, and displayed using point clouds. In almost all applications of
telepresence, humans are the main subject of the communication. Therefore, it
is important to capture the point clouds that represent humans. Additionally,
pets or other animals can be part of the scene in some applications. Inanimate
objects will be in the scene, and they can also be a topic of work meetings
(e.g., designing objects) or education applications (e.g., museum artifacts).
Lastly, buildings or background landscape can also be part of the contents in
telepresence applications.

Considering the above discussion, we identify fundamental building blocks and
corresponding semantic categories for point cloud contents to include in the
database. Three main categories are selected for prospective 3D models for the
reference content: (i) humans & animals, (ii) inanimate objects, and (iii)
buildings & landscapes. Screenshots of sample figures from each category are
shown in Fig. 1.

((a)) HA - Two people

((b)) HA - Dedalus

((c)) HA - Stuffed bird

((d)) IO - Violin

((e)) IO - Statue

((f)) IO - Antique horn

((g)) BL - Opera

((h)) BL - Bunker

((i)) BL - Château

Figure 1: Three semantic categories of point clouds in the BASICS database:
Humans & Animals (HA), Inanimate Objects (IO), and Buildings and Landscapes
(BL).

For the data collection and material selection part, we aimed to collect as
many publicly available point clouds as possible, which could be
redistributable. However, many data sources restrict redistribution.
Therefore, we acquired the 3D models from two sources: collaborator studios
(i.e., V-SENSE studio and XD Productions) and an online repository for 3D
model sharing called SketchFab111https://www.sketchfab.com/. Even in this
case, there were not many point cloud sources. Therefore, we gathered 3D
meshes and generated point clouds via sampling the mesh surface (cf. Section
III-B).

In total, 104 models were handpicked by three authors of this paper
considering the semantic categories described above. After eliminating very
similar materials, the models with non-ideal characteristics (e.g., highly
reflective material, imperfect texturing etc.), and the least relevant
semantic categories, the total number of models were dropped to 75.

###  III-B Pre-processing

To discount for all other possible aspects that can introduce distortions, the
collected 3D models needed a pre-processing and a conversion into point clouds
before any further processing, as these models were in different formats.

Models that were already in point cloud PLY format did not need too much
attention, except for voxelization. 3D meshes, on the other hand, needed to go
through several steps. These steps are further discussed below.

####  III-B1 Making 3D meshes uniform

Among collected 3D mesh models, some of them were in OBJ format and some
others were in FBX format. Using Blender222https://www.blender.org/ and
Meshlab333https://www.meshlab.net/, all 3D meshes were converted into OBJ
format.

####  III-B2 Cleaning 3D meshes

Some of the 3D meshes had either parts that had transparent or reflective
properties (e.g., glasses in some models). Some other meshes had parts the
reconstruction of which were incomplete (e.g., trees, some of the building
façades) which would decrease the users’ quality of experience and introduce
other sources of distraction and distortion. To avoid such effects, these
parts were removed or cleaned in Blender.

Following this, the mesh files were unified into a single OBJ file, so that
the sampling process in the pipeline could be done with ease. Next, the
material properties (which are described in the .mtl files) are checked to
eliminate the any other reflective properties of the materials, which could
not be reproduced correctly in the point cloud format. After all these
operations, the 3D meshes were ready for the point cloud sampling step.

####  III-B3 Sampling point clouds

Using CloudCompare444https://www.cloudcompare.org/main.html, point clouds were
sampled from the 3D meshes’ surfaces using … command. During this operation
more than 5 million points were sampled on the surfaces of the said meshes.
The sampling operation extracted the location, color, and normal attributes
for each point in the PCs. At the end of this stage, all 3D models were in or
converted to point cloud format.

####  III-B4 Point cloud voxelization

We perform point cloud voxelization using 10 bit quantization. That is, the
spatial coordinates are normalized such that they are integers between 0 and
1023. This has two main advantages: first, the coordinates are in a range that
is predictable for point cloud processing but also with respect to rendering
and second, we use voxelized coordinates in combination with cube based
rendering to improve stability, predictability and quality of renderings.

###  III-C Compression

As mentioned above, the main goal for the BASICS database is to provide a
foundation for further point cloud compression and point cloud quality
assessment research, especially for the telepresence use case. In this use
case, capturing and display needs to be real-time. Therefore, compression
carries a huge importance.

For preparing the processed point clouds (PPC), the following compression
methods are selected: the octree-based compression method MPEG GPCC
[14]555github.com/MPEGGroup/mpeg-pcc-tmc13/releases/tag/release-v14.0, the
video-based compression method MPEG VPCC [14]666github.com/MPEGGroup/mpeg-pcc-
tmc2/releases/tag/release-v15.0, and a learning-based compression method
GeoCNN [15]777github.com/mauriceqch/pcc_geo_cnn_v2. GPCC and VPCC are selected
as they are the MPEG standardization efforts which focus the main expert
knowledge in the standardization field. GeoCNN was selected to include more
recent learning-based algorithms, which was one of the best performing
learning-based compression method at the time of dataset preparation.

Geometry-based point cloud compresison (GPCC) focuses on encoding the point
cloud directly in the 3D space [14] using octree or trisoup (triangle soup)
methods. The attributes (such as color) can also be coded using either Region
Adaptive Hierarchical Transform (RAHT) or Predicting/Lifting (PredLift)
transform. Video-based point cloud compression (VPCC) instead uses a different
approach and is focused on compression of dynamic point clouds (i.e., point
clouds changing in time). VPCC projects the point cloud content onto a depth
map and a texture map and uses a state-of-the-art video encoder (e.g., HEVC)
to compress the PCs. GeoCNN [15] compresses voxelized point clouds by first
performing block partitioning. Then, each block is passed to a variational
autoencoder [16] where the encoder transforms the input binary occupancy voxel
grid to a latent space. The latent space is then quantized and entropy coded
using a learned entropy model. After entropy decoding the bitstream, the
latent space is transformed back to a voxel grid containing predicted
occupancy probabilities. The probabilities are then thresholded to binary
values which yields the decoded block. With the result of each block, the
entire decompressed point cloud is obtained.

In this work, we used GPCC-Octree-RAHT, GPCC-Octree-Predlift, VPCC, and GeoCNN
for compression of the PCs. The details regarding the compression parameters
will be provided after the ICIP 2023 Grand Challenge is completed.

##  IV Subjective quality assessment

We conducted a large-scale subjective experiment in Prolific [17]
crowdsourcing platform with more than 3000 participants. More than 1200
stimuli from 75 original point clouds were generated for the experiment
wherein each processed point cloud (PPC) were evaluated by around 60 unique
participants on average. This section describes the detail regarding the
crowdsourcing study.

Subjective quality assessment of point cloud content can be categorized into
two as interactive and passive [5]. In the interactive paradigm, observers
have the freedom to inspect the point cloud from any point of view without any
restriction often with an augmented reality or virtual reality application. In
the passive approach, point clouds are rendered with a predefined camera
trajectory as a traditional video. Although both paradigm has their own
advantages and disadvantages, there is no statistically significant difference
between the subjective opinions collected with each [18]. In order to minimize
the variance between observer opinions and allow ourselves a more practical
data collection through crowdsourcing, we adopted the passive approach [19].

###  IV-A Methodology

Several methodologies can be found in the literature and recommendations for
subjective quality assessment of traditional image and video sequences [20].
Commonly used methodologies include, but not limited to: Absolute Category
Rating (ACR), Double Stimulus Impairment Scale (DSIS) and Two-Alternative
Forced Choice (2AFC) or Pairwise Comparison (PC). Several studies compared the
accuracy and reliability of each methodologies for varying multimedia content.
For traditional image and videos, Mantiuk et al. denote that the PC
methodology tends to be more accurate due to straightforward experiment
procedure and there is no statistically significant difference between ACR and
DSIS methodology [21]. However, despite the simplicity of the task, PC
methodology may require exponentially more comparisons and may become
impractical with high number of test conditions [22]. On the other hand, the
recent study by Nehme et al. [23] suggests that the DSIS method is more
accurate than ACR for 3D graphical content. It is suggested that, in ACR
experiments, participants who are unfamiliar with the pristine models are not
able to discriminate all type of distortions. DSIS methodology leads to a more
accurate evaluation by presenting the reference and the distorted model prior
to rating. Therefore, we utilized DSIS methodology with a side-by-side
presentation as suggested by Nehme et al. [23].

###  IV-B Generating Visual Stimuli

In a voxelized point cloud, points have a one-to-one mapping to a voxel as
part of a voxel grid. Building on this property, each point is rendered as a
cube spanning the volume of its voxel. This is different from common ”point”
(OpenGL point primitive) based rendering which renders points as screen-
aligned squares of a given window space size. The main issue is that the size
is given in window space, thus when a zoom is performed the points become
smaller and the point cloud appears sparser. In addition, point based
rendering causes flicker artifacts due to spatial overlaps especially during
perspective changes. Cube based rendering corrects this issue and enables
watertight rendering from all perspectives.

Moreover, cube based rendering has a particular relationship with octree based
compression. A typical octree compression algorithm represents the point cloud
using an octree, providing a natural decomposition in level of details.
Specifically, at each octree subdivision, each occupied voxel is subdivided
into eight equal voxels and their occupancies are then encoded. Typically, a
desired level of detail is selected and each occupied voxel at this level of
detail is transformed into a point. However, this neglects that each point
actually corresponds to a volume. Using this property, rendering each point as
a cube spanning this volume greatly improves rendering results for octree
methods. In particular, a watertight voxel point cloud remains watertight
regardless of the level of detail. Compared to the previous approach, point
clouds look visually ”blockier” rather than sparser which preserves visual
continuity of the rendered objects.

In practice, we specify cube sizes. For octree based methods, the size of the
cube is defined based on the number of removed octree levels $n_{r}$. With
$n_{l}$ bit quantization, the maximum is $n_{l}$ levels. Thus, we specify the
size of the cube as $2^{n_{l}-n_{r}}$: that is, a size of 1 when lossless, a
size of 2 when removing one octree level, a size of 4 when removing two octree
levels, etc. For other methods, the cube size is determined empirically in a
pilot test by a subset of the authors manually to ensure that the output looks
watertight.

Helix-like rendering trajectory is utilized as visualized in Figure 2. Front
direction of each point cloud were assigned manually and rendering trajectory
is always initiated from the assigned front side. A small overlap is adopted
between the start and end point of the trajectory to ensure that the front
side of the point cloud is seen either at the end or at the beginning of the
rendered video. Certain point clouds are unnatural to observe from lower
angles(e.g., landscape, buildings). Therefore, after a pilot test, each point
cloud manually assigned to one of the following categories: low, mid, high. It
is used to determine the starting elevation of the rendering trajectory. While
moving on the rendering trajectory, the camera is always directed towards the
point cloud center.

Figure 2: Visualization of the rendering trajectory from top and front views.

###  IV-C Test Procedure

Earlier studies suggests that the crowdsourcing experiments can be as accurate
as laboratory experiments for various QoE tasks and with different experiment
designs [24, 19]. To benefit from the wide participant pool and faster data
collection, we utilized the Prolific [17] crowdsourcing platform to recruit
participants and to conduct the subjective experiment. On Prolific, the
participants are clearly informed that they are being recruited as part of a
research study and the requirements for the experiments are well balanced to
benefit both sides; researchers and participants [25].

Figure 3: Sample screenshots from the experiment. Rendered point cloud videos
were shown side-by-side (above), and each stimulus was followed by a voting
screen (below).

Test sessions & Duration: Due to lack of supervision on participants during
the experiment, the number of stimuli and the duration of the test in
crowdsourcing settings should be kept much lower than the laboratory
experiments. To this end, we split the experiment into 60 sessions each
containing 25 stimuli and 2 dummies. One dummy from the highly compressed
stimuli and one dummy from the lowest compressed stimuli were selected to be
shown to every participant to create expectations about the range of
distortions. Dummy stimuli were the same for every participant and
participants were not informed that these stimuli were shown for training
purposes. In total, every participant rated 27 stimuli of 10 seconds video
renderings. With unlimited voting time after each stimuli, test sessions
lasted around 5 minutes 30 seconds on average. See a sample screenshot from
the experiment in Figure 3.

Participants & Requirements: We recruited 60 participants (50% female - 50%
male) on average per session, more than 3000 participants in total. Every
participant was compensated for their time and the age of the participants
range from 18 to 70. Moreover, to ensure all stimuli were shown as intended,
participants were limited to use selected browsers on full screen with 1080p
resolution. In addition, participants were required to complete at least 200
submissions with 100% approval rate on Prolific.

##  V Subjective Experiment Results

This section presents the result of subjective quality scores analyses.
Section V-A investigates the content ambiguity for each source point cloud.
Comparison of different methods to acquire MOS from individual observer
opinions presented in Section V-B. Finally, performance of the compression
algorithms are analyzed in Section V-C

Figure 4: Bit per point vs MOS plots for each SRC. Each point represents a PPC
acquired from the SRC indicated at the title of each plot. Vertical axis of
each plot is aligned to [1, 5] range and indicates the MOS of PPCs. Horizontal
axis represents the bit per point for each PPC and the bit per point ranges
are not necessarily the same for all SRCs. Since GeoCNN compresses only the
geometry information, it is excluded from the analysis.

###  V-A Content Ambiguity

Some contents can be more difficult to evaluate than the others. Depending on
the QoE scenario, various factors can lead to more/less ambiguous contents. In
order to estimate the content ambiguity of source point clouds in the dataset,
we used Netflix-Sureal package[26]. We observe that the ambiguity of the point
clouds are correlated with the visual quality of the source point cloud. In
other words, less artefacts (due to acquisition, processing, etc.) on the
source content, leads to easier evaluation of the compression distortions.
This phenomenon is in fact well known in image and video quality domain.

To further explore the content ambiguity, we analyzed the correlation between
number of points and content ambiguity for each semantic category in the
dataset. We observe a linear correlation between number of points and content
ambiguity for the point clouds in humans & animals category. However, the same
conclusion cannot be drawn for the buildings & landscapes and inanimate
objects categories. This can be explained by the similar geometric complexity
and physical size of the source point clouds in humans & animals category. Due
to low variance in geometric complexity and physical size, number of points
often dictates the visual quality of the source point clouds. Therefore, we
observe a greater correlation between content ambiguity and number of points
in this category. In contrast, variance of geometric complexity and physical
size is much higher in buildings & landscapes and inanimate objects
categories. Therefore, number of points alone cannot dictate the visual
quality. More details will be provided for content ambiguity after ICIP 2023
Grand Challenge is concluded.

###  V-B Mean Opinion Scores

In order to analyze the validity of the collected subjective opinion scores
and provide reliable MOS, we compared three methodologies to estimate MOS.
First, no observer screening was applied to the collected opinion scores. Raw
MOS is calculated simply averaging all opinions for each PPC. Secondly, BT500
MOS is calculated by following the recommendations in ITU-R BT.500-14 A1-2.3.1
[20]. Observer screening is applied once before calculating the MOS. Among
more than 3000 total observers participated to the subjective experiment, 47
of them found as outliers and omitted from the BT500 MOS calculation. As the
last method, Netflix Sureal [26] was used to estimate the MOS by taking
subject inconsistency and biases into account. All participants’ opinions were
included in the Sureal MOS estimation. Figure 5 presents the results as
scatter plots between each pair of methodologies as well as pearson and
spearman rank order correlation coefficients. Results clearly indicate that
there is no significant difference between the three methodologies. This
further confirms the validity of Prolific participant pool and the experiment
design. In the dataset public repository, we provide the raw opinion scores
and MOS acquired by following the ITU-R BT.500-14 recommendations.

Figure 5: Comparison of calculated MOS with three different methodologies.
Each plot contains a pair of comparison between the three methods. Higher MOS
indicates higher visual quality. Raw represents the simple mean over collected
opinion scores without outlier detection. BT500 represents the MOS values
after BT.500-14 [20] observer screening step and Sureal is the MOS estimated
with Netflix Sureal[26]. Pearson and spearman rank order correlations are
recorded on each plot.

Based on the MOS acquired by ITU-R BT.500-14 recommendations, the distribution
of MOS for each SRC was checked, and it was observed that, for a given SRC,
MOS distributions cover the whole quality range with few exceptions. Note that
the experiment is conducted with DSIS methodology. Consequently, acquired MOS
are content aware and equivalent to DMOS in an ACR-HR experiment methodology.

###  V-C Performance of compression algorithms

For completeness, the performance of compression algorithms were also
analyzed. For this analysis, the rate-distortion curves have been plotted as
shown in Figure 4. Bit-per-point is used for the rate, which is shown on the
x-axes of the plots, and mean opinion scores are shown on the y-axes of the
plots. For the sake of comparisons, the bitrate values were stretched to show
the full extent of the bitrate range. This means that the plots are only
meaningful within themselves and should not be compared to other plots.

The results show that, in general, VPCC is performing better than GPCC, which
supports the findings of earlier studies [27]. Among GPCC-RAHT and GPCC-
Predlift there seems to be no significant difference, considering all the
different SRC contents, even though GPCC-RAHT seems to yield slightly lower
bitrate for higher bitrates. As the cautious readers can identify, there is a
slight decrease in the subjective quality for the highest bitrate of p45,
which is caused by a hole in the final rendering, the cause of which is
unknown.

##  VI Objective Quality Assessment

Point cloud objective quality metrics can be categorized into three,
considering the type of input to the quality metrics: (i) image-based, (ii)
color-based, and (iii) geometry-based. Image-based metrics take the rendered
point cloud image or image sequences as input and assess the quality of the
point clouds. Geometry-based metrics rely only on the geometry information
(e.g., the location in 3D space) stored at each point in the point cloud,
ignoring the color attribute. Color-based metrics utilizes the both geometry
and color information of each point to assess the point cloud quality.
Moreover, each metric can be categorized into three based on the presence of
reference point cloud information as full-reference (FR), reduced-reference
(RR) and no-reference (NR). FR metrics access all information from the
reference point cloud in addition to the distorted point cloud. NR metrics can
access only partial information (features) from the reference point clouds. NR
metrics assess the quality of the point cloud without any access to the
reference point cloud.

In this section, we benchmark 14 image-based, 9 color-based and 17 geometry-
based metrics from the literature. Selected metrics are introduced in Section
VI-A. Various methodologies were adopted to evaluate metric performances and
introduced in Section VI-B. Results of these analyses are presented in Section
VI-C and Section VI-D

###  VI-A Selected Metrics

For all image-based metrics, average pooling over 30 fps video renderings has
been used to predict the final quality as recommended in [28]. Image-based
metrics include simple measures such as MSE, PSNR and SSIM [29] and 11 other
more sophisticated metrics. Feature similarity index (FSIM [30]) and its
color-dependent variant FSIMc [30] are full reference metrics that rely on
phase congruency and gradient magnitude to quantify image quality locally and
uses phase congruency as a weighting function to obtain a single quality
score. Gradient magnitude similarity deviation (GMSD[31]) is another full
reference metric utilizing the pixel-wise gradient magnitude similarity to
predict the image quality. D-JNDQ [32] is a learning-based full reference
metric that is trained on first just noticeable difference (JND) points of
JPEG compression artefacts. It combines a white-box optical and retinal
pathway model with a Siamese neural network to predict image quality. MW-PSNR
[33, 34] is based on morphological wavelet decomposition and MSE of the
wavelet sub-bands. Both full reference (MW-PSNR-FR) and reduced reference (MW-
PSNR-RR) versions are included in the evaluation.

The geometry-based metrics disregard the color information. The point-to-point
[35] and point-to-plane [36] metrics are most commonly used in MPEG
standardization activities, and they focus on distance among the nearest
points or distance between the point and the projected point on the second set
considering the normal, respectively. Plane-to-plane [37] measures the angular
difference between the normals of the closest points. PCQM [38] focuses on
finding the curvature of a predicted surface from a set of closest points with
normals and estimates quality based on this curvature estimate. PointSSIM [39]
calculates attributes such as geometry, normals, curvature, and colors. Once
the features are calculated, they are fed into a similarity function and
pooled all together. As geometry-based attributes were found second best (to
the color-based attributes), PointSSIM Geom- metrics were included in this
analysis.

The color-based metrics differ in nature. So far, two color-based metrics were
considered. The metrics named as “Color $<$channel name$>$ $<$pooling$>$” do
measure the color difference between the closest points and disregard any
geometrical information. The PointSSIM [39], on the other hand, have two
different modes and can take either a set distance or k-nearest neighbors. As
color-based attributes were found to be the highest performing among other
types of attributes, PointSSIM Color- metrics were included in this analysis.

###  VI-B Evaluation Criteria

The whole dataset was used for the evaluation. For learning-based objective
quality metrics, no training or fine-tuning is applied prior to evaluation.
Evaluation of the objective quality performances were made with 2 main
methods. First analysis relies on traditional measures to analyze the
correlation between metric predictions and MOS. Second analysis relies on
statistical significance of the differences between pairs of stimuli. It
measures the metric performance based on the capability of identifying
significantly different pairs.

TABLE III: Pearson and Spearman correlation coefficients between the listed
metric scores and MOS for each compression algorithm

|  | All | GeoCNN |  | GPCC
---
Predlift
| GPCC
---
Raht
VPCC
Category | Metric | CC | SROCC | CC | SROCC | CC | SROCC | CC | SROCC | CC | SROCC
Image Based | MSE | 0.2419 | 0.2274 | 0.4140 | 0.4197 | 0.3362 | 0.2614 | 0.3215 | 0.2443 | 0.0930 | 0.0731
PSNR | 0.2520 | 0.2318 | 0.4235 | 0.4244 | 0.3447 | 0.2657 | 0.3298 | 0.2483 | 0.1181 | 0.0775
SSIM[29] | 0.6219 | 0.5482 | 0.5505 | 0.5350 | 0.7508 | 0.6172 | 0.7628 | 0.6394 | 0.4115 | 0.3765
MS-SSIM | 0.5536 | 0.4660 | 0.4957 | 0.4797 | 0.6757 | 0.5322 | 0.6844 | 0.5501 | 0.3704 | 0.3043
FSIM[30] | 0.6375 | 0.5656 | 0.5748 | 0.5701 | 0.7621 | 0.6330 | 0.7700 | 0.6534 | 0.4166 | 0.3838
FSIMc[30] | 0.6374 | 0.5651 | 0.5750 | 0.5690 | 0.7618 | 0.6320 | 0.7697 | 0.6528 | 0.4164 | 0.3836
GMSD[31] | 0.6737 | 0.6143 | 0.6372 | 0.6320 | 0.8004 | 0.6730 | 0.7985 | 0.6885 | 0.4635 | 0.4463
D-JNDQ[32] | 0.6782 | 0.6349 | 0.7285 | 0.7204 | 0.7914 | 0.6765 | 0.8043 | 0.7045 | 0.4431 | 0.4469
MW-PSNR-FR[33] | 0.3504 | 0.3344 | 0.4613 | 0.4601 | 0.4509 | 0.3790 | 0.4390 | 0.3730 | 0.2071 | 0.1701
MW-PSNR-RR[34] | 0.5149 | 0.5015 | 0.5724 | 0.5623 | 0.6277 | 0.5611 | 0.6272 | 0.5625 | 0.3067 | 0.3170
ADM2 | 0.7348 | 0.6555 | 0.6457 | 0.6203 | 0.8412 | 0.6896 | 0.8369 | 0.7096 | 0.5588 | 0.5456
VIF-scale3 | 0.6556 | 0.5993 | 0.6000 | 0.6026 | 0.7721 | 0.6542 | 0.7791 | 0.6759 | 0.4386 | 0.4316
VMAF | 0.7429 | 0.6708 | 0.6573 | 0.6366 | 0.8545 | 0.7072 | 0.8537 | 0.7323 | 0.5594 | 0.5473
FVVDP | 0.6973 | 0.6433 | 0.6345 | 0.6427 | 0.8206 | 0.6999 | 0.8365 | 0.7287 | 0.4781 | 0.4712
Color Based | Color-Y-MSE | 0.5322 | 0.5283 | 0.2269 | 0.1340 | 0.7404 | 0.7004 | 0.7573 | 0.7215 | 0.4232 | 0.4371
Color-Y-PSNR | 0.5386 | 0.5283 | 0.1931 | 0.1494 | 0.7415 | 0.6992 | 0.7582 | 0.7213 | 0.4300 | 0.4371
Color-U-MSE | 0.5451 | 0.5096 | 0.2439 | 0.1365 | 0.6828 | 0.6472 | 0.6632 | 0.6303 | 0.3849 | 0.3770
Color-U-PSNR | 0.5430 | 0.5096 | 0.2447 | 0.1724 | 0.6812 | 0.6471 | 0.6629 | 0.6304 | 0.3845 | 0.3781
Color-V-MSE | 0.5731 | 0.5417 | 0.2508 | 0.1922 | 0.7006 | 0.6691 | 0.6917 | 0.6621 | 0.4450 | 0.4319
Color-V-PSNR | 0.5738 | 0.5403 | 0.3077 | 0.2719 | 0.7004 | 0.6721 | 0.6914 | 0.6620 | 0.4480 | 0.4327
PointSSIM-ColorAB[39] | 0.7291 | 0.6872 | 0.5890 | 0.4317 | 0.8019 | 0.7882 | 0.8257 | 0.8235 | 0.7383 | 0.7644
PointSSIM-ColorBA[39] | 0.7241 | 0.6890 | 0.6088 | 0.4564 | 0.8030 | 0.7904 | 0.8263 | 0.8246 | 0.7287 | 0.7580
PointSSIM-ColorSym[39] | 0.7250 | 0.6883 | 0.6059 | 0.4454 | 0.8027 | 0.7895 | 0.8260 | 0.8237 | 0.7320 | 0.7603
Geometry Based | p2point-PSNR[35] | 0.6884 | 0.5393 | 0.2141 | 0.2140 | 0.7756 | 0.6540 | 0.7882 | 0.6800 | 0.5426 | 0.5171
p2point-Haus[35] | 0.0941 | 0.3614 | 0.4861 | 0.6044 | 0.0602 | 0.8818 | 0.9027 | 0.9050 | 0.1467 | 0.4867
p2point-HausPSNR[35] | 0.1978 | 0.2417 | 0.4729 | 0.4343 | 0.7784 | 0.6225 | 0.7848 | 0.6510 | 0.4612 | 0.4301
p2plane-MSE[40] | 0.0748 | 0.8363 | 0.6962 | 0.6357 | 0.3543 | 0.8850 | 0.9584 | 0.9013 | 0.5001 | 0.8204
p2plane-PSNR[40] | 0.7079 | 0.5750 | 0.2535 | 0.2604 | 0.7637 | 0.6428 | 0.7773 | 0.6650 | 0.6133 | 0.5688
p2plane-Haus[40] | 0.1086 | 0.3938 | 0.5001 | 0.6176 | 0.0631 | 0.8844 | 0.4227 | 0.9065 | 0.1701 | 0.5558
p2plane-HausPSNR[40] | 0.2373 | 0.2753 | 0.4552 | 0.4228 | 0.8384 | 0.6705 | 0.8369 | 0.6912 | 0.4769 | 0.4389
pl2plane-Min[41] | 0.0494 | 0.0676 | 0.1009 | 0.1457 | 0.0564 | 0.0971 | 0.0590 | 0.0917 | 0.1122 | 0.0476
pl2plane-Mean[41] | 0.1519 | 0.1281 | 0.3420 | 0.2069 | 0.1853 | 0.1403 | 0.1944 | 0.1747 | 0.1225 | 0.1114
pl2plane-Median[41] | 0.1519 | 0.1281 | 0.3420 | 0.2069 | 0.1853 | 0.1403 | 0.1944 | 0.1747 | 0.1225 | 0.1114
pl2plane-Max[41] | 0.2123 | 0.1453 | 0.1472 | 0.0648 | 0.2907 | 0.2898 | 0.2020 | 0.2563 | 0.0392 | 0.0231
pl2plane-RMS[41] | 0.1341 | 0.1040 | 0.3466 | 0.2159 | 0.1625 | 0.1204 | 0.1788 | 0.1579 | 0.1150 | 0.0924
pl2plane-MSE[41] | 0.1281 | 0.1038 | 0.3428 | 0.2138 | 0.1603 | 0.1241 | 0.1742 | 0.1564 | 0.1025 | 0.0876
PCQM[38] | 0.8855 | 0.8060 | 0.4407 | 0.2923 | 0.9490 | 0.8671 | 0.9570 | 0.8924 | 0.8548 | 0.8371
PointSSIM-GeomAB[39] | 0.7747 | 0.7172 | 0.5494 | 0.5400 | 0.9072 | 0.8465 | 0.9100 | 0.8771 | 0.5787 | 0.5714
PointSSIM-GeomBA[39] | 0.7639 | 0.7118 | 0.5638 | 0.5714 | 0.9080 | 0.8427 | 0.9114 | 0.8741 | 0.5404 | 0.5470
PointSSIM-GeomSym[39] | 0.7720 | 0.7201 | 0.5561 | 0.5697 | 0.9098 | 0.8464 | 0.9124 | 0.8770 | 0.5755 | 0.5685

Correlation measures: Pearson’s linear correlation coefficient (PLCC) measures
the prediction accuracy of the objective metrics whereas Spearman’s rank-order
correlation coefficient (SROCC) measures the strength of prediction
monotonicity [42]. Following the recommendations [20, 42], a 4 parameter
polynomial function was fitted prior to evaluation. Both PLCC and SROCC, the
values are in the range [0, 1] and higher values indicate a better
correlation.

Figure 6: Ideal distributions of metric score differences for “Different vs
Similar” and “Better vs Worse” analysis. A greater metric score difference is
expected for different pairs in “Different vs Similar” analysis. For “Better
vs Worse” analysis, metric score differences are expected to be positive and
negative respectively for better and worse pairs.

Krasula’s method: This method evaluates the objective metric performances in 2
stages, namely “Different vs Similar” and “Better vs Worse”. For “Different vs
Similar” analysis, pairs of PPCs from the dataset are split into two
categories as pairs with (i.e., different) and without (i.e., similar)
statistically significant differences. For a given pair of PPC, one way ANOVA
followed by Tukey’s honest significance difference test [43] is used to
measure the statistical significance of the differences. Krasula’s method
assumes that the absolute difference of metric predictions for different pairs
should be larger than the similar pairs. To quantify the performance Receiver
Operating Characteristic (ROC) analysis is used and the performance of the
metrics are expressed as Area Under the ROC Curve (AUC). The second stage uses
the pairs that are identified as different in the first stage. In the “Better
vs Worse” stage, the aim is to measure the performance of metrics on
identifying the better PPC in pairs with statistically significant difference.
Performance of the metrics in “Better vs Worse” analysis can then be expressed
as correct classification percentage as well as AUC values similar to first
stage.

Figure 6 depicts the ideal distributions of the metric score differences for
each stage of the analysis. In “Different vs Similar” analysis, we expect
higher metric score differences for “Different” pairs and lower for “Similar”
pairs. In “Better vs Worse” analysis, after fixing the order of the pairs as
“Better” or “Worse”, we expect positive and negative metric score differences
respectively for each category of pairs.

###  VI-C Correlation Analysis Results

Table III presents the PLCC and SROCC of each metric. Metrics are categorized
into three as previously discussed in Section VI-A. First two column presents
the metrics’ PLCC and SROCC scores on the whole dataset. Moreover, metric
performances were evaluated for individual compression algorithms and the
results are presented in following columns as indicated above.

PCQM [38] performs the best compared to other selected metrics in the whole
dataset, despite its poor performance on prediction of GeoCNN compression
distortions. Among color-based metrics, we again notice a similar pattern on
the accuracy of metrics when it comes to GeoCNN compression distortions.
PointSSIM variants perform relatively better than other metrics in this
category.

Simple image-based metrics (e.g., MSE, PSNR, SSIM, MS-SSIM) have low accuracy
across all compression categories and consequently on the whole dataset. VMAF
shows the best performance among image-based metrics in the whole dataset. We
also observe a general trend among image-based metrics towards a lack of
accuracy on VPCC compression distortions.

To sum up, PCQM [38] performs the best on predicting GPCC-Predlift, GPCC-Raht
and VPCC compression distortions whereas D-JNDQ [32] provides the highest
accuracy on GeoCNN compression distortions.

###  VI-D Results of the Analysis by Krasula’s Method

Prior to the analysis, we pre-process the subjective scores as described in
Krasula’s method[44]. First, 20 PPCs were paired within each source point
cloud, generating $(20\times(20-1)/2)$ pairs per SRC. In total, we end up with
14143 pairs. Thanks to high number of PPCs in the dataset, we kept the
analysis within SRC. Afterwards, a one-way ANOVA test is applied to individual
scores collected for each stimulus in each pair, followed by Tukey’s Honest
test. 5019 pairs among the total 14143 were identified as “Similar” whereas
9124 contains a statistically significant different between the two PPC and
thus identified as “Different”. From those “Different” pairs, we split them
into two roughly equal sized groups as “Better” and “Worse” depending on the
order of the pair. There are 4075 “Better” and 5049 “Worse” pairs. We report
the result of the analysis on the top performing 10 metrics among the initial
40. Rest of the results can be found in the github repository, which will be
added after ICIP 2023 Grand Challenge is concluded.

####  VI-D1 Different vs Similar Analysis

Figure 7: Metric score differences for pairs categorized as “Different” and
“Similar”. Metric score differences are normalized individually for each
metric within minimum and maximum ranges. Height of the bars denote the
occurrences and each bar ranges between [0, 1500] for each plot. Metric names
are indicated at the top of each plot. Area under the curve (AUC) values are
reported below each metric name.

Figure 7 presents the results of the analysis as histograms of metric score
differences for “Different” and “Similar” pairs. We expect better performing
metrics to provide metric score distributions similar to the ideal case as
depicted in Figure 6. Additionally, performance of each metric quantified with
AUC values, reported under each metric name. In line with the results of the
correlation analysis, we observe a better performance from PCQM, providing a
higher AUC value and a very similar distribution to the ideal case.
Statistical significance tests on this task also reveals that PCQM performs
significantly better than all other metrics except ADM2 in “Different vs
Similar” task.

####  VI-D2 Better vs Worse Analysis

Figure 8: Metric score differences for pairs categorized as “Better” and
“Worse”. Metric score differences are normalized individually for each metric
within minimum and maximum ranges. Height of the bars denote the occurrences
of each bar ranges between [0, 800] for each plot. Metric names are indicated
at top left corner of each plot. Area under the curve (AUC) and correct
classification percentages (CC) are reported in top right corner of each plot.

Similar to the previous stage, Figure 8 presents the results as histograms of
metric score differences and quantifies the performance of each metric with
AUC and CC values. We observe that most metrics perform relatively well on
identifying “Better” and “Worse” pairs apart. PCQM performs significantly
better than all other metrics in this task.

##  VII Conclusion

We conducted a large-scale crowdsourcing study on point cloud compression
quality assessment. To the best of our knowledge, this is the largest publicly
available point cloud quality assessment dataset containing 75 source point
clouds each compressed with 4 different point cloud coding algorithms
resulting in nearly 1500 processed point clouds. More than 3500 naive
observers participated to do experiment.

Although most point cloud objective quality metrics accurately predicts GPCC
distortions, VPCC distortions still poses a challenge to majority of the
metrics. Moreover, most point cloud quality metrics fail to assess the quality
of the point clouds compressed with GeoCNN. There is definitely a room for
improvement for assessing learning-based coding algorithm related distortions.

We expect that the created database will provide a solid foundation for
further point cloud compression and point cloud quality assessment research.
This will then allow better telepresence experience in different applications
and platforms.

The point cloud compression and quality assessment are both hot topics and the
number of methods evolve rapidly. In the future, this database can be extended
with further with more learning-based approaches to find the challenges and
the limitations of the ongoing research and development on compression and
quality estimation research.

## Acknowledgments

This work has received funding from the European Union’s Horizon 2020 research
and innovation programme under the Marie Sklodowska-Curie Grant Agreement No.
765911 (RealVision) and from Science Foundation Ireland (SFI) under the Grant
Number 15/RP/27760.

[List of selected 3D models] A complete list of the selected 3D models will be
provided after the ICIP 2023 challenge is concluded, including various
metadata such as the PC ID (PC#), the semantic category (Cat), original 3D
model format, how the 3D model is captured, number of points in the reference
PC (Pt#), a brief explanation of contents, the name of the creator along with
a link to the original 3D model, and the copyright licence. All of the used 3D
models were licensed by a Creative Commons license .

## References

  * [1] A. Chetouani, G. Valenzise, A. Ak, E. Zerman, M. Quach, M. Tliba, M. A. Kerkouri, and P. Le Callet, “ICIP 2023 - Point cloud visual quality assessment grand challenge,” https://sites.google.com/view/icip2023-pcvqa-grand-challenge, 2023.
  * [2] R. Pagés, K. Amplianitis, D. Monaghan, J. Ondřej, and A. Smolić, “Affordable content creation for free-viewpoint video and VR/AR applications,” _Journal of Visual Communication and Image Representation_ , vol. 53, pp. 192–201, 2018.
  * [3] A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan, “High-quality streamable free-viewpoint video,” _ACM Trans. Graphics_ , vol. 34, no. 4, Jul. 2015.
  * [4] E. Zerman, C. Ozcinar, P. Gao, and A. Smolic, “Textured mesh vs coloured point cloud: A subjective study for volumetric video compression,” in _2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX)_ , 2020, pp. 1–6.
  * [5] E. Alexiou, Y. Nehmé, E. Zerman, I. Viola, G. Lavoué, A. Ak, A. Smolic, P. Le Callet, and P. Cesar, “Subjective and objective quality assessment for volumetric video,” in _Immersive Video Technologies_. Elsevier, 2023, pp. 501–552.
  * [6] S. Perry, H. P. Cong, L. A. da Silva Cruz, J. Prazeres, M. Pereira, A. Pinheiro, E. Dumic, E. Alexiou, and T. Ebrahimi, “Quality evaluation of static point clouds encoded using mpeg codecs,” in _2020 IEEE International Conference on Image Processing (ICIP)_ , 2020, pp. 3428–3432.
  * [7] L. A. da Silva Cruz, E. Dumić, E. Alexiou, J. Prazeres, R. Duarte, M. Pereira, A. Pinheiro, and T. Ebrahimi, “Point cloud quality evaluation: Towards a definition for test conditions,” in _2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX)_ , 2019, pp. 1–6.
  * [8] Q. Yang, H. Chen, Z. Ma, Y. Xu, R. Tang, and J. Sun, “Predicting the perceptual quality of point cloud: A 3d-to-2d projection-based exploration,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 3877–3891, 2021.
  * [9] H. Su, Z. Duanmu, W. Liu, Q. Liu, and Z. Wang, “Perceptual quality assessment of 3d point clouds,” in _2019 IEEE International Conference on Image Processing (ICIP)_ , 2019, pp. 3182–3186.
  * [10] L. Hua, M. Yu, G. yi Jiang, Z. He, and Y. Lin, “Vqa-cpc: a novel visual quality assessment metric of color point clouds,” in _SPIE/COS Photonics Asia_ , 2020.
  * [11] X. Wu, Y. Zhang, C. Fan, J. Hou, and S. Kwong, “Siat-pcqd: Subjective point cloud quality database with 6dof head-mounted display,” 2021. [Online]. Available: https://dx.doi.org/10.21227/ad8d-7r28
  * [12] S. Subramanyam, J. Li, I. Viola, and P. Cesar, “Comparing the quality of highly realistic digital humans in 3dof and 6dof: A volumetric video case study,” in _2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_ , 2020, pp. 127–136.
  * [13] E. Alexiou, N. Yang, and T. Ebrahimi, “PointXR: A toolbox for visualization and subjective evaluation of point clouds in virtual reality,” in _12th International Conference on Quality of Multimedia Experience (QoMEX)_ , 2020.
  * [14] D. B. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, and A. Tabatabai, “An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC),” _APSIPA Transactions on Signal and Information Processing_ , vol. 9, 2020.
  * [15] M. Quach, G. Valenzise, and F. Dufaux, “Improved deep point cloud geometry compression,” in _2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)_. IEEE, 2020, pp. 1–6.
  * [16] ——, “Learning convolutional transforms for lossy point cloud geometry compression,” in _2019 IEEE international conference on image processing (ICIP)_. IEEE, 2019, pp. 4320–4324.
  * [17] “Prolific,” https://www.prolific.com/, accessed: Nov. 2022. [Online].
  * [18] I. Viola, S. Subramanyam, J. Li, and P. Cesar, “On the impact of vr assessment on the quality of experience of highly realistic digital humans: A volumetric video case study,” _Quality and User Experience_ , vol. 7, 05 2022.
  * [19] Y. Nehmé, P. L. Callet, F. Dupont, J.-P. Farrugia, and G. Lavoué, “Exploring crowdsourcing for subjective quality assessment of 3d graphics,” in _2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP)_ , 2021, pp. 1–6.
  * [20] ITU-R, “Methodology for the subjective assessment of the quality of television pictures,” ITU-R Recommendation BT.500-14, 2019.
  * [21] R. K. Mantiuk, A. Tomaszewska, and R. Mantiuk, “Comparison of four subjective methods for image quality assessment,” _Comput. Graph. Forum_ , vol. 31, no. 8, p. 2478–2491, dec 2012. [Online]. Available: https://doi.org/10.1111/j.1467-8659.2012.03188.x
  * [22] E. Zerman, V. Hulusic, G. Valenzise, R. Mantiuk, and F. Dufaux, “The relation between MOS and pairwise comparisons and the importance of cross-content comparisons,” in _Human Vision and Electronic Imaging Conference, IS &T International Symposium on Electronic Imaging (EI 2018)_, Burlingame, United States, Jan. 2018. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01654133
  * [23] Y. Nehmé, J.-P. Farrugia, F. Dupont, P. LeCallet, and G. Lavoué, “Comparison of subjective methods, with and without explicit reference, for quality assessment of 3d graphics,” in _ACM Symposium on Applied Perception 2019_ , ser. SAP ’19. New York, NY, USA: Association for Computing Machinery, 2019. [Online]. Available: https://doi.org/10.1145/3343036.3352493
  * [24] A. Goswami, A. Ak, W. Hauser, P. L. Callet, and F. Dufaux, “Reliability of crowdsourcing for subjective quality evaluation of tone mapping operators,” in _2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP)_ , 2021, pp. 1–6.
  * [25] S. Palan and C. Schitter, “Prolific.ac—a subject pool for online experiments,” _Journal of Behavioral and Experimental Finance_ , vol. 17, pp. 22–27, 2018.
  * [26] Z. Li and C. G. Bampis, “Recover subjective quality scores from noisy measurements,” in _2017 Data compression conference (DCC)_. IEEE, 2017, pp. 52–61.
  * [27] E. Zerman, C. Ozcinar, P. Gao, and A. Smolic, “Textured mesh vs coloured point cloud: A subjective study for volumetric video compression,” in _12th International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, May 2020.
  * [28] A. Ak, E. Zerman, S. Ling, P. L. Callet, and A. Smolic, “The effect of temporal sub-sampling on the accuracy of volumetric video quality assessment,” in _2021 Picture Coding Symposium (PCS)_ , 2021, pp. 1–5.
  * [29] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [30] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarity index for image quality assessment,” _IEEE Transactions on Image Processing_ , vol. 20, no. 8, pp. 2378–2386, 2011.
  * [31] W. Xue, L. Zhang, X. Mou, and A. C. Bovik, “Gradient magnitude similarity deviation: A highly efficient perceptual image quality index,” _IEEE Transactions on Image Processing_ , vol. 23, no. 2, pp. 684–695, 2014.
  * [32] A. Ak, A. Pastor, and P. Le Callet, “From just noticeable differences to image quality,” in _Proceedings of the 2nd Workshop on Quality of Experience in Visual Multimedia Applications_ , ser. QoEVMA ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 23–28. [Online]. Available: https://doi.org/10.1145/3552469.3555712
  * [33] D. Sandić-Stanković, D. Kukolj, and P. Le Callet, “DIBR synthesized image quality assessment based on morphological wavelets,” in _2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX)_ , 2015, pp. 1–6.
  * [34] D. Sandić-Stanković, D. Kukolj, and P. Le Callet, “DIBR-synthesized image quality assessment based on morphological multi-scale approach,” _EURASIP Journal on Image and Video Processing_ , vol. 2017, 07 2016.
  * [35] R. Mekuria, Z. Li, C. Tulvan, and P. Chou, “Evaluation criteria for PCC (Point Cloud Compression),” ISO/IEC JTC 1/SC29/WG11 Doc. N16332, 2016.
  * [36] D. Tian, H. Ochimizu, C. Feng, R. Cohen, and A. Vetro, “Evaluation metrics for point cloud compression,” ISO/IEC JTC 1/SC29/WG11 Doc. M39966, 2017.
  * [37] E. Alexiou and T. Ebrahimi, “Point cloud quality assessment metric based on angular similarity,” in _International Conference on Multimedia & Expo (ICME)_, 2018.
  * [38] G. Meynet, Y. Nehmé, J. Digne, and G. Lavoué, “PCQM: A full-reference quality metric for colored 3D point clouds,” in _12th International Conference on Quality of Multimedia Experience (QoMEX)_ , 2020.
  * [39] E. Alexiou and T. Ebrahimi, “Towards a point cloud structural similarity metric,” in _2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_, 2020, pp. 1–6.
  * [40] D. Tian, H. Ochimizu, C. Feng, R. Cohen, and A. Vetro, “Geometric distortion metrics for point cloud compression,” in _IEEE International Conference on Image Processing (ICIP)_ , Sept 2017, pp. 3460–3464.
  * [41] E. Alexiou and T. Ebrahimi, “Point cloud quality assessment metric based on angular similarity,” in _International Conference on Multimedia & Expo (ICME)_, 2018.
  * [42] ITU-R, “Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models,” ITU-R Recommendation P.1401, 2020.
  * [43] J. W. Tukey, “Comparing individual means in the analysis of variance.” _Biometrics_ , vol. 5 2, pp. 99–114, 1949.
  * [44] L. Krasula, K. Fliegel, P. Le Callet, and M. Klíma, “On the accuracy of objective image and video quality models: New methodology for performance evaluation,” in _2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX)_ , 2016, pp. 1–6.


# Multifunctional Superconducting Nanowire Quantum Sensors

Benjamin J. Lawrie<EMAIL_ADDRESS>This manuscript has been authored by
UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of
Energy (DOE). The US government retains and the publisher, by accepting the
article for publication, acknowledges that the US government retains a
nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce
the published form of this manuscript, or allow others to do so, for US
government purposes. DOE will provide public access to these results of
federally sponsored research in accordance with the DOE Public Access Plan
(http://energy.gov/downloads/doe-public-access-plan).  Materials Science and
Technology Division, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak
Ridge, TN 37831  Claire E. Marvinney<EMAIL_ADDRESS>Materials Science
and Technology Division, Oak Ridge National Laboratory, 1 Bethel Valley Rd,
Oak Ridge, TN 37831  Yun-Yi Pai  Materials Science and Technology Division,
Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37831
Matthew A. Feldman  Materials Science and Technology Division, Oak Ridge
National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37831  Jie Zhang
Materials Science and Technology Division, Oak Ridge National Laboratory, 1
Bethel Valley Rd, Oak Ridge, TN 37831  Aaron J. Miller  Quantum Opus LLC,
22241 Roethel Dr Ste A, Novi, MI 48375  Chengyun Hua  Materials Science and
Technology Division, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak
Ridge, TN 37831  Eugene Dumitrescu  Computational Science and Engineering
Division, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN
37831  Gábor B. Halász  Materials Science and Technology Division, Oak Ridge
National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37831

###### Abstract

Superconducting nanowire single photon detectors (SNSPDs) offer high-quantum-
efficiency and low-dark-count-rate single photon detection. In a growing
number of cases, large magnetic fields are being incorporated into quantum
microscopes, nanophotonic devices, and sensors for nuclear and high-energy
physics that rely on SNSPDs, but superconducting devices generally operate
poorly in large magnetic fields. Here, we demonstrate robust performance of
amorphous SNSPDs in magnetic fields of up to $\pm 6$ T with a negligible dark
count rate and unchanged quantum efficiency at typical bias currents.
Critically, we also show that in the electrothermal oscillation regime, the
SNSPD can be used as a magnetometer with sensitivity of better than 100
$\mathrm{\mu T/\sqrt{Hz}}$ and as a thermometer with sensitivity of 20
$\mathrm{\mu K/\sqrt{Hz}}$ at 1 K. Thus, a single photon detector integrated
into a quantum device can be used as a multifunctional quantum sensor capable
of describing the temperature and magnetic field on-chip simply by varying the
bias current to change the operating modality from single photon detection to
thermometry or magnetometry.

###### pacs:

Valid PACS appear here

Superconducting nanowire single photon detectors (SNSPDs) offer high speed,
high quantum-efficiency, and low dark-count-rate single photon detection [1].
There is an emerging need for SNSPDs capable of operating in large magnetic
fields for potential integration with quantum nanophotonic circuits [2, 3, 4]
and for quantum sensors relevant to nuclear physics [5] and dark matter
detection [6, 7, 8]. In general, SNSPDs do not perform well in large magnetic
fields, but some research has demonstrated that small magnetic fields can
improve SNSPD performance as described below.

SNSPDs can detect photons when (a) the photon provides enough energy to break
an ensemble of Cooper pairs and generate a bath of quasiparticles that form a
belt across the width of the nanowire, (b) the photon provides enough energy
to unpin a vortex, enabling it to sweep across the nanowire under a Lorentz
force, and (c) the photon provides enough energy to form and unbind a vortex-
antivortex pair that are swept in opposite directions across the nanowire
under a Lorentz force. Field-dependent studies of bright count rates have
suggested that vortex motion is the primary detection mechanism in typical NbN
SNSPDs [9, 10]. Similar measurements of $\mathrm{Mo_{x}Si_{1-x}}$ SNSPDs with
varying wire width have concluded that vortex-antivortex interactions are
responsible for bright counts in wide devices made from micron-scale wires
while direct quasiparticle belt formation is responsible for bright counts in
nanowires of order 100 nm in width [11]. A growing body of literature has made
it clear that SNSPDs constructed of different materials with different designs
can rely on any of the bright-count mechanisms described above [12].

Dark counts are observed as a result of thermally induced vortex motion in the
absence of any photons, and are generally present even in well shielded
detectors operated at 0 T [13, 14]. Because the local vortex pinning potential
can be a spatially heterogeneous function of the superconducting film growth
conditions and the device geometry, dark counts may arise preferentially from
weak spots within the device. Early models suggested that the energy barrier
for single vortex crossing is lower than that for phase slips and vortex-
antivortex nucleation and annihilation [13] and that vortex-antivortex
interactions and phase slips may be ignored, but competing models and
measurements have suggested that vortex-antivortex interactions resulting from
a Berezinskii-Kosterlitz-Thouless transition are responsible for SNSPD dark
counts [15, 16, 17].

At high bias currents, hotspots periodically form and disappear with frequency
determined by the device inductance and load impedance, leading to detection
of dark counts at this electrothermal oscillation frequency [18, 19, 20, 21,
22]. The propagation velocity of a normal-superconducting boundary in the
nanowire due to Joule heating depends on its critical current [19], which
itself has a strong dependence on the applied external magnetic field. While
detection events in the electrothermal oscillation regime are a result of a
fundamentally different physical process than dark counts detected at lower
bias currents, we describe both conventional dark counts and electrothermal
oscillations as dark counts in this article for linguistic convenience.

Current crowding at sharp bends can result in a reduced potential barrier for
vortex motion and an increased vortex nucleation density. However, external
magnetic fields and associated Meissner currents can reduce the effect of
current crowding [23, 24]. Early modeling efforts suggested that perpendicular
magnetic fields would reduce the critical current for a conventional meander
line SNSPD, but small negative perpendicular magnetic fields could increase
the critical current of devices patterned in a spiral layout [23]. Other
experimental efforts found a slight asymmetry in the dark count rate of TaN
and NbN SNSPDs as a function of field [25, 26], consistent with previous
modeling [23], subject to the assumption that dark counts originate largely at
the sharp corners of the device and allowing for some heterogeneity in the
device fabrication [25].

Understanding and controlling vortex motion and hotspot formation in SNSPDs is
critical to the development of quantum sensors capable of operating in high
magnetic fields, but fully predictive models of single-photon interactions
with superconducting nanowires remain a challenge, despite the recent
demonstration of a probabilistic criterion for single photon detection based
on single vortex motion [27]. Very little work has explored the field
dependence of SNSPDs in large magnetic fields, though one recent article did
demonstrate that NbN nanowire single photon detectors can be operated in
fields as high as 5 T [28], and many researchers have explored the response of
superconducting thin films and nanowires in larger magnetic fields [29, 30,
31, 32, 33, 34]. Further research is needed to provide an improved
understanding of vortex motion in superconducting nanowires and to define the
limits of high-field SNSPD operation.

Figure 1: Measured count rate as a function of bias current and magnetic field
for bias currents of 0-14 $\mu$A and magnetic fields of -6 to 6 T oriented
roughly parallel to the film and $\sim 45^{\circ}$ to the length of the
nanowires. Dark counts begin to contaminate the bright count measurements at
bias currents above 10.0 $\mu$A. Inset of the bias current at which the dark
counts are equal to 10x the plateau bright counts, emphasizing the asymmetry
of the field dependence of dark counts at high bias currents.

Here, we explore the magnetic field dependence of dark counts and bright
counts generated by a commercially available amorphous transition-metal
silicide near-infrared SNSPD with a critical temperature of 5 K. In order to
minimize the density of vortices generated by the magnetic field, we focus on
magnetic fields oriented parallel to the device. The 7 nm thick SNSPD used
here has a 70 nm wide meander line with a 50 percent fill fraction that spans
an 11 x 11 $\mu$m active area (Quantum Opus, LLC). It was mounted on a PCB and
suspended in a dilution refrigerator with free-space optical access to the
mixing chamber at an angle of $<5^{\circ}$ from a magnetic field that was
swept from -6 to 6 T. The SNSPD meander line pattern was oriented at an angle
of $\sim 45^{\circ}$ relative to the field. Unless otherwise specified, the
SNSPD temperature was held at 100 mK. An attenuated 1062 nm CW laser source
was delivered to the SNSPD through a free-space optical interface with the
dilution refrigerator that is described elsewhere [35]. Dark counts were
collected with optical access to the dilution refrigerator blocked. All
signals were passed through a low frequency filter mounted directly to the
SNSPD at the mixing chamber in order to minimize latching and enable high
count rate detection [36]. The bias current and the SNSPD waveforms were
delivered along the same semirigid coaxial channel with a room temperature
bias tee and low noise amplifiers integrated before time tagging all detection
events with a 62 mV threshold. Data was collected (1) continuously for counts
as a function of magnetic field for a constant bias current and (2) in 1 T
increments for counts as a function of bias current.

Figure 2: Dark count rate as a function of field for bias currents of
11.10-13.30 $\mu$A.

The magnetic field dependence of the measured counts is shown as a function of
bias current in Fig. 1. At each point on the curve, the bright count rate was
averaged for 20 seconds and the dark count rate was averaged for 10 seconds.
At all field values, the relative quantum efficiency of the device remains
unchanged at bias currents within the quantum efficiency plateau between
7.5-9.5 $\mu$A. Additionally, the dark count rate was measured to be $<0.1$
counts per second within the same 7.5-9.5 $\mu$A range for all measured
fields. However, as emphasized in the inset of Fig. 1, the maximum operating
bias current is suppressed in an asymmetric manner with increasing field, with
positive parallel magnetic fields reducing the maximum operational bias
current to $\sim$ 10.0 $\mu$A at 6 T, while the maximum operational bias
current at -6 T is only reduced to $\sim$ 11.5 $\mu$A. A similar, if less
pronounced asymmetry is present in the onset of bright count detection events
at bias currents of 4-5.5 $\mu$A as seen in Fig. 1.

This asymmetric magnetic field response leads to a decrease in the dark count
rate of the device at some negative fields, and thus an improved operation of
the device between 0 T and -2.5 T, as shown in Fig. 2. This asymmetry was
repeatable for multiple zero-field recooling measurements. The transport
properties of the high-count-rate board were also tested separately as a
function of field, resulting in no evidence of asymmetry. Thus the asymmetric
response shown in Fig. 2 appears to be due to the SNSPD itself. While slight
asymmetries in the field-dependent SNSPD response have been reported before
for mT-scale perpendicular fields and ascribed to magnetic-field induced
currents that oppose current crowding effects at the corners of the meander
line [25, 26, 23], the asymmetries reported here rely on similar bias currents
but magnetic fields three orders of magnitude larger. Thus, the previous
models for asymmetric SNSPD responses to magnetic fields do not appear to
explain this result. Notably, while the onset of bright counts and dark counts
is a function of magnetic field, and while the SNSPD waveform can be a
function of the photon-nanowire interaction parameters [22, 37], the SNSPD
waveforms recorded here were independent of field.

While a detailed model of this field-reversal asymmetry is beyond the scope of
this article, we can understand why the asymmetry is much larger than in
previous works by recognizing the different orientation of the magnetic field
and considering the symmetries of the meander line and the bias current. The
amorphous superconductor is deposited onto a 120 nm $\mathrm{SiO_{2}}$
backreflector and the meander line is then capped with a 1.5 nm conformal
$\mathrm{Al_{2}O_{3}}$ coating to prevent oxidation before 100 nm
$\mathrm{SiO_{2}}$ and $\mathrm{Si_{3}N_{4}}$ antireflective coating layers
are deposited. If the top and bottom interfaces of the SNSPD are not
identical, as is the case here, a field with a finite component along the long
line segments is expected to have a field-reversal asymmetry. Indeed, based on
the model in Ref. [23], such a field should, depending on its sign, induce
current crowding close to the top or the bottom interface, thus giving
different rates for positive and negative fields. Moreover, even if the two
interfaces are identical, a field-reversal asymmetry is expected if all three
components of the field (parallel to the long line segments, the short line
segments, and the perpendicular direction, respectively) are finite. In
contrast, such a field-reversal asymmetry is forbidden for a purely
perpendicular field, regardless of whether the top and bottom interfaces are
identical or distinct. For a perpendicular field, it can then only appear due
to imperfections of the meander line [25] and is expected to be much smaller.

The magnetic-field dependence of the dark counts in the high-bias-current
regime is intriguing because it suggests that the SNSPD can be utilized as an
on-chip magnetometer. For a well characterized SNSPD, monitoring changes in
the dark count rate after initially setting the bias current to achieve dark
counts of $\sim 10^{2}-10^{6}cps$ provides magnetic field sensitivity
determined by the slope of the curves shown in Figs 2 and 3 and limited by the
uncertainty in counting statistics and bias current. Here, we assume that the
field sensitivity can be calculated based on exponential fits to the measured
dark counts and that the measured dark counts exhibit Poissonian counting
statistics with uncertainty in count rate scaling as $\sqrt{N}$. Uncertainty
in the bias current is neglected as a small component relative to the
uncertainty in count rate. Further, uncertainty in photon counting is a
fundamental limit that cannot be improved upon for this type of measurement,
whereas the uncertainty in bias current can be improved beyond current limits
with further engineering.

The maximum sensitivity of the curves shown in Fig. 2 ranges from 1.8
$mT/\sqrt{Hz}$ for a bias current of 13.3 $\mu$A at -1 T to 600 $\mu
T/\sqrt{Hz}$ for a bias current of 11.10 $\mu$A at 6T, with the best
sensitivity at smaller bias currents closer to the onset of the electrothermal
oscillation regime. The best sensitivity measured here was 75 $\mu
T/\sqrt{Hz}$ for a bias current of 11.60 $\mu A$ at 4.8 T. Thus, the SNSPD can
be used as a magnetometer by sweeping the bias current while monitoring the
dark count rate to coarsely determine the magnetic field and by monitoring
changes in the dark count rate at a constant bias current to track smaller
changes in the magnetic field.

Figure 3: Dark count rate as a function of field for bias currents of
12.10-12.75 $\mu$A at $\sim$600 mK.

Extrapolating the results shown in Fig. 1 to larger fields suggests that a
plateau with optimized quantum efficiency and minimized dark counts should
still exist for fields as large as 18 T and -68 T. It is likely that this
extrapolation overestimates the robustness of the SNSPD, as the SNSPD
operation relies in part on the description of the SNSPD as a 2D
superconductor. For fields above 6.5 T, the characteristic length scale
$\sqrt{\hbar/2eB}$ is smaller than the 7 nm film thickness, and that
description fails [30]. Nonetheless, it is clear from the results in Fig. 1
that the SNSPD is capable of robust operation for fields much larger than 6 T.

At higher temperatures, the maximum functional bias current is reduced as
thermally induced vortex motion begins to increase, but the same robust high
quantum-efficiency, low dark-count-rate operation is observed at all measured
fields for bias currents of 6-12.1 $\mu$A. As seen in Fig. 3 for dark counts
measured at $\sim 600$ mK, a similar asymmetry is present in the dark count
rate, with the minimum dark counts measured for fields between -1 T and -2 T.

Because this SNSPD was mounted to a PCB at a distance of several inches from
the nearest thermometer, there is some uncertainty in the SNSPD temperature
that would normally be challenging to quantify. However, the dark counts can
be used for thermometry just as they can be used for magnetometry. Figure 4
illustrates the measured dark counts for 12.60 $\mu A$ and 12.75 $\mu A$ bias
currents at 0 T. As above, the sensitivity of this superconducting nanowire
thermometer can be calculated by curve fitting the dark count rate and
assuming that the uncertainty in the count rate is limited by the photon
counting statistics. The data shown in Fig. 4a illustrate temperature
sensitivity of 20 $\mu K/\sqrt{Hz}$ at a temperature of 1 K and a bias of
12.75 $\mu A$, and a temperature sensitivity of 45 $\mu K/\sqrt{Hz}$ for a
12.60 $\mu A$ current. At 800 mK, the sensitivity is reduced to 0.1
$mK/\sqrt{Hz}$ and 1.5 $mK/\sqrt{Hz}$, and at 400 mK, the sensitivity is
reduced to 6 $mK/\sqrt{Hz}$ and 40 $mK/\sqrt{Hz}$, respectively at the larger
and smaller bias currents. However, further increasing the bias current does
continue to increase the sensitivity, and the SNSPD thermometer is most
sensitive when the device is operating in the electrothermal oscillation
regime with greater than $10^{6}$ dark counts per second. With the device
operating at a higher bias current, this regime can be reached at lower
temperatures, and the onset of the electrothermal oscillations near 100 mK
will improve the temperature sensitivity. For example, on the cusp of the
electrothermal oscillating regime at 60 mK, the sensitivity is improved to
better than 0.5 $mK/\sqrt{Hz}$, as shown with a bias current of 12.95 $\mu A$
in Fig. 4b. This improved sensitivity comes at the expense of increasing the
device temperature $\sim 5$ mK at a bias current of 14 $\mu A$.

Figure 4: Temperature dependent dark counts for 12.60 $\mu A$ and 12.75 $\mu
A$ bias currents at 0 T. The inset illustrates the temperature sensitivity at
lower temperatures for a 12.95 $\mu A$ bias current at 0T.

The ability to use the SNSPD in large magnetic fields for thermometry or
magnetometry in addition to single photon detection is of interest, not just
to better quantify the SNSPD operating conditions, but to provide a
multifunctional sensor in integrated quantum nanophotonic devices. While the
results described in this article provide a device-level understanding of the
limitations of thermometry and magnetometry with SNSPDs in large fields, a
microscopic understanding of the quasiparticle and vortex interactions that
drive this functionality is still needed. Confocal optical microscopies
capable of monitoring the SNSPD response as a function of the position and
wavelength of incident photons at milliKelvin temperatures and in variable
magnetic fields would provide an improved understanding of these interactions
at the mesoscale. Additionally, a quantitative model of the magnetic field
dependent and temperature dependent electrothermal oscillation frequencies
will be pursued as a future work. MilliKelvin scanning probe microscopies
including scanning SQUID microscopies [38] and scanning single photon
microscopies [35] could help to provide further understanding of these
interactions.

###### Acknowledgements.

This research was sponsored by the U. S. Department of Energy, Office of
Science, Basic Energy Sciences, Materials Sciences and Engineering Division.
Postdoctoral (CEM) research support was provided by the Intelligence Community
Postdoctoral Research Fellowship Program at the Oak Ridge National Laboratory,
administered by Oak Ridge Institute for Science and Education through an
interagency agreement between the U.S. Department of Energy and the Office of
the Director of National Intelligence. Student (MAF, BEL) research support
were provided by the Department of Defense through the National Defense
Science $\&$ Engineering Graduate Fellowship (NDSEG) and by the DOE Science
Undergraduate Laboratory Internships (SULI) program.

## References

  * Natarajan  _et al._ [2012] C. M. Natarajan, M. G. Tanner, and R. H. Hadfield, Superconducting nanowire single-photon detectors: physics and applications, Superconductor science and technology 25, 063001 (2012).
  * Khasminskaya  _et al._ [2016] S. Khasminskaya, F. Pyatkov, K. Słowik, S. Ferrari, O. Kahl, V. Kovalyuk, P. Rath, A. Vetter, F. Hennrich, M. M. Kappes, _et al._ , Fully integrated quantum photonic circuit with an electrically driven light source, Nature Photonics 10, 727 (2016).
  * Dreyer  _et al._ [2018] C. E. Dreyer, A. Alkauskas, J. L. Lyons, A. Janotti, and C. G. Van de Walle, First-principles calculations of point defects for quantum technologies, Annual Review of Materials Research 48, 1 (2018).
  * Rath  _et al._ [2015] P. Rath, O. Kahl, S. Ferrari, F. Sproll, G. Lewes-Malandrakis, D. Brink, K. Ilin, M. Siegel, C. Nebel, and W. Pernice, Superconducting single-photon detectors integrated with diamond nanophotonic circuits, Light: Science & Applications 4, e338 (2015).
  * Azzouz  _et al._ [2012] H. Azzouz, S. N. Dorenbos, D. De Vries, E. B. Ureña, and V. Zwiller, Efficient single particle detection with a superconducting nanowire, AIP Advances 2, 032124 (2012).
  * Berggren [2021] K. K. Berggren, Snowmass2021-letter of interest dark matter detection using layered materials and superconducting nanowires, SnowMass (2021).
  * JPL  _et al._ [2021] J. A. JPL, J. Chiles, B. K. JPL, A. Lita, J. Luskin, S. W. Nam, M. S. JPL, V. Verma, and E. W. JPL, Snowmass2021-letter of interest superconducting nanowire single-photon detectors, SnowMass (2021).
  * Hochberg  _et al._ [2019] Y. Hochberg, I. Charaev, S.-W. Nam, V. Verma, M. Colangelo, and K. K. Berggren, Detecting sub-gev dark matter with superconducting nanowires, Physical review letters 123, 151802 (2019).
  * Vodolazov  _et al._ [2015] D. Y. Vodolazov, Y. P. Korneeva, A. Semenov, A. Korneev, and G. Goltsman, Vortex-assisted mechanism of photon counting in a superconducting nanowire single-photon detector revealed by external magnetic field, Physical Review B 92, 104503 (2015).
  * Lusche  _et al._ [2014] R. Lusche, A. Semenov, Y. Korneeva, A. Trifonov, A. Korneev, G. Gol’Tsman, and H.-W. Hübers, Effect of magnetic field on the photon detection in thin superconducting meander structures, Physical Review B 89, 104513 (2014).
  * Korneeva  _et al._ [2020] Y. P. Korneeva, N. Manova, I. Florya, M. Y. Mikhailov, O. Dobrovolskiy, A. Korneev, and D. Y. Vodolazov, Different single-photon response of wide and narrow superconducting mo x si 1- x strips, Physical Review Applied 13, 024011 (2020).
  * Engel  _et al._ [2014] A. Engel, J. Lonsky, X. Zhang, and A. Schilling, Detection mechanism in snspd: numerical results of a conceptually simple, yet powerful detection model, IEEE Transactions on Applied Superconductivity 25, 1 (2014).
  * Bulaevskii  _et al._ [2011] L. Bulaevskii, M. Graf, C. Batista, and V. Kogan, Vortex-induced dissipation in narrow current-biased thin-film superconducting strips, Physical Review B 83, 144526 (2011).
  * Bulaevskii  _et al._ [2012] L. N. Bulaevskii, M. J. Graf, and V. G. Kogan, Vortex-assisted photon counts and their magnetic field dependence in single-photon superconducting detectors, Physical Review B 85, 014505 (2012).
  * Yamashita  _et al._ [2011] T. Yamashita, S. Miki, K. Makise, W. Qiu, H. Terai, M. Fujiwara, M. Sasaki, and Z. Wang, Origin of intrinsic dark count in superconducting nanowire single-photon detectors, Applied Physics Letters 99, 161105 (2011).
  * Engel  _et al._ [2006] A. Engel, A. Semenov, H.-W. Hübers, K. Il’in, and M. Siegel, Fluctuation effects in superconducting nanostrips, Physica C: Superconductivity and its applications 444, 12 (2006).
  * Kitaygorsky  _et al._ [2007] J. Kitaygorsky, I. Komissarov, A. Jukna, D. Pan, O. Minaeva, N. Kaurova, A. Divochiy, A. Korneev, M. Tarkhov, B. Voronov, _et al._ , Dark counts in nanostructured nbn superconducting single-photon detectors and bridges, IEEE Transactions on Applied Superconductivity 17, 275 (2007).
  * Hadfield  _et al._ [2005] R. Hadfield, A. J. Miller, S. W. Nam, R. L. Kautz, and R. E. Schwall, Low-frequency phase locking in high-inductance superconducting nanowires, Applied Physics Letters 87, 203505 (2005).
  * Kerman  _et al._ [2009] A. J. Kerman, J. K. Yang, R. J. Molnar, E. A. Dauler, and K. K. Berggren, Electrothermal feedback in superconducting nanowire single-photon detectors, Physical review B 79, 100509 (2009).
  * Liu  _et al._ [2013] D. Liu, L. You, S. Chen, X. Yang, Z. Wang, Y. Wang, X. Xie, and M. Jiang, Electrical characteristics of superconducting nanowire single photon detector, IEEE transactions on applied superconductivity 23, 2200804 (2013).
  * Berggren  _et al._ [2018] K. K. Berggren, Q.-Y. Zhao, N. Abebe, M. Chen, P. Ravindran, A. McCaughan, and J. C. Bardin, A superconducting nanowire can be modeled by using spice, Superconductor Science and Technology 31, 055010 (2018).
  * Marvinney  _et al._ [2021] C. E. Marvinney, B. E. Lerner, A. A. Puretzky, A. J. Miller, and B. J. Lawrie, Waveform analysis of a large-area superconducting nanowire single photon detector, Superconductor Science and Technology 34, 035020 (2021).
  * Clem  _et al._ [2012] J. R. Clem, Y. Mawatari, G. Berdiyorov, and F. Peeters, Predicted field-dependent increase of critical currents in asymmetric superconducting nanocircuits, Physical Review B 85, 144511 (2012).
  * Ilin and Siegel [2014] K. Ilin and M. Siegel, Magnetic field stimulated enhancement of the barrier for vortex penetration in bended bridges of thin tan films, Physica C: Superconductivity and its Applications 503, 58 (2014).
  * Engel  _et al._ [2012] A. Engel, A. Schilling, K. Il’in, and M. Siegel, Dependence of count rate on magnetic field in superconducting thin-film tan single-photon detectors, Physical Review B 86, 140506 (2012).
  * Charaev  _et al._ [2019] I. Charaev, A. Semenov, K. Ilin, and M. Siegel, Magnetic-field enhancement of performance of superconducting nanowire single-photon detector, IEEE Transactions on Applied Superconductivity 29, 1 (2019).
  * Jahani  _et al._ [2020] S. Jahani, L.-P. Yang, A. Buganza Tepole, J. C. Bardin, H. X. Tang, and Z. Jacob, Probabilistic vortex crossing criterion for superconducting nanowire single-photon detectors, Journal of Applied Physics 127, 143101 (2020).
  * Polakovic  _et al._ [2020] T. Polakovic, W. Armstrong, V. Yefremenko, J. Pearson, K. Hafidi, G. Karapetrov, Z.-E. Meziani, and V. Novosad, Superconducting nanowires as high-rate photon detectors in strong magnetic fields, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 959, 163543 (2020).
  * Gardner  _et al._ [2011] H. J. Gardner, A. Kumar, L. Yu, P. Xiong, M. P. Warusawithana, L. Wang, O. Vafek, and D. G. Schlom, Enhancement of superconductivity by a parallel magnetic field in two-dimensional superconductors, Nature physics 7, 895 (2011).
  * Rosenbaum [1985] R. Rosenbaum, Superconducting fluctuations and magnetoconductance measurements of thin films in parallel magnetic fields, Phys. Rev. B 32, 2190 (1985).
  * Rouco  _et al._ [2019] V. Rouco, C. Navau, N. Del-Valle, D. Massarotti, G. P. Papari, D. Stornaiuolo, X. Obradors, T. Puig, F. Tafuri, A. Sanchez, _et al._ , Depairing current at high magnetic fields in vortex-free high-temperature superconducting nanowires, Nano letters 19, 4174 (2019).
  * Córdoba  _et al._ [2013] R. Córdoba, T. Baturina, J. Sesé, A. Y. Mironov, J. De Teresa, M. Ibarra, D. Nasimov, A. Gutakovskii, A. Latyshev, I. Guillamón, _et al._ , Magnetic field-induced dissipation-free state in superconducting nanostructures, Nature communications 4, 1 (2013).
  * Samkharadze  _et al._ [2016] N. Samkharadze, A. Bruno, P. Scarlino, G. Zheng, D. DiVincenzo, L. DiCarlo, and L. Vandersypen, High-kinetic-inductance superconducting nanowire resonators for circuit qed in a magnetic field, Physical Review Applied 5, 044004 (2016).
  * Zhang  _et al._ [2016] X. Zhang, A. Engel, Q. Wang, A. Schilling, A. Semenov, M. Sidorova, H.-W. Hübers, I. Charaev, K. Ilin, and M. Siegel, Characteristics of superconducting tungsten silicide w x s i 1- x for single photon detection, Physical Review B 94, 174509 (2016).
  * Lawrie  _et al._ [2021] B. Lawrie, M. Feldman, C. Marvinney, and Y. Pai, Free-space confocal magneto-optical spectroscopies at millikelvin temperatures, arXiv preprint arXiv:2103.06851 (2021).
  * Miller and Rambo [2020] A. Miller and T. Rambo, Noise suppressing interface circuit for device with control circuits in different noise environments (2020), US Patent App. 16/399,207.
  * Cahall  _et al._ [2017] C. Cahall, K. L. Nicolich, N. T. Islam, G. P. Lafyatis, A. J. Miller, D. J. Gauthier, and J. Kim, Multi-photon detection using a conventional superconducting nanowire single-photon detector, Optica 4, 1534 (2017).
  * Ceccarelli  _et al._ [2019] L. Ceccarelli, D. Vasyukov, M. Wyss, G. Romagnoli, N. Rossi, L. Moser, and M. Poggio, Imaging pinning and expulsion of individual superconducting vortices in amorphous mosi thin films, Physical Review B 100, 104504 (2019).