# Cross-modal Learning for Image-Guided Point Cloud Shape Completion

Emanuele Aiello
Politecnico di Torino, Italy
<EMAIL_ADDRESS>
&Diego Valsesia
Politecnico di Torino, Italy
<EMAIL_ADDRESS>
Enrico Magli
Politecnico di Torino, Italy
<EMAIL_ADDRESS>
Code of the project: https://github.com/diegovalsesia/XMFnet

###### Abstract

In this paper we explore the recent topic of point cloud completion, guided by
an auxiliary image. We show how it is possible to effectively combine the
information from the two modalities in a localized latent space, thus avoiding
the need for complex point cloud reconstruction methods from single views used
by the state-of-the-art. We also investigate a novel weakly-supervised setting
where the auxiliary image provides a supervisory signal to the training
process by using a differentiable renderer on the completed point cloud to
measure fidelity in the image space. Experiments show significant improvements
over state-of-the-art supervised methods for both unimodal and multimodal
completion. We also show the effectiveness of the weakly-supervised approach
which outperforms a number of supervised methods and is competitive with the
latest supervised models only exploiting point cloud information.

##  1 Introduction

The rise in popularity of 3D sensing technologies such as depth cameras, laser
scanners, LiDARs, etc. is making the processing of point cloud data ever more
important. The acquisitions produced by those instruments are often incomplete
due to occlusions by objects in the environment, reflections, and viewing
angles. This limits the exploitability of those data in tasks like scene
understanding [1], robotic vision [2], autonomous driving [3] and many more.
Completing a point cloud from partial observations is a challenging ill-posed
inverse problem that requires strong prior knowledge about shapes to be
effectively regularized.

At the same time, we know that humans are very proficient at mapping the
visual concepts learnt from 2D images to understand the 3D world, and are able
to successfully infer the shape of partial 3D objects from their 2D
experiences. It is thus sensible to expect that point cloud completion
techniques can benefit from 2D images to better characterize the 3D shape to
be completed. Indeed, several applications of interest in robotic vision can
take advantage of multimodal data where the 3D acquisitions of a depth-sensing
instrument are paired with images from an RGB camera. It is also worth noting
that the two modalities may be acquired from different vantage points, either
thanks to disparities in the acquisition geometry or because the vantage point
has changed with the passing of time. This makes it clear that the two
modalities may carry complementary information and effectively fusing it is
key to unlock better completion performance. Nevertheless, the literature on
the topic of point cloud completion [4, 5, 6, 7, 8, 9, 10, 11] has largely
focused on the single-modality problem, where only priors about 3D shapes are
exploited. Only recently, image-guided completion has started to receive
attention [12].

In this paper, we study how the side information offered by a single image can
be used in addition to shape priors to complete a partial point cloud. While
following the setting of ViPC [12], we extend the multimodal completion
methodology in several different ways. First, ViPC [12] is bottlenecked by the
need to estimate a coarse point cloud from the image via single-view
reconstruction techniques to fuse the information. We avoid this task by
proposing a novel architecture that performs fusion in a latent domain via
cross-attention operations on fine-grained, localized representations of the
two modalities, coupled with a flexible decoder that allows to complete areas
of varied size. Moreover, the multimodal setting is uniquely poised for
weakly-supervised learning. In fact, the input image, especially when captured
from a different vantage point, may offer a supervisory signal to guide the
completion of those areas occluded in the partial point cloud but visible in
the image. This is especially interesting for practical applications where it
could be difficult to have access to complete shapes, but significantly easier
to have images from a different viewing angle. Therefore, we propose to
augment the known 3D self-reconstruction losses with the exploitation of a
differentiable renderer to measure the fidelity of the completed point cloud
in the image space.

Our experiments show that the proposed model significantly outperforms the
state-of-the-art on both the supervised and weakly-supervised settings. In
particular, the addition of the rendering loss allows the weakly-supervised
image-guided model to outperform several supervised baselines and to be
competitive with the latest supervised models only exploiting point cloud
information.

##  2 Related work

##### Point cloud completion

3D shape completion is a long-standing problem in computer vision. Early works
devised explicit geometric descriptors or relied on shape retrieval from large
datasets [13] [14] [15] [16]. Since the advent of neural networks operating on
raw point cloud data, several models for the completion problem have been
studied [17]. They are mostly based on the encoder-decoder architecture,
pioneered by PCN [4], which was the first model that did not require any
assumption of structure or annotation information about the underlying shape.
TopNet [5] presents a hierarchical rooted tree structure that generates
structured point clouds as a collection of its subsets. AtlasNet [6] and MSN
[18], on the other hand, recreate the point cloud by assessing a set of
parametric surface elements. Convolutional-based approaches ([19, 20]) use a
voxelixed representation of shapes as input to 3D CNNs; nevertheless, this
representation introduces undesirable approximations in the shape due to
coordinate quantization effects. GRNet [21] uses techniques to represent point
cloud onto a 3D grid, so that CNNs can be exploited, without losing structural
information. Recently, VRCNet [8] has proposed a dual path architecture and a
VAE-based relation enhancement module for probabilistic modeling.
Architectures based on transformers have also been proposed. PointTr [9]
changes the transformer block to take advantage of the inductive bias of 3D
geometries, creating a geometry-aware block that models local geometry
relations. Moreover, SnowflakeNet [22] generates child points by gradually
splitting parent points by means of a Skip-Transformer that learns the
appropriate splitting modes for particular regions. As a result, the network
is able to predict highly detailed shape geometries. Finally, it is worth
mentioning that the point cloud completion literature is split between two
settings: one, as in the aforementioned works, where the partial input has the
same number of points as the completed point cloud and another where the
completed point cloud has more points than the input such as in [23, 24]. In
this paper, we will consider the former setting.

##### View-guided completion

Recently, the usage of auxiliary data to complement point cloud completion has
been introduced by ViPC [12]. The idea is to help the reconstruction objective
using side information available as a different imaging modality. In
particular, ViPC assumes that an image corresponding to a view of the same
object is also available for the completion task, and it exploits the image to
retrieve the global shape information that is lacking in the incomplete point
cloud. The image is processed by a pre-trained single-view reconstruction
model, which estimates a coarse point cloud from the image, representing the
entire shape. The key challenge in this setting is how to effectively combine
features extracted from the two modalities. Unlike ViPC, our approach
leverages direct fusion at a feature level, avoiding the need to explicitly
reconstruct a coarse point cloud from a single image, a generally hard inverse
problem in itself and full of pitfalls.

##### Self-supervised strategies for completion

All the previously mentioned approaches rely on complete ground truth as a
supervisory training signal. This may be difficult to obtain in real-word
scenarios. Self-supervised training strategies avoid the need for retrieving
such expensive ground truths. However, the amount of work on this topic is
rather limited. Wang et al. [10] use resampling that removes further points
from an already partial point cloud and mixup among shapes to train their
completion network in a self-supervised manner. Similarly, Mittal et al. [11]
also propose an inpainting procedure that leverages further partializations of
partial point clouds. To the best of our knowledge, there is no work exploring
the availability of a different modality, namely an image to provide a weak
supervisory signal to the point cloud completion task.

##  3 Proposed Methods

Figure 1: Architecture overview. Localized features from the partial point
cloud and the input image are jointly processed via cross- and self-
attentions. A decoder reconstructs the target number of points from the
feature space with attention-based upsampling. The input partial point cloud
is downsampled with farthest point sampling (FPS) and concatenated. Supervised
training only uses the point cloud reconstruction loss with respect to the
complete point cloud. Weakly-supervised training has a point cloud
reconstruction loss with respect to a less partial point cloud and a rendering
loss. $N$ is the number of points of the input point cloud; $M$ is the number
of points generated by each branch of the decoder, while $F$ represents the
feature dimension.

We address the setting in which a partial point cloud needs to be completed
with the assistance of an image of the object taken from a certain viewpoint.
Our goal is to study how to leverage this side information in the most
effective manner. To this purpose we study i) a supervised learning setting,
for which we show that the image features can be effectively fused with those
of the partial point cloud in a latent space; ii) a weakly-supervised setting
based on the idea that the image may contain clues about the missing part and
can thus serve as a supervisory signal. Fig. 1 shows an overview of the
proposed method, named XMFnet (Cross-Modal Fusion network) which will be
detailed in the next sections.

###  3.1 Architecture and Supervised Setting

At a high level, the architecture of XMFnet is composed by two modality-
specific feature extractors that capture localized features of the input point
cloud and image, summarized at a small number of points/pixels, followed by a
sequence of cross-attention and self-attention operations that progressively
merge the two feature spaces. Finally, a decoder upsamples this localized
information to estimate a predefined number of points of the missing
component. More formally, we denote the partial point cloud as
$\mathbf{X}\in\mathbb{R}^{N\times 3}$, the input view as an image
$\mathbf{I}\in\mathbb{R}^{P_{x}\times P_{y}\times 3}$ and the complete point
cloud as $\mathbf{Y}\in\mathbb{R}^{N\times 3}$. The task of our model is to
predict a complete shape $\mathbf{\hat{Y}}\in\mathbb{R}^{N\times 3}$ given
$\mathbf{X}$ and $\mathbf{I}$ as inputs. Notice we follow the conventional
setting where the partial and complete point clouds have the same number of
points, meaning that there is a resampling of the known part.

####  3.1.1 Point Cloud and Image Encoder

The point cloud encoder should extract localized features from the partial
shape $\mathbf{X}$. It is important to keep a degree of locality, i.e.,
associating features to a small number of points $N_{X}<N$ rather than a
single global embedding because the information about the missing part that
needs to be estimated by the entire model is also mostly localized. However,
it is also important to have a sufficiently large receptive field to infer
some global information about the object. For this reason, we adopt a graph-
convolutional architecture with graph pooling operations. The architecture is
a sequence of graph-convolutional layers (EdgeConv [25]) interleaved by
pooling operations (Self-Attention Graph Pooling [26]) to reduce the
cardinality of the point cloud. Pooling has the double purpose of expanding
the receptive field to also include more global information and reduce the
complexity of the subsequent cross-attention operations fusing the two
modalities.

Any network that extracts features from an image can be utilized as encoder
for view $\mathbf{I}$. The design principles follow those of the point-cloud
encoder, i.e., features localized at a subset $N_{I}<P_{x}P_{y}$ of the image
pixels obtained from a sufficiently large receptive field are produced as
output.

We will refer to the features produced by the point cloud encoder as
$\mathbf{H}_{X}\in\mathbb{R}^{N_{X}\times F_{X}}$ and by the image encoder as
$\mathbf{H}_{I}\in\mathbb{R}^{N_{I}\times F_{I}}$.

####  3.1.2 Modality Fusion

Once we have collected localized information from the two modalities, we need
to combine them effectively to capture their complementary information,
despite the obvious domain gap. The attention mechanism is particularly suited
to find correspondences between the features of a region of the point cloud
and a region of the image. The cross-attention layer in our architecture uses
the Transformer’s multihead attention mechanism [27]. The point cloud features
are projected to form the query tensor, while the image features are projected
to form the key and value tensors, and then attention mechanism aggregates the
features from different image regions according to the weights determined by
the cross-correlation between the two modalities. More formally:

$\displaystyle\mathbf{Q}_{X}=\mathbf{H}_{X}\mathbf{W}_{Q},\quad\mathbf{K}_{I}=\mathbf{H}_{I}\mathbf{W}_{K},\quad\mathbf{V}_{I}=\mathbf{H}_{I}\mathbf{W}_{V}$
(1)
$\displaystyle\mathbf{H}_{\text{fused}}=\text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{F}}\right)\mathbf{V}$
(2)

being $\mathbf{W}_{Q}\in\mathbb{R}^{F_{X}\times F}$,
$\mathbf{W}_{K},\mathbf{W}_{V}\in\mathbb{R}^{F_{I}\times F}$ the projection
weights. The fused features
$\mathbf{H}_{\text{fused}}\in\mathbb{R}^{N_{X}\times F}$ produced by the
cross-attention mechanism can be regarded as the original point cloud features
enriched by the image features.

The XMFnet architecture depicted in Fig. 1 shows a self-attention layer after
the cross-attention fusion. The goal of this operation is to have a
permutation-invariant transformation of the features with a global receptive
field so that any information from the image not properly integrated can be
rectified. Self-attention works exactly like Eq. (2) except for the fact that
$\mathbf{Q},\mathbf{K},\mathbf{V}$ are all different projections of the same
features. Furthermore, a sequence of multiple cross- and self-attention layers
can be used to more effectively integrate the information from the two
modalities via a “slow” fusion. We remark that at the end of this sequence we
use a special cross-attention layer that merges information from the end and
the beginning of the sequence allowing better flexibility in the decision of
the desired abstraction level (higher-level features cross-attend lower-level
features).

####  3.1.3 Decoder and Supervised Loss

The decoder is a crucial component of our architecture as it should take the
joint feature embedding and learn to reconstruct a complete point cloud
preserving both global and local structure. To be precise, the decoder seeks
to estimate the positions of a number of points that upper bounds the size of
the missing part, so that they can be concatenated to a version of the input
partial point cloud subsampled by means of farthest point sampling (FPS). This
mechanism is reminiscent of what is done in the setting with a variable number
of points where only the missing part is estimated [23, 24]. For example, in
our experiments, we upper bounded the size of the missing part to $50\%$ of
the total number of points, thus having $N^{\prime}$ points estimated by the
decoder concatenated to $N^{\prime}$ points from the subsampled input.
However, our method allows to be flexible and handle more incomplete inputs by
simply tuning the desired ratio of points to be estimated and points provided
from the partial input. Typically, the latent space where we perform feature
fusion is much more localized to constrain complexity and allow higher-level
features so that $N_{X}\ll N^{\prime}$, thus requiring the decoder to upsample
the feature field. We perform this operation by using a number of attention-
based operations, inspired by the work in [28], that convert features to
points in parallel with the idea that each branch specializes on the
reconstruction of a sub-region of the missing part. The structure is depicted
in Fig.1. More formally, calling $\mathbf{H}\in\mathbb{R}^{N_{X}\times F}$ the
features provided to the decoder, the output
$\hat{\mathbf{Y}}_{i}\in\mathbb{R}^{N^{\prime}/K\times 3}$ of each of the $K$
branches is computed as:

$\displaystyle\mathbf{Z}_{i}$
$\displaystyle=\text{MLP}_{i}^{\text{proj}}(\mathbf{H})\qquad i=1,\dots,K$ (3)
$\displaystyle\hat{\mathbf{Y}}_{i}$
$\displaystyle=\left(\text{softmax}(\text{MLP}_{i}^{\text{dec}}(\mathbf{Z}_{i}))^{T}\mathbf{Z}_{i}\right)\mathbf{W}_{\text{out,}i}\qquad
i=1,\dots,K$ (4)

where
MLP${}_{i}^{\text{proj}}:\mathbb{R}^{F}\rightarrow\mathbb{R}^{F^{\prime}}$,
MLP${}_{i}^{\text{dec}}:\mathbb{R}^{F^{\prime}}\rightarrow\mathbb{R}^{N^{\prime}/K}$
are multilayer perceptrons with different weights for each branch, projecting
features to $K$ subspaces and generating attention weights for the resampling
process, respectively.
$\mathbf{W}_{\text{out,}i}\in\mathbb{R}^{F^{\prime}\times 3}$ is projection
matrix to 3D space. Finally, the completed point cloud is generated by
concatenation of the outputs of all decoder branches and the partial input
subsampled by FPS as:

$\displaystyle\hat{\mathbf{Y}}=\left[\hat{\mathbf{Y}}_{1},\hat{\mathbf{Y}}_{2},\dots,\hat{\mathbf{Y}}_{K},\text{FPS}(\mathbf{X})\right].$
(5)

Supervised training is performed using the L1 Chamfer Distance (CD) between
the generated shapes and the ground truth shapes, defined as follows:

$\displaystyle\mathcal{L_{\text{CD}}}(\mathbf{Y},\hat{\mathbf{Y}})=\frac{1}{2N}\sum_{\mathbf{y}\in\mathbf{Y}}\min_{\hat{\mathbf{y}}\in\hat{\mathbf{Y}}}\|\mathbf{y}-\hat{\mathbf{y}}\|+\frac{1}{2N}\sum_{\hat{\mathbf{y}}\in\hat{\mathbf{Y}}}\min_{\mathbf{y}\in\mathbf{Y}}\|\hat{\mathbf{y}}-\mathbf{y}\|.$
(6)

###  3.2 Weakly-supervised Setting

The multimodal completion problem addressed in this paper is uniquely poised
for weakly-supervised learning. In fact, the image available as input may
contain complementary information with respect to the point cloud and,
crucially, cues about the missing part. This is especially true if the image
is collected from a different viewpoint or at a different time with respect to
the point cloud, resulting in different kinds of occlusions.

Figure 2: Weakly-supervised training.

Existing architectures for self-supervised completion [10] [11] rely solely on
point cloud supervision, due to their unimodal nature. The key insight of the
proposed method (Fig. 2) is to supplement completion losses on points with a
loss measuring a reprojection error in the image space. In particular, we
measure whether the reconstructed point cloud produced by the architecture
described in Sec. 3.1 leads to an image similar to the input one, when
captured from the correct viewpoint.

In order to measure this information and use it in the training process, we
include a differentiable rendering module based on alpha compositing [29]
which generates a rasterized version of the object, employing provided camera
parameters. In order to ensure consistency with the input image, intrinsic and
extrinsic camera parameters may be estimated with a number of well-known
methods [30] [31] [32]. In order to minimize the domain shift between the
input image and the result of the rendering process, we work on silhouettes,
i.e., binary masks of objects. The differentiable renderer produces a soft
silhouette with continuous values, while the input image is directly
binarized. Inevitable inaccuracies in the camera parameters and rendering
process will typically yield unreliable borders of the silhouette. Therefore,
we also compute a border mask with a simple edge detector (Laplacian of
Gaussian) and discount the loss function by a factor $\varepsilon<1$ for the
pixels in the mask. In summary, our rendering loss is defined as:

$\displaystyle\mathcal{L}_{\text{render}}=\left\|\mathbf{M}\odot\left[R(\hat{\mathbf{Y}})-S(\mathbf{I})\right]\right\|_{1},\qquad\mathbf{M}_{i,j}=\begin{cases}\varepsilon\quad\text{if
}(i,j)\in\text{edge}\\\ 1\quad\text{otherwise}\end{cases}$ (7)

being $R$ the differentiable silhouette renderer and $S$ the silhouette
binarizer. We remark that [33] proposes to use rendering to improve
performance in point cloud completion, aiding the learning process through an
image domain supervision. However, their approach is supervised and is based
on rendering the ground truth and generated point cloud in depth-maps with
different view-points.

In addition to the rendering loss, our weakly-supervised framework also adopts
self-supervision in the point cloud domain. In particular, we use a
combination of the resampling and mixup approaches proposed in [11, 10].
Resampling consists in removing random portions of the original partial input
point cloud, yielding even more partial shapes. As a result, the original
partial input is employed as a pseudo-ground truth. Mixup combines a pair of
partial shapes weighed according to Beta distribution in an attempt at
increasing the complexity of the shapes processed by the network. Differently
from [10], we also have images associated with a partial shape in our setting.
Hence, we also mix the images in such a way that the mixup technique is
carried out symmetrically for the two modalities.

We remark that the rendering loss is comparatively weaker than the point cloud
loss and, by itself, has a number of ambiguities due to the lack of depth
information and the use of silhouettes. For this reason, it is important to
combine it with the point cloud loss. We found that the density-aware Chamfer
Distance (DCD) [34], a version of CD that is more sensitive to non-uniform
point distributions is superior in this weakly-supervised scenario to
regularize the ambiguities of the rendering loss. Our overall weakly-
supervised training procedure is therefore as follows. We alternate between a
step that optimizes the point cloud loss consisting in a weighted CD:

$\displaystyle\mathcal{L}_{\text{PC}}=\frac{1-\beta}{2N}\sum_{\mathbf{y}\in\mathbf{Y}}\min_{\hat{\mathbf{y}}\in\hat{\mathbf{Y}}}\|\mathbf{y}-\hat{\mathbf{y}}\|+\frac{\beta}{2N}\sum_{\hat{\mathbf{y}}\in\hat{\mathbf{Y}}}\min_{\mathbf{y}\in\mathbf{Y}}\|\hat{\mathbf{y}}-\mathbf{y}\|,$
(8)

and a step optimizing a combination of DCD and rendering loss:

$\displaystyle\mathcal{L}_{\text{I}}=\left[\frac{1}{2N}\sum_{\mathbf{y}\in\mathbf{Y}}\left(1-\frac{1}{N}e^{-\alpha\|\mathbf{y}-\mathbf{w}\|_{2}}\right)+\frac{1}{2N}\sum_{\hat{\mathbf{y}}\in\hat{\mathbf{Y}}}\left(1-\frac{1}{N}e^{-\alpha\|\hat{\mathbf{y}}-\mathbf{z}\|_{2}}\right)\right]+\lambda\mathcal{L}_{\text{render}}$
(9)

where
$\mathbf{w}=\arg\min_{\hat{\mathbf{y}}\in\hat{\mathbf{Y}}}\|\mathbf{y}-\hat{\mathbf{y}}\|_{2}$
and
$\mathbf{z}=\arg\min_{\mathbf{y}\in\mathbf{Y}}\|\mathbf{y}-\hat{\mathbf{y}}\|_{2}$.
Notice that $\mathcal{L}_{\text{PC}}$ and the DCD part of
$\mathcal{L}_{\text{I}}$ use resampling and mixup for point clouds, while
$\mathcal{L}_{\text{render}}$ does not ($\mathcal{L}_{\text{I}}$ is computed
with full minibatch for the point-cloud part and half minibatch with the
original partials for rendering).

##  4 Experimental results

###  4.1 Experimental Settings and Implementation Details

All the experiments are conducted on the ShapeNet-ViPC[12] dataset. The
dataset contains 38,328 objects from 13 categories; for each object it
comprises 24 partial point clouds with occlusions generated under 24
viewpoints, using the same settings as ShapeNetRendering [35]. The input and
ground truth point clouds contains $N=2048$ points each. Each 3D shape is
rotated to the pose corresponding to a certain view point after being
normalized within the bounding sphere with radius of 1. Images are generated
from the 24 view points of ShapeNetRendering and have a resolution of
$224\times 224$ pixels. For all the experiments in this paper, we employ the
same selection used in [12]: we used 31,650 objects from eight categories,
with $80\%$ of them for training and $20\%$ for testing.

The partial point cloud is downsampled by farthest point sampling to
$N^{\prime}=1024$ points and concatenated to the output of the decoder that
produces $N^{\prime}=1024$ points leading to a completed point cloud with 2048
points. The decoder has $K=8$ branches, each of them producing $M=128$ points.
The point cloud encoder employs EdgeConv and SAGPooling layers; the EdgeConv
layers selects $k=20$ nearest neighbors, while the two pooling layers use
$k=16$ and $k=6$ nearest neighbors, respectively. The original point cloud is
overall downsampled by a factor of 16, resulting in $N_{X}=128$ points with
$F_{X}=256$ features. The image encoder is built with a ResNet18 [36] as
backbone, it extracts $N_{I}=14\times 14=196$ pixels with $F_{I}=256$
features. The multihead attention has 4 attention heads, with embedding size
$F=256$. In the $\mathcal{L}_{\text{I}}$ loss we use $\lambda=0.15$. The mask
factor for the edge detector has been set to $\varepsilon=0.4$.

The differentiable renderer has been implemented with PyTorch3D[37]. The
rendered silhouettes $H\times W$ has size $224\times 224$ that is the same
size of input views in our experiments. We adopt radius $\rho=0.025$ in point
rasterization. The proposed framework is implemented in PyTorch and trained on
an Nvidia V100 GPU. Class-specific training is performed for all models, using
the Adam optimizer [38] for roughly 200 epochs with a batch size of 128. The
learning rate is initialized to 0.001 and reduced by a factor of 10 at epoch
25 and 125.

Table 1: Mean Chamfer Distance per point ($\times 10^{-3}$). ShapeNet-ViPC dataset, supervised. Methods | Avg | Airplane | Cabinet | Car | Chair | Lamp | Sofa | Table | Watercraft
---|---|---|---|---|---|---|---|---|---
AtlasNet [6] | 6.062 | 5.032 | 6.414 | 4.868 | 8.161 | 7.182 | 6.023 | 6.561 | 4.261
FoldingNet [39] | 6.271 | 5.242 | 6.958 | 5.307 | 8.823 | 6.504 | 6.368 | 7.080 | 3.882
PCN [4] | 5.619 | 4.246 | 6.409 | 4.840 | 7.441 | 6.331 | 5.668 | 6.508 | 3.510
TopNet [5] | 4.976 | 3.710 | 5.629 | 4.530 | 6.391 | 5.547 | 5.281 | 5.381 | 3.350
ECG [7] | 4.957 | 2.952 | 6.721 | 5.243 | 5.867 | 4.602 | 6.813 | 4.332 | 3.127
VRC-Net [8] | 4.598 | 2.813 | 6.108 | 4.932 | 5.342 | 4.103 | 6.614 | 3.953 | 2.925
ViPC [12] | 3.308 | 1.760 | 4.558 | 3.183 | 2.476 | 2.867 | 4.481 | 4.990 | 2.197
XMFnet | 1.443 | 0.572 | 1.980 | 1.754 | 1.403 | 1.810 | 1.702 | 1.386 | 0.945
Table 2: Mean F-Score @ 0.001. ShapeNet-ViPC dataset, supervised Methods | Avg | Airplane | Cabinet | Car | Chair | Lamp | Sofa | Table | Watercraft
---|---|---|---|---|---|---|---|---|---
AtlasNet [6] | 0.410 | 0.509 | 0.304 | 0.379 | 0.326 | 0.426 | 0.318 | 0469 | 0.551
FoldingNet [39] | 0.331 | 0.432 | 0.237 | 0.300 | 0.204 | 0.360 | 0.249 | 0.351 | 0.518
PCN [4] | 0.407 | 0.578 | 0.270 | 0.331 | 0.323 | 0.456 | 0.293 | 0.431 | 0.577
TopNet [5] | 0.467 | 0.593 | 0.358 | 0.405 | 0.388 | 0.491 | 0.361 | 0.528 | 0.615
ECG [7] | 0.704 | 0.880 | 0.542 | 0.713 | 0.671 | 0.689 | 0.534 | 0.792 | 0.810
VRC-Net [8] | 0.764 | 0.902 | 0.621 | 0.753 | 0.722 | 0.823 | 0.654 | 0.810 | 0.832
ViPC [12] | 0.591 | 0.803 | 0.451 | 0.512 | 0.529 | 0.706 | 0.434 | 0.594 | 0.730
XMFnet | 0.796 | 0.961 | 0.662 | 0.691 | 0.809 | 0.792 | 0.723 | 0.830 | 0.901

###  4.2 Main Results

####  4.2.1 Supervised Learning

We first compare XMFnet against several baselines for the point cloud
completion under supervised learning. Since the new multimodal setting with an
auxiliary image has been introduced only recently, ViPC [12] represents the
only method fully comparable to ours. However, we also report the results of a
number of state-of-the-art architectures for completion with only point cloud
input, when retrained on the ViPC dataset. AtlasNet[6] reconstructs a point
cloud by estimating parametric surface elements. FoldingNet[39] is a 2-D grid
based auto-encoder. PCN[4] is an encoder-decoder framework that reconstructs
the point cloud in a coarse-to-fine manner. TopNet[5] has a rooted tree
structure in the decoder. ECG[7] is an edge-aware completion method based on
Graph Convolutions. VRC-Net[8] is the most recent method adopting a VAE-based
model with a dual path architecture and probabilistic modeling. In line with
the previous evaluation protocols, we use CD and F-score [40] as metrics for
the reconstruction quality. Before evaluating the CD, we normalize the output
of the models to fit into the unit sphere. Table 1 and Table 2 report the
experimental results and show XMFnet outperforming the other techniques by a
significant margin. While part of this gain with respect to state-of-the-art
models for point cloud completion can be attributed to the use of the input
image, it is worth noting that we also report significant improvements over
the multimodal ViPC. This highlights the sub-optimality of performing modality
fusion by resorting to estimating a coarse point cloud from the single input
image, as in ViPC, rather than working in a latent feature space. Qualitative
comparisons111Visualizations for ViPC [12] have been kindly provided by the
original authors. are shown in Fig. 3. Our method is capable of producing
cleaner completions than the other baselines, with fewer outliers and a more
uniform point distribution.

Partial | PCN | ECG | VRCnet | ViPC | XMFnet | GT
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
Figure 3: Qualitative comparison of completed point clouds for different
classes.

####  4.2.2 Weakly-supervised

Table 3: Mean Chamfer Distance per point ($\times 10^{-3}$). ShapeNet-ViPC
dataset.

Methods | Airplane | Lamp | Watercraft
---|---|---|---
AtlasNet [6] | 5.032 | 7.182 | 4.261
FoldingNet [39] | 6.504 | 6.368 | 3.882
PCN [4] | 4.246 | 6.331 | 3.510
TopNet [5] | 3.710 | 5.547 | 3.350
ECG [7] | 2.952 | 4.602 | 3.127
VRC-Net [8] | 2.813 | 4.103 | 2.925
VRC-Net(self-sup.) | 4.315 | 8.023 | 7.259
ViPC [12] | 1.760 | 2.867 | 2.197
XMFnet (sup.) | 0.572 | 1.810 | 0.945
XMFnet (weakly-sup.) | 2.426 | 6.269 | 3.423

Table 4: Mean F-Score @ 0.001. ShapeNet-ViPC dataset.

Methods | Airplane | Lamp | Watercraft
---|---|---|---
AtlasNet [6] | 0.509 | 0.426 | 0.551
FoldingNet [39] | 0.432 | 0.360 | 0.518
PCN [4] | 0.578 | 0.456 | 0.577
TopNet [5] | 0.593 | 0.491 | 0.615
ECG [7] | 0.880 | 0.689 | 0.810
VRC-Net [8] | 0.902 | 0.823 | 0.832
VRC-Net(self-sup.) | 0.689 | 0.710 | 0.673
ViPC [12] | 0.803 | 0.706 | 0.730
XMFnet (sup.) | 0.961 | 0.792 | 0.901
XMFnet (weakly-sup.) | 0.742 | 0.542 | 0.704

Table 5: Unimodal vs. Multimodal completion (supervised)

Method | Avg | Airplane | Cabinet | Lamp
---|---|---|---|---
Unimodal | 1.570 | 0.626 | 2.114 | 1.980
Multimodal | 1.470 | 0.572 | 1.973 | 1.810
$\drsh$ best view | 1.223 | 0.545 | 1.426 | 1.697
$\drsh$ worst view | 1.819 | 0.722 | 2.621 | 2.115

Table 6: Ablation study for the Weakly-Supervised method (airplane)

Resampling | Mixup | Rendering | CD($10^{-3}$)
---|---|---|---
✓ | ✗ | ✗ | 4.568
✓ | ✓ | ✗ | 4.239
✓ | ✓ | ✓ | 2.426

Table 7: Ablation study for the Weakly-Supervised method - DCD (cabinet)

DCD | CD($10^{-3}$)
---|---
✗ | 3.012
✓ | 2.426

We remark that we are the first to propose a weakly-supervised training
strategy for multimodal completion, so the setting in which supervisory
information can be gathered from an input image is unexplored. For this
reason, we compare to a number of unimodal and multimodal supervised methods
as well as a self-supervised version using resampling and mixup of a unimodal
state-of-the-art model222We remark the difficulty in reproducing several
published methods in the self-supervised setting. The code for [11, 10] was
not available. The code to reproduce ViPC [12] is also incomplete so we cannot
retrain the most sensible self-supervised baseline of ViPC + Resampling +
Mixup.. The results are reported in Table 7 for the CD and Table 7 for the
F-Score. We notice that our weakly-supervised method outperforms a number of
supervised baselines and it is close to the performance of the most recent
unimodal supervised models.

###  4.3 Ablation Studies

In order to verify the effectiveness of the proposed design, we study the
impact of the auxiliary image input on the completion performance. To ensure a
comparison as fair as possible, the version of XMFnet that uses only point
cloud information has the image encoder removed and the cross-attention blocks
replaced with self-attention ones.

The unimodal architecture is then trained with the same settings as the
multimodal one, and the results are reported in Table 7. The results show that
the addition of the image input provides a significant improvement in
performance. Notice that besides the result averaged over all the possible 24
views, we also report the performance with the worst view and the best view.
Indeed, we are interested in investigating how the viewpoint of the image
affects completion.

Figure 4: Impact of image contribution as function of point of view, sorted by
reconstruction CD (from worst to best) averaged over cabinet category,
supervised setting.

Fig. 4 reports the average CD for different views, ordered from worst to best,
for the cabinet category. It is clear that some views provide complementary
information due to their different vantage point and allow to substantially
improve over the average result. A small number of “bad” views leads to
results comparable to the unimodal case.

Furthermore, it is interesting to study the impact of our novel rendering loss
in the weakly-supervised setting. We noticed that it allows the training
process to have a faster and smoother convergence and that the overall
completion performance is increased from both a qualitative and quantitative
point of view. A qualitative comparison between the weakly-supervised strategy
with and without the rendering module can be visualized in Fig. 5 and
quantitative results are reported in Table 7. The mixup loss provides only a
small improvement from the perspective of the CD. However, it substantially
improves the completed shape from a qualitative point of view, helping the
network generate more complete shapes. Moreover, the computational overhead
due to creating the mixed input shape is very small, so we decided to keep the
method as it offered a very favorable cost-performance trade-off. We also
include the ablation for the DCD component of the weakly-supervised loss, we
found it helpful from both a qualitative and quantitative point of view. Table
7 reports the impact of the DCD on the cabinet category, while Fig. 6 provides
a qualitative example, where the shape completed with the DCD presents a more
uniform distribution of points and a better overall quality.

Partial | Without Rendering | With Rendering | GT
---|---|---|---
|  |  |
Figure 5: Qualitative visualization of the effect of the proposed weakly-supervised rendering loss. The sample without rendering has CD $=7.812$, the one with rendering has CD$=3.743$.  Partial | Normal CD | DCD | GT
---|---|---|---
|  |  |
Figure 6: Qualitative visualization of the effect of the DCD for the weakly-
supervised setting.

##  5 Conclusions

In this paper, we explored the topic of point cloud completion guided by an
auxiliary image, discovering that effective fusion can be achieved in a latent
space via cross-attention. Our method achieves state-of-the-art results on the
ShapeNetViPC-Dataset. Moreover, we showed how this setting lends itself to
weakly-supervised learning where the image can be used for supervision via a
differentiable rendering approach. The major limitation of our work is the
lack of study of a real-world scenario for the proposed framework. In future
work, we will focus on extending the work to real acquisitions, thus dealing
with complex effects like acquisition noise, background or additional
occlusions in the auxiliary images, and many more. This paper has provided a
proof of concept that effective multimodal completion is possible but a more
in-depth study of such issues on real scenes is needed, along with suitable
improvements to our design towards increased robustness.

## Acknowledgments and Disclosure of Funding

Computational resources were provided by HPC@POLITO, a project of Academic
Computing within the Department of Control and Computer Engineering at the
Politecnico di Torino (http://www.hpc.polito.it).

## References

  * [1] J. Hou, A. Dai, and M. Nießner, “3D-SIS: 3D semantic instance segmentation of RGB-D scans,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4421–4430.
  * [2] J. Varley, C. DeChant, A. Richardson, J. Ruales, and P. Allen, “Shape completion enabled robotic grasping,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 2442–2447.
  * [3] Y. Li, L. Ma, Z. Zhong, F. Liu, M. A. Chapman, D. Cao, and J. Li, “Deep learning for lidar point clouds in autonomous driving: A review,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 8, pp. 3412–3432, 2020.
  * [4] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert, “PCN: Point completion network,” in _2018 International Conference on 3D Vision (3DV)_. IEEE, 2018, pp. 728–737.
  * [5] L. P. Tchapmi, V. Kosaraju, H. Rezatofighi, I. Reid, and S. Savarese, “Topnet: Structural point cloud decoder,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 383–392.
  * [6] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry, “A papier-mâché approach to learning 3d surface generation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 216–224.
  * [7] L. Pan, “ECG: Edge-aware Point Cloud Completion with Graph Convolution,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4392–4398, 2020\.
  * [8] L. Pan, X. Chen, Z. Cai, J. Zhang, H. Zhao, S. Yi, and Z. Liu, “Variational relational point completion network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8524–8533.
  * [9] X. Yu, Y. Rao, Z. Wang, Z. Liu, J. Lu, and J. Zhou, “PoinTr: Diverse point cloud completion with geometry-aware transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 12 498–12 507.
  * [10] X. Wang, M. H. Ang, and G. Lee, “Cascaded refinement network for point cloud completion with self-supervision,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [11] H. Mittal, B. Okorn, A. Jangid, and D. Held, “Self-Supervised Point Cloud Completion via Inpainting,” in _British Machine Vision Conference_ , 2021\.
  * [12] X. Zhang, Y. Feng, S. Li, C. Zou, H. Wan, X. Zhao, Y. Guo, and Y. Gao, “View-guided point cloud completion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 890–15 899.
  * [13] M. Sung, V. G. Kim, R. Angst, and L. Guibas, “Data-driven structural priors for shape completion,” _ACM Transactions on Graphics (TOG)_ , vol. 34, no. 6, pp. 1–11, 2015.
  * [14] N. J. Mitra, L. J. Guibas, and M. Pauly, “Partial and approximate symmetry detection for 3d geometry,” _ACM Transactions on Graphics (TOG)_ , vol. 25, no. 3, pp. 560–568, 2006.
  * [15] M. Pauly, N. J. Mitra, J. Wallner, H. Pottmann, and L. J. Guibas, “Discovering structural regularity in 3d geometry,” _ACM Transactions on Graphics (TOG)_ , vol. 27, no. 3, pp. 1–11, 2008.
  * [16] R. Schnabel, P. Degener, and R. Klein, “Completion and reconstruction with primitive shapes,” in _Computer Graphics Forum_ , vol. 28, no. 2. Wiley Online Library, 2009, pp. 503–512.
  * [17] B. Fei, W. Yang, W. Chen, Z. Li, Y. Li, T. Ma, X. Hu, and L. Ma, “Comprehensive review of deep learning-based 3d point clouds completion processing and analysis,” _arXiv preprint arXiv:2203.03311_ , 2022.
  * [18] M. Liu, L. Sheng, S. Yang, J. Shao, and S.-M. Hu, “Morphing and sampling network for dense point cloud completion,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 07, 2020, pp. 11 596–11 603.
  * [19] A. Dai, C. Ruizhongtai Qi, and M. Nießner, “Shape completion using 3D-encoder-predictor CNNs and shape synthesis,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5868–5877.
  * [20] X. Han, Z. Li, H. Huang, E. Kalogerakis, and Y. Yu, “High-resolution shape completion using deep neural networks for global structure and local geometry inference,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 85–93.
  * [21] H. Xie, H. Yao, S. Zhou, J. Mao, S. Zhang, and W. Sun, “GRNet: Gridding residual network for dense point cloud completion,” in _European Conference on Computer Vision_. Springer, 2020, pp. 365–381.
  * [22] P. Xiang, X. Wen, Y.-S. Liu, Y.-P. Cao, P. Wan, W. Zheng, and Z. Han, “Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5499–5509.
  * [23] Z. Huang, Y. Yu, J. Xu, F. Ni, and X. Le, “Pf-net: Point fractal network for 3d point cloud completion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7662–7670.
  * [24] A. Alliegro, D. Valsesia, G. Fracastoro, E. Magli, and T. Tommasi, “Denoise and contrast for category agnostic shape completion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4629–4638.
  * [25] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph cnn for learning on point clouds,” _ACM Transactions on Graphics (TOG)_ , 2019.
  * [26] J. Lee, I. Lee, and J. Kang, “Self-attention graph pooling,” in _International conference on machine learning_. PMLR, 2019, pp. 3734–3743.
  * [27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [28] K. Zhang, X. Yang, Y. Wu, and C. Jin, “Attention-based transformation from latent features to point clouds,” _arXiv preprint arXiv:2112.05324_ , 2021\.
  * [29] O. Wiles, G. Gkioxari, R. Szeliski, and J. Johnson, “Synsin: End-to-end view synthesis from a single image,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7467–7477.
  * [30] E. Insafutdinov and A. Dosovitskiy, “Unsupervised learning of shape and pose with differentiable point clouds,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * [31] A. Grabner, Y. Wang, P. Zhang, P. Guo, T. Xiao, P. Vajda, P. M. Roth, and V. Lepetit, “Geometric correspondence fields: Learned differentiable rendering for 3d pose refinement in the wild,” in _European Conference on Computer Vision_. Springer, 2020, pp. 102–119.
  * [32] A. Palazzi, L. Bergamini, S. Calderara, and R. Cucchiara, “End-to-end 6-dof object pose estimation through differentiable rasterization,” in _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018, pp. 0–0.
  * [33] C. Xie, C. Wang, B. Zhang, H. Yang, D. Chen, and F. Wen, “Style-based point generator with adversarial rendering for point cloud completion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4619–4628.
  * [34] T. Wu, L. Pan, J. Zhang, T. Wang, Z. Liu, and D. Lin, “Balanced Chamfer Distance as a Comprehensive Metric for Point Cloud Completion,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [35] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “ShapeNet: An Information-Rich 3D Model Repository,” Stanford University — Princeton University — Toyota Technological Institute at Chicago, Tech. Rep. arXiv:1512.03012 [cs.GR], 2015.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [37] N. Ravi, J. Reizenstein, D. Novotný, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari, “Accelerating 3d deep learning with pytorch3d,” _SIGGRAPH Asia 2020 Courses_ , 2020.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Internation Conference on Learning Representations_ , 2015.
  * [39] Y. Yang, C. Feng, Y. Shen, and D. Tian, “FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 206–215.
  * [40] A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun, “Tanks and temples: Benchmarking large-scale scene reconstruction,” _ACM Transactions on Graphics (TOG)_ , vol. 36, pp. 1 – 13, 2017.


# News and views regarding PSR J1757–1854, a highly-relativistic binary pulsar

A. D. Cameron∗ and M. Bailes  ARC Centre of Excellence for Gravitational Wave
Discovery (OzGrav)
Centre for Astrophysics and Supercomputing, Swinburne University of
Technology,
PO Box 218, Hawthorn VIC 3122, Australia
∗E-mail<EMAIL_ADDRESS>V. Balakrishnan  D. J. Champion  P. C. C.
Freire  M. Kramer and N. Wex  Max-Planck-Institut für Radioastronomie (MPIfR)
Auf dem Hügel 69, D-53121 Bonn, Germany  S. Johnston  Australia Telescope
National Facility, CSIRO Space and Astronomy
PO Box 76, Epping NSW 1710, Australia  A. G. Lyne and B. W. Stappers  Jodrell
Bank Center for Astrophysics, University of Manchester
Alan Turing Building, Oxford Road, Manchester M13 9PL, UK  M. A. McLaughlin
N. Pol and H. Wahl  Department of Physics and Astronomy, West Virginia
University
PO Box 6315, Morgantown, WV 26506, USA  C. Ng  Dunlap Institute for Astronomy
& Astrophysics, University of Toronto
50 St. George Street, Toronto, ON M5S 3H4, Canada  A. Possenti  Istituto
Nazionale di Astrofisica (INAF), Osservatorio Astronomico di Cagliari
Via della Scienza 5, 09047 Selargius (CA), Italy  A. Ridolfi  Istituto
Nazionale di Astrofisica (INAF), Osservatorio Astronomico di Cagliari
Via della Scienza 5, 09047 Selargius (CA), Italy  Max-Planck-Institut für
Radioastronomie (MPIfR)
Auf dem Hügel 69, D-53121 Bonn, Germany

###### Abstract

We provide an update on the ongoing monitoring and study of the highly-
relativistic double neutron star binary system PSR J1757–1854, a 21.5-ms
pulsar in a highly eccentric, 4.4-hour orbit. The extreme nature of this
pulsar’s orbit allows it to probe a parameter space largely unexplored by
other relativistic binary pulsars. For example, it displays one of the highest
gravitational wave (GW) luminosities of any known binary pulsar, as well as
the highest rate of orbital decay due to GW damping. PSR J1757–1854 is also
notable in that it is an excellent candidate for exploring new tests of
General Relativity and other gravitational theories, with possible
measurements of both Lense-Thirring precession and relativistic orbital
deformation (through the post-Keplerian parameter $\delta_{\theta}$)
anticipated within the next 3–5 years.

Here we present a summary of the latest interim results from the ongoing
monitoring of this pulsar as part of an international, multi-telescope
campaign. This includes an update of the pulsar’s long-term timing and post-
Keplerian parameters, new constraints on the pulsar’s proper motion and
corresponding Shklovskii kinematic correction, and new limits on the pulsar’s
geodetic precession as determined by monitoring for secular changes in the
pulse profile. We also highlight prospects for future work, including an
updated timeline on new relativistic tests following the introduction of
MeerKAT observations.

###### keywords:

gravitation; binaries: close; pulsars: individual: PSR J1757–1854

##  1 Introduction

For many decades, relativistic binary pulsars have been one of the key tools
for studying different theories of gravity in the strong field regime, the
current paradigm being Einstein’s theory of General Relativity (GR).
Discovered in 2016 as part of the HTRU-S Galactic Plane pulsar survey[1, 2],
the 21.5-ms pulsar PSR J1757–1854 ranks as one of the most extreme examples of
this class. Its high eccentricity ($e\simeq 0.606$) and compact orbit
(measured via the projected semi-major axis, $x\simeq 2.24\,\text{lt-s}$)
around a neutron star companion combine some of the best properties of other
notable relativistic binary pulsars, including the high eccentricity of PSR
B1913+16[3] (the 7.75-hr orbital period ‘Hulse-Taylor pulsar’, with $e\simeq
0.617$) and the compact orbit of PSR J0737–3039A/B[4, 5] (the 2.45-hr orbital
period‘Double Pulsar’, with $x\simeq 1.42\,\text{lt-s}$). The extreme nature
of PSR J1757–1854 is further demonstrated by multiple records set by the
pulsar, including the highest acceleration ($\sim 700\,\text{m\,s}^{-2}$) and
highest relative velocity ($\sim 1000\,\text{km\,s}^{-1}$) seen in a binary
pulsar, as well as the highest rate of orbital decay due to gravitational wave
damping and one of the shortest predicted merger times of any Galactic double
neutron star system ($\sim 76\,\text{Myr}$). Together, these properties make
PSR J1757–1854 a highly promising candidate for providing new insights into
theories of gravity.

Here, we describe some of the ongoing aspects of the work involving this
pulsar, in advance of a more detailed peer-reviewed publication anticipated by
mid-2022.

##  2 Current timing and post-Keplerian tests

Since its discovery in 2016, PSR J1757–1854 has been observed by multiple
radio telescopes as part of an international collaboration, including the
Parkes111Also known by its indigenous Wiradjuri name ‘Murriyang’. (Australia),
Effelsberg (Germany), Lovell (UK), Green Bank (GBT; USA) and MeerKAT (South
Africa) telescopes. This data was analysed using a standard pulsar timing
technique[6], wherein summed, average pulse profiles are cross-correlated
against high signal-to-noise (S/N) template profiles to produce a dataset of
high-precision pulse ‘times of arrival’ (TOAs). Each TOA represents the mean
arrival time of an integrated pulse profile from the pulsar. These are then
iteratively fit using a timing ephemeris in order to develop a high-precision
model of the pulsar’s behaviour.

Currently, the GBT accounts for the overwhelming majority of PSR J1757–1854’s
TOAs, having produced over 22,000 TOAs since it began observing in mid-2016.
Data from the GBT is recorded in a full-Stokes coherent search mode at both L
and S-bands (800 MHz bandwidth centered at 1500 and 2000 MHz respectively). We
currently record one 4.4-hr orbit at each frequency every two months.

Latest timing parameters for PSR J1757–1854, based on data from the GBT and
Parkes between MJD 57405–59363 and employing the DDH[7] binary model. Values
in parentheses give the 1-$\sigma$ uncertainties on the final digit.
Astrometric & spin parameters Right ascension, $\alpha$ (J2000)
17:57:03.78412(2) Declination, $\delta$ (J2000) $-$18:54:03.359(4) Spin
period, $P$ (ms) 21.4972318900292(6) Spin period derivative, $\dot{P}$
$2.627335(9)\times 10^{-18}$ Dispersion measure (pc $\text{cm}^{-3}$)
378.203(2) Proper motion in RA, $\mu_{\alpha}$ (mas $\text{yr}^{-1}$)
$-$4.36(12) Proper motion in DEC, $\mu_{\delta}$ (mas $\text{yr}^{-1}$)
$-$0.8(14) Period and position epoch (MJD) 57701 Orbital parameters Orbital
period, $P_{\text{b}}$ (d) 0.183537835854(7) Eccentricity, $e$ 0.6058171(3)
Projected semimajor axis, $x$ (lt-s) 2.2378060(15) Epoch of periastron,
$T_{0}$ (MJD) 57700.92599421(3) Longitude of periastron, $\omega$ (∘)
279.34090(9) Post-Keplerian parameters Rate of periastron advance,
$\dot{\omega}$ (${}^{\circ}\,\text{yr}^{-1}$) 10.36498(3) Einstein delay,
$\gamma$ (ms) 3.5891(16) Orbital period derivative, $\dot{P}_{\text{b}}$
$-$5.294(6)$\times 10^{-12}$ Orthometric amplitude, $h_{3}$ ($\mu\text{s}$)
5.08(18) Orthometric ratio, $\varsigma$ 0.899(10) Mass measurements (based on
$\dot{\omega}$ and $\gamma$) Pulsar mass, $m_{\text{p}}$ ($\text{M}_{\odot}$)
1.3406(5) Companion mass, $m_{\text{c}}$ ($\text{M}_{\odot}$) 1.3922(5)
Inclination angle, $i$ (∘) 85.0(2)

The most recent timing ephemeris for PSR J1757–1854 is given in Table 2. This
includes five theory-independent post-Keplerian (PK) relativistic parameters,
from which we can derive three independent tests of gravity. A mass-mass
diagram showing the mass constraints imposed by each PK parameter under GR is
shown in Figure 1. By fixing a solution for the masses of the pulsar and the
companion neutron star using the intersection of $\dot{\omega}$ and $\gamma$
(the most precisely measured PK parameters), the further intersection of each
remaining PK parameter provides one additional test of consistency. Both the
orthometric amplitude and ratio parameters of the Shapiro delay[7] ($h_{3}$
and $\varsigma$) are consistent to within 3.7% and 1.9% of their GR-predicted
values respectively, and are approximately consistent with the
$\dot{\omega}-\gamma$ intersection. However, although the observed value of
the orbital period derivative $\dot{P}_{\text{b}}$ is also consistent with GR
to within at least 1%, its location on the mass-mass diagram is inconsistent
with the constraints set by $\dot{\omega}$ and $\gamma$ under the assumption
of GR being correct. This is further discussed in Section 2.1.

Figure 1: Mass-mass diagram for PSR J1757–1854. Each coloured triplet of lines
shows the constraints (with 1-$\sigma$ uncertainty) placed by each of the
measured PK parameters in Table 2 on the mass of the pulsar and the companion
neutron star, in this case according to GR. The measured uncertainty of
$\dot{\omega}$ is so small that it cannot be seen at this scale. The grey
region in the bottom right is excluded due to orbital geometry.

###  2.1 Proper motion and the radiative test of gravity

A recent development in the timing of PSR J1757–1854 has been the first
constrained measurement of the pulsar’s proper motion, which is also listed in
Table 2. The component in right ascension ($\mu_{\alpha}$) is detectable at
approximately 36 $\sigma$, while the component in declination ($\mu_{\delta}$)
remains poorly constrained, although its contribution to the total proper
motion remains small given its value relative to $\mu_{\alpha}$. Together,
these values indicate a total proper motion of

$\mu_{\text{T}}=\sqrt{\mu_{\alpha}^{2}+\mu_{\delta}^{2}}=4.4\left(3\right)\,\text{mas\,yr}^{-1}.$
(1)

This allows us to attempt to quantify the contributions contaminating the
observed value of the orbital period derivative, $\dot{P}_{\text{b,obs}}$. We
define the excess contribution as

$\dot{P}_{\text{b,exs}}=\dot{P}_{\text{b,obs}}-\left(\dot{P}_{\text{b,GR}}+\dot{P}_{\text{b,Gal}}+\dot{P}_{\text{b,Shk}}\right),$
(2)

where $\dot{P}_{\text{b,GR}}$ is the expected intrinsic contribution from GR
(here calculated using the mass values derived from the observed
$\dot{\omega}$ and $\gamma$ as given in Table 2); $\dot{P}_{\text{b,Gal}}$ is
the contribution from the acceleration of the pulsar within the Galactic
potential[8] (dependent on position, distance and the chosen model of the
Galactic potential); and $\dot{P}_{\text{b,Shk}}$ is the contribution from the
Shklovskii effect[9] (dependent on distance and proper motion).

Figure 2: The excess contribution to the orbital period derivative
$\dot{P}_{\text{b,exs}}$ as calculated by Equation 2, plotted (red) as a
function of distance. The shaded region shows the 1-$\sigma$ uncertainty.
$\dot{P}_{\text{b,Gal}}$ was calculated using the McMillan Galactic
potential[10]. The vertical blue (left) and magenta (right) lines indicate the
NE2001[11] and YWM16[12] DM-distance estimates respectively. The horizontal
dashed black line shows the GR expectation of $\dot{P}_{\text{b,exs}}=0$.

For the radiative $\dot{P}_{\text{b}}$ test of GR to be considered ‘passed’,
$\dot{P}_{\text{b,exs}}$ must be consistent with zero, given that the other
contributions have been accurately accounted for. However, although we have
now quantified the pulsar’s proper motion, we still lack an accurate estimate
of the pulsar’s distance. This is demonstrated in Figure 2, where we show how
$\dot{P}_{\text{b,exs}}$ depends on the distance to PSR J1757–1854. Assuming
GR, the distance to PSR J1757–1854 should fall between approximately 8.5–14.5
kpc. Therefore, available distance estimates based on the pulsar’s dispersion
measure (DM), including both the NE2001[11] (7.4 kpc) and the YMW16[12] (19.6
kpc) electron density models, are currently excluded. This is also shown in
Figure 2. However, we note that these DM-distance estimates typically come
with large uncertainties[13], making a definitive determination difficult.
Additionally, the McMillan Galactic potential model[10] we have used in our
analysis has known shortcomings at distances of approximately 7–9 kpc in the
central regions of the Galaxy near PSR J1757–1854’s position, further
complicating this assessment.

Without an accurate distance to PSR J1757–1854, the utility of the radiative
test of gravity is unfortunately limited. As seen in Figure 2, the unknown
distance introduces an approximate uncertainty to $\dot{P}_{\text{b,exs}}$ of
at least $\pm 20\times 10^{-15}$, more than three times the current
uncertainty of $\dot{P}_{\text{b,obs}}=6\times 10^{-15}$, such that the
precision of the radiative test of GR is limited to about 0.4 %. We are
currently exploring ways of further refining the estimated distance to PSR
J1757–1854 (e.g. through VLBI astrometry). However, by assuming GR, we will
still be able to inform our understanding of the pulsar’s evolution. For
example, the currently allowed distance range indicates a transverse velocity
based on $\mu_{\text{T}}$ between 180–300 km s-1, well in agreement with
simulations of the neutron star kick associated with the second progenitor
supernova[2].

##  3 Attempts to detect the presence of geodetic precession

Our current understanding of the binary evolution of PSR J1757–1854 suggests
the strong likelihood of a misalignment between the pulsar’s spin vector and
the orbital angular momentum vector[2]. As a result, we anticipate the
presence of geodetic precession at a rate of $\Omega_{\text{geod}}\simeq
3.07\,^{\circ}\,\text{deg}\,\text{yr}^{-1}$, which if detected may serve as
another test of GR. We have therefore analysed the current GBT dataset for
evidence of changes in the pulse profile, which would indicate a changing
line-of-sight through the emission cone of the pulsar as expected from
geodetic precession[14].

Our method adapts an approach previously applied[15] to PSR J1022+1001. Each
GBT observation (at both L and S-bands) spanning greater than 80% the orbital
period is summed in time, frequency and polarisation before being fit by a
template composed of three component Gaussian curves. We then normally re-
sample each profile bin using the off-pulse root-mean-squared noise value to
generate at least 1,000 randomly ‘re-noised’ profiles, to which the template
is re-fit. From each new fitted template, the positions and amplitudes of the
major and minor peaks are then measured, as well as the intercept positions at
10% and 50% of the profile’s maximum amplitude. The mean and standard
deviation of each parameter’s statistical distribution across all fitted
templates are then taken as the nominal value and its uncertainty.

Figure 3: Change in phase separation of the major and minor peaks of PSR
J1757–1854’s pulse profile, as measured by the GBT at S-band using the GUPPI
and VEGAS backends. The line of best fit corresponds to a decreasing
separation of $0.14\left(2\right)\,^{\circ}\,\text{yr}^{-1}$.

This approach has for the first time confirmed the presence of statistically
significant profile change. An example of this is given in Figure 3, which
shows that the separation between the two peaks of PSR J1757–1854’s profile
(termed the ‘major’ and ‘minor’ peaks) is decreasing (i.e. moving closer
together) by an average rate of approximately
$0.14\left(2\right)\,^{\circ}\,\text{yr}^{-1}$. The width of the profile at
both 10% and 50% also appears to be shrinking at a commensurate rate, with all
trends most pronounced at S-Band due to the decreased influence of scattering
(with respect to lower-frequency L-band observations). The precise physical
implications of these changes are to be explored in a future publication, but
their detection provides fundamental evidence for the presence of geodetic
precession, as well as evidence for the suspected spin/orbit misalignment
present in the binary system.

##  4 Searches for the companion neutron star

If PSR J1757–1854’s neutron star companion was confirmed to be an active radio
pulsar, it would immensely enhance the scientific utility of this pulsar and
its binary system. This is most obviously demonstrated by the prior example of
the Double Pulsar, where the presence of the second pulsar PSR J0737–3039B
allowed additional tests of gravity beyond those normally available, including
the detection of geodetic precession[16] and an independent determination of
the mass ratio[17].

We have therefore conducted an ongoing search for evidence of pulsations from
the neutron star companion to PSR J1757–1854. This search has focused on
observations from the GBT, which are recorded in a coherently-dedispersed
search mode so as to allow for optimal search sensitivity across all pulse
periods. Given our precise knowledge of PSR J1757–1854’s orbit as detailed in
Table 2, each observation is re-sampled using the inferred orbit of the
companion via the custom software package pysolator[18]. This allows the
companion to appear stationary such that any pulsations would appear as a
single narrow peak in a resulting Fourier spectrum. To account for the small
degree of uncertainty in the companion orbit, different re-sampled trials are
produced using values of the mass ratio $q=m_{\text{p}}/m_{\text{c}}$ between
$0.9554<q<0.9661$ in increments of $0.00005$. The resulting Fourier power
spectra from each observation are then stacked and summed, boosting the total
S/N of any pulsations.

To date, this search has unfortunately returned negative results, although the
complete GBT dataset has yet to be analysed. In total, approximately 389.7 hr
of GBT observations are currently on file, although there is significant
overlap in this dataset between the VEGAS and GUPPI backends during intervals
when both were recorded simultaneously. Ongoing searches for the companion
neutron star will continue, as the companion may become detectable in the
event that it precesses into view in the near future, given the expected
precession rate of $\Omega_{\text{geod}}\simeq
2.97\,^{\circ}\,\text{deg}\,\text{yr}^{-1}$.

##  5 Future observations and anticipated tests of gravity

###  5.1 Addition of MeerKAT

Although observations with South Africa’s MeerKAT telescope have been ongoing
since 2019 as part of the MeerTIME program[19], all these observations took
place with MeerKAT’s L-band receiver, which has a large 856 MHz bandwidth
centered at 1283.5 MHz. This frequency band is located at lower frequencies
than those of the GBT L-band receiver and therefore is much more susceptible
to pulse broadening via scattering than the GBT L-band[20]. As a consequence,
even though MeerKAT can achieve about twice the S/N of the GBT for this
target, the available timing precision from both telescopes is roughly
equivalent.

These effects are expected to be mitigated as part of the ongoing rollout of a
fleet of MeerKAT S-band receivers (with a bandwidth of approximately 1750 MHz
centred at 2625 MHz), designed by the MPIfR. The decreased influence of
scattering at these higher frequencies combined with the wider bandwidth and
the pulsar’s relatively flat spectral index suggest an improvement in TOA
precision to within 10 $\mu\text{s}$ or better (compared with a typical TOA
precision of 25–30 $\mu\text{s}$ currently achievable using similar TOA
integration times with the current GBT and MeerKAT receivers). This will
provide a significant advantage in the ongoing timing of the pulsar.

###  5.2 Future tests of gravity

PSR J1757–1854 is expected to provide new insight into two rarely-explored
tests of gravity within the next few years. The first of these is the
relativistic orbital deformation, characterised by the PK parameter
$\delta_{\theta}$, which describes the deviation of the orbit from a pure
Keplerian ellipse due to the effects of relativity. The measurability of
$\delta_{\theta}$ scales with both eccentricity and $\dot{\omega}$,
positioning PSR J1757–1854 as an ideal system for providing constraint in the
short-term. Meanwhile, the Hulse-Taylor pulsar[21] (lower $\dot{\omega}$) and
the Double Pulsar (lower eccentricity) have to date only weakly constrained
$\delta_{\theta}$ despite having been studied for far longer.

The second anticipated test involves the detection of Lense-Thirring
precession, which to date has not been detected in a double neutron star
binary system. We intend to detect this effect via the measurement of a change
in the semi-major axis[22], $\dot{x}_{\text{LT}}$. Although in general
$\dot{x}_{\text{LT}}$ becomes negligible for small misalignment angles between
the pulsar spin and orbital angular momentum vectors, our recent detection of
geodetic precession confirms that a significant misalignment is likely. The
Lense-Thirring effect is also proportional to the neutron star moment of
inertia; should the Lense-Thirring effect be detected here, it would lead to a
rare measurement of this quantity and thereby provide insight into the neutron
star equation of state.

The timescale over which each of these parameters is anticipated to be
constrained was determined by a simulation of the anticipated timing
campaigns. A simulation was produced by extrapolating the current GBT timing
campaign until the end of 2024, and including an equivalent MeerKAT S-band
campaign comprising one full orbit recorded every two months, starting in
2022. Timing ephemerides were fit to the data at set intervals, while
averaging the results of 30 realisations of the dataset. Based on these
simulations, we anticipate the achievement of a 3-$\sigma$ constraint of
$\delta_{\theta}$ between approximately 2024–2025, and of
$\dot{x}_{\text{LT}}$ between 2025–2026.

## Acknowledgments

The Parkes radio telescope is part of the Australia Telescope National
Facility which is funded by the Australian Government for operation as a
National Facility managed by CSIRO. We acknowledge the Wiradjuri people as the
traditional owners of the Observatory site. The Green Bank Observatory is a
facility of the National Science Foundation operated under cooperative
agreement by Associated Universities, Inc. The MeerKAT telescope is operated
by the South African Radio Astronomy Observatory, which is a facility of the
National Research Foundation, an agency of the Department of Science and
Innovation. Pulsar research at the Jodrell Bank Centre for Astrophysics and
the observations using the Lovell Telescope are supported by a consolidated
grant from the STFC in the UK. This work is also based on observations with
the 100-m telescope of the Max-Planck-Institut für Radioastronomie at
Effelsberg. AC and MB acknowledge OzGrav ARC grant CE170100004. AP and AR
acknowledge that (part of) this work has been funded using resources from the
research grant “iPeska” (P.I. Andrea Possenti) funded under the INAF national
call Prin-SKA/CTA approved with the Presidential Decree 70/2016. AP and AR
also acknowledge the support from the Ministero degli Affari Esteri e della
Cooperazione Internazionale - Direzione Generale per la Promozione del Sistema
Paese - Progetto di Grande Rilevanza ZA18GR02. AR acknowledges continuing
valuable support from the Max-Planck Society. MAM and HW are members of the
NANOGrav Physics Frontiers Center and supported by NSF awards 1430284 and
2020265.

## References

  * [1] M. J. Keith, A. Jameson, W. van Straten, M. Bailes, S. Johnston, M. Kramer, A. Possenti, S. D. Bates, N. D. R. Bhat, M. Burgay, S. Burke-Spolaor, N. D’Amico, L. Levin, P. L. McMahon, S. Milia and B. W. Stappers, The High Time Resolution Universe Pulsar Survey - I. System configuration and initial discoveries, Monthly Notices of the Royal Astronomical Society 409, 619 (December 2010).
  * [2] A. D. Cameron, D. J. Champion, M. Kramer, M. Bailes, E. D. Barr, C. G. Bassa, S. Bhandari, N. D. R. Bhat, M. Burgay, S. Burke-Spolaor, R. P. Eatough, C. M. L. Flynn, P. C. C. Freire, A. Jameson, S. Johnston, R. Karuppusamy, M. J. Keith, L. Levin, D. R. Lorimer, A. G. Lyne, M. A. McLaughlin, C. Ng, E. Petroff, A. Possenti, A. Ridolfi, B. W. Stappers, W. van Straten, T. M. Tauris, C. Tiburzi and N. Wex, The High Time Resolution Universe Pulsar Survey - XIII. PSR J1757-1854, the most accelerated binary pulsar, Monthly Notices of the Royal Astronomical Society 475, L57 (March 2018).
  * [3] R. A. Hulse and J. H. Taylor, Discovery of a pulsar in a binary system., Astrophysical Journal 195, L51 (January 1975).
  * [4] M. Burgay, N. D’Amico, A. Possenti, R. N. Manchester, A. G. Lyne, B. C. Joshi, M. A. McLaughlin, M. Kramer, J. M. Sarkissian, F. Camilo, V. Kalogera, C. Kim and D. R. Lorimer, An increased estimate of the merger rate of double neutron stars from observations of a highly relativistic system, Nature 426, 531 (December 2003).
  * [5] A. G. Lyne, M. Burgay, M. Kramer, A. Possenti, R. N. Manchester, F. Camilo, M. A. McLaughlin, D. R. Lorimer, N. D’Amico, B. C. Joshi, J. Reynolds and P. C. C. Freire, A Double-Pulsar System: A Rare Laboratory for Relativistic Gravity and Plasma Physics, Science 303, 1153 (February 2004).
  * [6] D. R. Lorimer and M. Kramer, Handbook of Pulsar Astronomy (Cambridge Univ. Press, Cambridge, 2005).
  * [7] P. C. C. Freire and N. Wex, The orthometric parametrization of the Shapiro delay and an improved test of general relativity with binary pulsars, Monthly Notices of the Royal Astronomical Society 409, 199 (November 2010).
  * [8] T. Damour and J. H. Taylor, On the Orbital Period Change of the Binary Pulsar PSR 1913+16, Astrophysical Journal 366, p. 501 (January 1991).
  * [9] I. S. Shklovskii, Possible Causes of the Secular Increase in Pulsar Periods., Soviet Astronomy 13, p. 562 (feb 1970).
  * [10] P. J. McMillan, The mass distribution and gravitational potential of the Milky Way, Monthly Notices of the Royal Astronomical Society 465, 76 (February 2017).
  * [11] J. M. Cordes and T. J. W. Lazio, NE2001.I. A New Model for the Galactic Distribution of Free Electrons and its Fluctuations, arXiv e-prints (Jul 2002).
  * [12] J. M. Yao, R. N. Manchester and N. Wang, A New Electron-density Model for Estimation of Pulsar and FRB Distances, Astrophysical Journal 835, p. 29 (January 2017).
  * [13] A. D. Cameron, D. J. Champion, M. Bailes, V. Balakrishnan, E. D. Barr, C. G. Bassa, S. Bates, S. Bhandari, N. D. R. Bhat, M. Burgay, S. Burke-Spolaor, C. M. L. Flynn, A. Jameson, S. Johnston, M. J. Keith, M. Kramer, L. Levin, A. G. Lyne, C. Ng, E. Petroff, A. Possenti, D. A. Smith, B. W. Stappers, W. van Straten, C. Tiburzi and J. Wu, The High Time Resolution Universe Pulsar Survey - XVI. Discovery and timing of 40 pulsars from the southern Galactic plane, Monthly Notices of the Royal Astronomical Society 493, 1063 (March 2020).
  * [14] M. Bailes, Geodetic precession in binary pulsars., Astronomy and Astrophysics 202, 109 (August 1988).
  * [15] P. V. Padmanabh, E. D. Barr, D. J. Champion, R. Karuppusamy, M. Kramer, A. Jessner and P. Lazarus, Revisiting profile instability of PSR J1022+1001, Monthly Notices of the Royal Astronomical Society 500, 1178 (January 2021).
  * [16] R. P. Breton, V. M. Kaspi, M. Kramer, M. A. McLaughlin, M. Lyutikov, S. M. Ransom, I. H. Stairs, R. D. Ferdman, F. Camilo and A. Possenti, Relativistic Spin Precession in the Double Pulsar, Science 321, p. 104 (July 2008).
  * [17] M. Kramer, I. H. Stairs, R. N. Manchester, M. A. McLaughlin, A. G. Lyne, R. D. Ferdman, M. Burgay, D. R. Lorimer, A. Possenti, N. D’Amico, J. M. Sarkissian, G. B. Hobbs, J. E. Reynolds, P. C. C. Freire and F. Camilo, Tests of General Relativity from Timing the Double Pulsar, Science 314, 97 (October 2006).
  * [18] A. Ridolfi, PYSOLATOR: Remove orbital modulation from a binary pulsar and/or its companion (March 2020).
  * [19] M. Bailes, A. Jameson, F. Abbate, E. D. Barr, N. D. R. Bhat, L. Bondonneau, M. Burgay, S. J. Buchner, F. Camilo, D. J. Champion, I. Cognard, P. B. Demorest, P. C. C. Freire, T. Gautam, M. Geyer, J. M. Griessmeier, L. Guillemot, H. Hu, F. Jankowski, S. Johnston, A. Karastergiou, R. Karuppusamy, D. Kaur, M. J. Keith, M. Kramer, J. van Leeuwen, M. E. Lower, Y. Maan, M. A. McLaughlin, B. W. Meyers, S. Osłowski, L. S. Oswald, A. Parthasarathy, T. Pennucci, B. Posselt, A. Possenti, S. M. Ransom, D. J. Reardon, A. Ridolfi, C. T. G. Schollar, M. Serylak, G. Shaifullah, M. Shamohammadi, R. M. Shannon, C. Sobey, X. Song, R. Spiewak, I. H. Stairs, B. W. Stappers, W. van Straten, A. Szary, G. Theureau, V. Venkatraman Krishnan, P. Weltevrede, N. Wex, T. D. Abbott, G. B. Adams, J. P. Burger, R. R. G. Gamatham, M. Gouws, D. M. Horn, B. Hugo, A. F. Joubert, J. R. Manley, K. McAlpine, S. S. Passmoor, A. Peens-Hough, Z. R. Ramudzuli, A. Rust, S. Salie, L. C. Schwardt, R. Siebrits, G. Van Tonder, V. Van Tonder and M. G. Welz, The MeerKAT telescope as a pulsar facility: System verification and early science results from MeerTime, Publications of the Astronomical Society of Australia 37, p. e028 (July 2020).
  * [20] M. Kramer, I. H. Stairs, V. Venkatraman Krishnan, P. C. C. Freire, F. Abbate, M. Bailes, M. Burgay, S. Buchner, D. J. Champion, I. Cognard, T. Gautam, M. Geyer, L. Guillemot, H. Hu, G. Janssen, M. E. Lower, A. Parthasarathy, A. Possenti, S. Ransom, D. J. Reardon, A. Ridolfi, M. Serylak, R. M. Shannon, R. Spiewak, G. Theureau, W. van Straten, N. Wex, L. S. Oswald, B. Posselt, C. Sobey, E. D. Barr, F. Camilo, B. Hugo, A. Jameson, S. Johnston, A. Karastergiou, M. Keith and S. Osłowski, The relativistic binary programme on MeerKAT: science objectives and first results, Monthly Notices of the Royal Astronomical Society 504, 2094 (June 2021).
  * [21] J. M. Weisberg and Y. Huang, Relativistic Measurements from Timing the Binary Pulsar PSR B1913+16, Astrophysical Journal 829, p. 55 (September 2016).
  * [22] T. Damour and J. H. Taylor, Strong-field tests of relativistic gravity and binary pulsars, Physical Review D (Particles, Fields, Gravitation, and Cosmology) 45, 1840 (March 1992).