11institutetext: Sano Centre for Computational Medicine, Cracow, Poland
11email<EMAIL_ADDRESS>22institutetext: Informatics Institute,
University of Amsterdam, Amsterdam, The Netherlands 33institutetext: The
Medical Centre of Postgraduate Education, Warsaw, Poland 44institutetext:
Poznan University of Medical Sciences, Poznan, Poland 55institutetext: Medical
University of Warsaw, Warsaw, Poland 66institutetext: Warsaw University of
Technology, Warsaw, Poland

# BabyNet: Residual Transformer Module for Birth Weight Prediction on Fetal
Ultrasound Video

Szymon Płotka 11 2 2 Michal K. Grzeszczyk 11 Robert Brawura-Biskupski-Samaha
33 Paweł Gutaj 44 Michał Lipa 55 Tomasz Trzciński 66 Arkadiusz Sitek 11

###### Abstract

Predicting fetal weight at birth is an important aspect of perinatal care,
particularly in the context of antenatal management, which includes the
planned timing and the mode of delivery. Accurate prediction of weight using
prenatal ultrasound is challenging as it requires images of specific fetal
body parts during advanced pregnancy which is difficult to capture due to poor
quality of images caused by the lack of amniotic fluid. As a consequence,
predictions which rely on standard methods often suffer from significant
errors. In this paper we propose the Residual Transformer Module which extends
a 3D ResNet-based network for analysis of $2D+t$ spatio-temporal ultrasound
video scans. Our end-to-end method, called BabyNet, automatically predicts
fetal birth weight based on fetal ultrasound video scans. We evaluate BabyNet
using a dedicated clinical set comprising 225 2D fetal ultrasound videos of
pregnancies from 75 patients performed one day prior to delivery. Experimental
results show that BabyNet outperforms several state-of-the-art methods and
estimates the weight at birth with accuracy comparable to human experts.
Furthermore, combining estimates provided by human experts with those computed
by BabyNet yields the best results, outperforming either of other methods by a
significant margin. The source code of BabyNet is available at
https://github.com/SanoScience/BabyNet.

###### Keywords:

Deep learning Fetal birth weight Transformer

††S. Płotka and M. K. Grzeszczyk – Authors contributed equally.

##  1 Introduction

Fetal birth weight (FBW) is a significant indicator of perinatal health
prognosis. Accurate prediction of FBW, as well as gestational age,
complications in pregnancy, and maternal physical parameters are critical in
determining the best method of delivery (natural or Cesarean). These factors
are widely used as a part of the hospital admission procedure in the World
[14]. However, FBW prediction is a challenging task, requiring highly visible
fetal body standard planes, which can only be identified by experienced
sonographers. Unfortunately, weight predictions provided by experienced
sonographers are often imprecise, with up to 10% mean absolute percentage
errors. Currently, FBW is estimated on the basis of fetal biometric
measurements of body organs – head circumference (HC), biparietal diameter
(BPD), abdominal circumference (AC), femur length (FL), which are used as the
input to heuristic formulae [6], [11].

In recent years, machine learning-based methods have been proposed as a
possible means of automating FBW prediction. Lu et al. [9], [10] presents a
solution based on an ensemble model consisting of Random Forest, XGBoost and
LightGBM algorithms. Tao et al. [20] use a hybrid-LSTM network model [24] for
temporal data analysis. Convolutional neural network (CNN)-based models are
also proposed to estimate fetal weight based on ultrasound images [2], [5] or
videos [13], [12]. However, such methods do not rely on the true FBW as the
ground truth, but instead predict it through heuristic formulae using
estimated fetal body-part biometrics, which is prone to errors.

Recently, Transformers [22] have been proposed as an alternative architecture
to CNNs, and have achieved competitive performance for many computer vision
tasks e.g. Vision Transformer (ViT) for image classification [3] or Video
Vision Transformer (ViViT) for video recognition [1]. Transformers utilize the
Multi-Head Self-Attention (MHSA) mechanism to learn the global context between
input sequence elements. Unfortunately, due to their high computational
complexity, Transformers require a large amount of training data and long
training times. Many methods have been developed to bridge the gap between
sample-efficient learning with a high inductive bias of CNNs and performance
but data-inefficient Transformers. Hybrid models utilizing CNN layers and
Transformer blocks have also been introduced [4], [8], [15].

In this paper we utilize Transformers for direct estimations of fetal weights
from US videos. We implement this solution as an extension of a 3D ResNet-
based network [21] with a Residual Transformer Module (RTM) called BabyNet.
The RTM allows local and global feature representation through residual
connections and utilization of convolutional layers. This representation is
refined through the global self-attention mechanism included inside RTM.
BabyNet is a hybrid neural network that efficiently bridges CNNs and
Transformers for $2D+t$ spatio-temporal ultrasound video scans analysis to
directly predict fetal birth weight. The main contribution of our work is as
follows: (1) We provide an end-to-end method for birth weight estimation based
directly on fetal ultrasound video scans, (2) We introduce a novel Residual
Transformer Module by adding temporal position encoding to 3D MHSA in 3D
ResNet-based neural network, (3) To the best of our knowledge, BabyNet is the
first framework to automate fetal birth weight prediction on fetal ultrasound
video scans trained and validated with data acquired one day prior to
delivery.

Figure 1: The overview of our proposed BabyNet method for birth weight
estimation directly from fetal US video scans. In BabyNet, we replace two
Residual Modules of 3D ResNet-18 with two Residual Transformer Modules (RTM)
containing 3D Multi-Head Self-Attention (MHSA) with Relative Positional
Encoding (RPE). RPE is calculated as the sum of height ($R_{h}$), width
($R_{w}$) and temporal ($R_{t}$) position encodings. For clarity, only one
attention head is presented in the image. The network takes 16 consecutive
frames as the input to make a single-segment prediction. All frames for a
given patient are divided into non-overlapping 16-frame segments and a
patient-level prediction is obtained by averaging all segment predictions.

##  2 Method

The overview of our method for end-to-end FBW prediction is presented in Fig.
1. We use 3D ResNet-18 for high-level US feature extraction. The RTM is
designed to learn local and global feature representation with 3D Multi-Head
Self-Attention mechanism and convolutional layers. We replace the last two
residual modules of ResNet with RTMs.

###  2.1 Feature Extraction

We employ 3D ResNet-18 [21] as the base network to extract high-level $2D+t$
spatio-temporal US feature representations. The initial input to the network
is US video sequence $S_{US}\in\mathbb{R}^{T_{0}\times 1\times H_{0}\times
W_{0}}$ of height $H_{0}$, width $W_{0}$ and frame number $T_{0}$. It is
transformed via convolutional residual modules to a low-resolution feature map
sequence $S^{{}^{\prime}}_{US}\in\mathbb{R}^{T_{1}\times D_{1}\times
H_{1}\times W_{1}}$, where $T_{1}=T_{0}/4$, $D_{1}=512$, $H_{1}=H_{0}/8$, and
$W_{1}=W_{0}/8$. Multi-channel, low-resolution feature map sequences are fed
to the RTM.

###  2.2 Residual Transformer Module

Residual modules are constructed from a layer followed by a rectified linear
unit (ReLU) and Batch Normalization. This structure is repeated two or three
times with a skip connection of the input added to the output of the previous
layers [7]. To include global low-resolution feature map context processing
via a self-attention mechanism we design RTM in a similar manner to BoT [19].
Our RTM extends BoT to 3D space by adding temporal position encoding [17] to
3D Multi-Head Self-Attention. BoT utilizes MHSA instead of $3\times 3$
convolution in the residual bottleneck module, created to decrease the
computational complexity in deeper ResNet architectures. 3D ResNets are often
shallower and do not contain Bottleneck blocks. Thus, to utilize the self-
attention mechanism in shallower ResNets we replace the last convolutional
layer in the residual module with MHSA, and define RTM as:

$y=BN\left(MHSA(\sigma(BN(Conv(x)\right)+x$ (1)

where $x$ and $y$ are input and output of the RTM respectively, $Conv$ denotes
the convolutional layer, $BN$ is Batch Normalization and $\sigma$ stands for
ReLU.

###  2.3 3D Multi-Head Self-Attention

To learn multiple attention representations at different positions, instead of
performing a single attention, many self-attention heads (Multi-Head Self-
Attention) are jointly trained with their outputs concatenated [22]. Since
such operation is permutation-invariant, positional encoding r needs to be
added to include positional information. Depending on the application,
absolute (e.g. sinusoidal) or relative positional encodings (RPE) [17],
recently identified as a better fit for vision tasks [23], can be used. To
process $2D+t$ US videos with MHSA we add temporal positional encoding to the
2D RPE and compute positional encoding r as the sum of
$R_{h}\in\mathbb{R}^{1{\times}D{\times}H{\times}1}$,
$R_{w}\in\mathbb{R}^{1{\times}D{\times}1{\times}W}$ and
$R_{t}\in\mathbb{R}^{T{\times}D{\times}1{\times}1}$, the height, width and
temporal positional encodings respectively. Finally, we compute the 3D MHSA
output of $S^{{}^{\prime\prime}}_{US}\in\mathbb{R}^{T\times D\times H\times
W}$ input as:

$MHSA\left(S^{{}^{\prime\prime}}_{US}\right)=concat\left[softmax\left(\frac{Q_{i}(K_{i}+r)^{T}}{\sqrt{d}}\right)V_{i}\right]$
(2)

where $T=\frac{T_{1}}{2}$, $D=D_{1}$, $H=\frac{H_{1}}{2}$,
$W=\frac{W_{1}}{2}$, $Q_{i}$, $K_{i}$, $V_{i}$ are queries, keys and values
for the ith attention head calculated from
$W_{Q}(S^{{}^{\prime\prime}}_{US})$, $W_{K}(S^{{}^{\prime\prime}}_{US})$ and
$W_{V}(S^{{}^{\prime\prime}}_{US})$ $1\times 1\times 1$ 3D convolutions
performed over input $S^{{}^{\prime\prime}}_{US}$ and $d$ is $D$ divided by
the number of heads.

##  3 Experiments

In this section, we describe our dataset and present architectural details of
BabyNet. We compare BabyNet’s performance with other $2D+t$ spatio-temporal
video analysis methods and with results obtained from clinicians. We show,
through an ablation study, the importance of BabyNet components that have been
added or replaced in 3D ResNet-18.

Figure 2: Sample US frames extracted from fetal US videos. The frames show the
fetal body part standard planes of the head, abdomen and femur respectively,
going from left to right. Images obtained several hours before delivery are of
lower quality than at earlier stages of pregnancy due to the lack of amniotic
fluid.

####  3.0.1 Dataset and Pre-processing.

Ethical Committee approval was obtained for all subjects enrolled in the
study. The dataset consists of 225 2D fetal ultrasound video scans in standard
plane view of fetal head, abdomen, and femur. The multi-centre dataset was
obtained from 75 pregnant women aged 21 to 42 and acquired through routine US
examinations less than 24 hours prior to delivery. The data was acquired by
three experienced sonographers using GE Voluson E6 and S10 devices. Each US
video scan is stored in DICOM file format, captured in two resolutions:
$960\times 720$ and $852\times 1136$ pixels. The number of frames is between
463 and 1448, with a mean of $852$. The US videos were obtained in sector scan
sweep mode with frame per second (FPS) between 24 and 37. For each video, we
resample pixel spacing to $0.2\times 0.2$ mm. As the ground truth, we use the
true fetal weight measured at birth. The ground truth values were between
$2085$ and $4995$, with a mean of $3454$ grams [g].

####  3.0.2 Implementation Details.

We adopt 3D ResNet-18 [21] as our base neural network. Table 1 presents the
architectural details of BabyNet, as compared to 3D ResNet-18. BabyNet
comprises a 3D convolutional stem followed by conv stages: three with two
residual modules each, and one final stage implemented with two RTMs. The
output of the final RTM is global average pooled (GAP) and fed to the fully-
connected (FC) layer with one neuron (512 input weights) for fetal birth
weight prediction. We implement our model with PyTorch and train it using an
NVIDIA RTX 2080 Ti 24GB GPU with a mini-batch size of 2 and an initial
learning rate of $1\times 10^{-4}$ with a step decay by a factor of $g=0.1$
every $160^{th}$ epochs until convergence over 200 epochs. To minimize the
Mean Squared Error (MSE) loss function, we employ an ADAM optimizer with
$1\times 10^{-4}$ weight decay. During training, we apply data augmentation
including rotate ($\pm$ 25∘), random brightness and contrast, horizontal flip,
image compression and blur for each mini-batch. We retain height and width
ratio and resize video frames to $64\times 64$ ($H_{0}\times W_{0}$) with
padding. The number of attention heads is empirically set to 4, while the
temporal sequence length $T_{0}$ is 16. Thus, BabyNet transforms US input
sequence $S_{US}\in\mathbb{R}^{16\times 1\times 64\times 64}$ to the output
$O_{S_{US}}\in\mathbb{R}^{1}$ of predicted fetal birth weights. We perform
5-fold cross-validation (CV) to compare and verify the robustness of the
regression algorithm. We ensure that data from a single patient appears only
in a single fold.

Table 1: Comparison of ResNet3D-18 and BabyNet architectures. We replace the last two residual modules of 3D ResNet-18 with two Residual Transformer Modules containing a 3D MHSA instead of the second $3\times 3$ 3D convolution. Stage name | Output size | 3D ResNet-18 | BabyNet
---|---|---|---
conv1 | $T_{0}\times\frac{H_{0}}{2}\times\frac{W_{0}}{2}$ |  $3\times 7\times 7,64,$ stride $1\times 2\times 2$
conv2 | $T_{0}\times\frac{H_{0}}{2}\times\frac{W_{0}}{2}$ |  [  $3\times 3\times 3,64$ $3\times 3\times 3,64$ ] $\times 2$ |  [  $3\times 3\times 3,64$ $3\times 3\times 3,64$ ] $\times 2$
conv3 | $\frac{T_{0}}{2}\times\frac{H_{0}}{4}\times\frac{W_{0}}{4}$ |  [  $3\times 3\times 3,128$ $3\times 3\times 3,128$ ] $\times 2$ |  [  $3\times 3\times 3,128$ $3\times 3\times 3,128$ ] $\times 2$
conv4 | $\frac{T_{0}}{4}\times\frac{H_{0}}{8}\times\frac{W_{0}}{8}$ |  [  $3\times 3\times 3,256$ $3\times 3\times 3,256$ ] $\times 2$ |  [  $3\times 3\times 3,256$ $3\times 3\times 3,256$ ] $\times 2$
conv5 | $\frac{T_{0}}{8}\times\frac{H_{0}}{16}\times\frac{W_{0}}{16}$ |  [  $3\times 3\times 3,512$ $3\times 3\times 3,512$ ] $\times 2$ |  [$\overbrace{\begin{tabular}[]{@{}c@{}}$3\times 3\times 3,512$\\\ {MHSA}\end{tabular}}^{RTM}$] $\times 2$
| $1\times 1\times 1$ | Global Avg Pooling, FC layer

####  3.0.3 Evaluation Metrics.

As measurement metrics, we use Root Mean Square Error (RMSE), Mean Absolute
Error (MAE), and Mean Absolute Percentage Error (MAPE) to evaluate the
regression performance.

####  3.0.4 Comparison with Clinicians and State-of-the-art Algorithms.

We compare BabyNet with several $2D+t$ spatio-temporal video analysis methods.
In particular, we compare it with results obtained by clinicians in [18] as
well as results obtained by clinicians for the dataset used in this work. We
also present results for Video Vision Transformer (ViViT) [1] and test the
hybrid approach of 2D ResNet-50 as a convolutional feature extractor (without
GAP and FC layers) to ViViT network. Finally, we utilize a vanilla 3D
ResNet-18 [21]. We train all models in the same fashion as BabyNet.

Table 2 presents a comparison of 5-fold CV results for all tested methods.
Results for machine learning methods are out-of-fold predictions. Combination
of estimations performed by clinicians with estimations provided by BabyNet is
the most accurate, with MAE of $180\pm 156$ (max p-value $<0.001$), RMSE of
$237\pm 145$ (max p-value $<0.001$), and MAPE of $5.2\pm 4.6$ (max p-value
$<0.001$). Max p-value is the maximum paired dual sided p-value computed for
results of the ”Clinicians (this work) & BabyNet” method and other methods
listed in Table 2.

We did not detect a statistically significant difference between the
performance of clinicians measured in [18] and our algorithm (p-value = 0.6).
Estimations provided by clinicians in our study seem to be better than those
provided by clinicians in [18] (p-value = 0.04) and BabyNet (p-value = 0.07).
Out of all neural networks investigated in this work, the hybrid approach of
utilizing 3D convolutions and 3D MHSA within RTM as a part of 3D ResNet-18
outperforms other methods based on plain CNNs, plain Transformer or
CNN+Transformer networks.

We noted that the best results were obtained by averaging estimations provided
by clinicians and by BabyNet. The performance of the ensemble of clinicians &
BabyNet was better by 18% compared to clinicians alone in terms of mMAPE,
which is a clear indication of added value and potential clinical benefits of
BabyNet.

Table 2: Five-fold cross-validation results and comparison of state-of-the-art methods. The mean of Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) across all folds are reported. Method | mMAE [g] | mRMSE [g] | mMAPE [%]
---|---|---|---
Clinicians (from [18]) | - | - | $7.9\pm 6.8$
Clinicians (this work) | $213\pm 155$ | $264\pm 158$ | $6.3\pm 4.8$
ViViT [1] | $361\pm 244$ | $444\pm 230$ | $10.6\pm 7.3$
2D ResNet + ViViT | $344\pm 241$ | $426\pm 226$ | $10.3\pm 7.2$
3D ResNet-18 [21] | $328\pm 234$ | $421\pm 225$ | $10.1\pm 7.1$
BabyNet | $254\pm 230$ | $341\pm 215$ | $7.5\pm 6.6$
Clinicians (this work) & BabyNet | $\mathbf{180\pm 156}$ | $\mathbf{237\pm 145}$ | $\mathbf{5.2\pm 4.6}$

####  3.0.5 Ablation study.

We conducted an ablation study to show the effectiveness of novel components
within BabyNet. In this experiment, we employ 3D ResNet-18 as the base neural
network for $2D+t$ spatio-temporal US video scan analysis. To learn multiple
relationships and enable capture of richer interpretations of the US video
sequence, we integrate CNN and Transformer by swapping the last convolutional
layer in the residual module for MHSA. To further enhance $2D+t$ spatio-
temporal feature representation in space and time, we add temporal position
encoding (TPE). Table 3 demonstrates that the combination of CNN with a
Transformer-based module, MHSA and temporal position encoding improves
performance of the weight-estimation task directly from US video scan.

Table 3: Ablation study. Method | mMAE [g] $\downarrow$ | mRMSE [g] $\downarrow$ | mMAPE [%] $\downarrow$
---|---|---|---
3D ResNet-18 (base) | $328\pm 234$ | $421\pm 225$ | $10.1\pm 7.1$
\+ RTM | $277\pm 228$ | $374\pm 221$ | $8.1\pm 7.0$
\+ RTM + TPE (ours) | $\mathbf{254\pm 230}$ | $\mathbf{341\pm 215}$ | $\mathbf{7.5\pm 6.6}$

##  4 Discussion

In this work we were not able to match the performance of the clinicians in
estimating fetal weight (mMAPE 7.5% vs. 6.3%); however, clinicians who worked
with us and provided measurements are top experts in performing biometric
measurements. On the other hand, we were able to match the performance of
clinicians reported in [18] (7.5% vs 7.9% p-value=0.6). The training data set
was relatively small and we expect to significantly improve the performance of
BabyNet by using more data in future work.

The method presented here can be characterized as end-to-end. Due to $2D+t$
spatio-temporal feature processing it does not require standard plane
detection which substantially reduces the workload involved in performing the
estimation of FBW. In clinical practice, BabyNet can be used as an aid for
clinicians in their decision-making process regarding the type of delivery.
According to literature [16], [18] the heavier the child, the greater the
likelihood of Cesarean delivery. Serious complications may arise when a heavy
child’s FBW is misjudged. Under these circumstances, if natural delivery is
decided upon severe complications for both mother and child may arise.

This work has certain limitations. A relatively small number of patients was
used in the study, which can affect the accuracy and generalization of
results. A related issue is that the patient population is limited and we do
not know if BabyNet would work on a different population (e.g. different
race). The algorithm is trained and evaluated on short clips of US videos
recorded by clinicians. To operate in a clinical setting, further effort would
be needed to create a system that extracts appropriate clips for BabyNet
analysis.

##  5 Conclusions

In this paper we presented an extension of the 3D ResNet-based network with a
Residual Transformer Module (RTM), named BabyNet, for $2D+t$ spatio-temporal
fetal ultrasound video scan analysis. The proposed framework is an end-to-end
method that automatically performs fetal birth weight prediction. This is done
without the need for finding standard planes in ultrasound video scans, which
are required in the classical method of estimating fetal weight. Combining
classical and BabyNet estimations provides the best results, significantly
outperforming top expert clinicians who use available commercial tools. Our
method has the potential to help clinicians select – on the basis of US
examination – the type of delivery which is safest for the mother and the
child. Future work includes testing BabyNet on external datasets which are
preferably acquired using different devices and by operators with different
levels of experience. Moreover, we plan to use multimodal data – combine the
fetal US video and clinical data to improve the performance and robustness of
the model.

## Acknowledgements

This work is supported by the European Union’s Horizon 2020 research and
innovation programme under grant agreement Sano No 857533 and the
International Research Agendas programme of the Foundation for Polish Science,
co-financed by the European Union under the European Regional Development
Fund. We would like to thank Piotr Nowakowski for his assistance with
proofreading the manuscript.

## References

  * [1] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6836–6846 (October 2021)
  * [2] Bano, S., Dromey, B., Vasconcelos, F., Napolitano, R., David, A.L., Peebles, D.M., Stoyanov, D.: Autofb: Automating fetal biometry estimation from standard ultrasound planes. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 228–238. Springer (2021)
  * [3] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021)
  * [4] d’Ascoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G., Sagun, L.: Convit: Improving vision transformers with soft convolutional inductive biases. In: International Conference on Machine Learning. pp. 2286–2296. PMLR (2021)
  * [5] Feng, M., Wan, L., Li, Z., Qing, L., Qi, X.: Fetal weight estimation via ultrasound using machine learning. IEEE Access 7, 87783–87791 (2019)
  * [6] Hadlock, F.P., Harrist, R., Sharman, R.S., Deter, R.L., Park, S.K.: Estimation of fetal weight with the use of head, body, and femur measurements—a prospective study. American journal of obstetrics and gynecology 151(3), 333–337 (1985)
  * [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [8] Liu, Y., Sun, G., Qiu, Y., Zhang, L., Chhatkuli, A., Van Gool, L.: Transformer in convolutional neural networks. arXiv preprint arXiv:2106.03180 (2021)
  * [9] Lu, Y., Fu, X., Chen, F., Wong, K.K.: Prediction of fetal weight at varying gestational age in the absence of ultrasound examination using ensemble learning. Artificial intelligence in medicine 102, 101748 (2020)
  * [10] Lu, Y., Zhang, X., Fu, X., Chen, F., Wong, K.K.: Ensemble machine learning for estimating fetal weight at varying gestational age. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 9522–9527 (2019)
  * [11] Milner, J., Arezina, J.: The accuracy of ultrasound estimation of fetal weight in comparison to birth weight: A systematic review. Ultrasound 26(1), 32–41 (2018)
  * [12] Płotka, S., Klasa, A., Lisowska, A., Seliga-Siwecka, J., Lipa, M., Trzcinski, T., Sitek, A.: Deep learning fetal ultrasound video model match human observers in biometric measurements. Physics in Medicine & Biology (2022)
  * [13] Płotka, S., Włodarczyk, T., Klasa, A., Lipa, M., Sitek, A., Trzciński, T.: Fetalnet: Multi-task deep learning framework for fetal ultrasound biometric measurements. In: International Conference on Neural Information Processing. pp. 257–265. Springer (2021)
  * [14] Pressman, E.K., Bienstock, J.L., Blakemore, K.J., Martin, S.A., Callan, N.A.: Prediction of birth weight by ultrasound in the third trimester. Obstetrics & Gynecology 95(4), 502–506 (2000)
  * [15] Reynaud, H., Vlontzos, A., Hou, B., Beqiri, A., Leeson, P., Kainz, B.: Ultrasound video transformers for cardiac ejection fraction estimation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 495–505. Springer (2021)
  * [16] Scioscia, M., Vimercati, A., Ceci, O., Vicino, M., Selvaggi, L.E.: Estimation of birth weight by two-dimensional ultrasonography: a critical appraisal of its accuracy. Obstetrics & Gynecology 111(1), 57–65 (2008)
  * [17] Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018)
  * [18] Sherman, D.J., Arieli, S., Tovbin, J., Siegel, G., Caspi, E., Bukovsky, I.: A comparison of clinical and ultrasonic estimation of fetal weight. Obstetrics & Gynecology 91(2), 212–217 (1998)
  * [19] Srinivas, A., Lin, T.Y., Parmar, N., Shlens, J., Abbeel, P., Vaswani, A.: Bottleneck transformers for visual recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16519–16529 (2021)
  * [20] Tao, J., Yuan, Z., Sun, L., Yu, K., Zhang, Z.: Fetal birthweight prediction with measured data by a temporal machine learning method. BMC Medical Informatics and Decision Making 21(1), 1–10 (2021)
  * [21] Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6450–6459 (2018)
  * [22] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
  * [23] Wu, K., Peng, H., Chen, M., Fu, J., Chao, H.: Rethinking and improving relative position encoding for vision transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10033–10041 (2021)
  * [24] Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.c.: Convolutional lstm network: A machine learning approach for precipitation nowcasting. In: Advances in neural information processing systems. pp. 802–810 (2015)


# RoRD: Rotation-Robust Descriptors and Orthographic Views for Local Feature
Matching

Udit Singh Parihar∗1, Aniket Gujarathi∗1, Kinal Mehta∗1, Satyajit Tourani∗1,
Sourav Garg2,
Michael Milford2 and K. Madhava Krishna1 ∗Denotes authors with equal
contribution1Robotics Research Center, IIIT Hyderabad.2QUT Centre for
Robotics, Queensland University of Technology (QUT), Australia.This research
is supported by MathWorks.Code and dataset link:
https://github.com/UditSinghParihar/RoRDIn the IEEE’s published version of
this paper (DOI: 10.1109/IROS51168.2021.9636619), SIFT’s results in the
‘Rotated’ and ‘Average’ columns of Table I are incorrect, which we have now
corrected in this arXiv version and over GitHub. This change only affects the
performance ranking of MMA evaluation on the HPatches dataset, where our
proposed method still ranks the best when considering standard and average
settings. Other results in the paper which benchmark against SIFT, that is,
Pose Estimation and Visual Place Recognition, are not affected.

###### Abstract

The use of local detectors and descriptors in typical computer vision
pipelines works well until variations in viewpoint and appearance change
become extreme. Past research in this area has typically focused on one of two
approaches to this challenge: the use of projections into spaces more suitable
for feature matching under extreme viewpoint changes, and attempting to learn
features that are inherently more robust to viewpoint change. In this paper,
we present a novel framework that combines the learning of invariant
descriptors through data augmentation and orthographic viewpoint projection.
We propose rotation-robust local descriptors, learnt through training data
augmentation based on rotation homographies, and a correspondence ensemble
technique that combines vanilla feature correspondences with those obtained
through rotation-robust features. Using a range of benchmark datasets as well
as contributing a new bespoke dataset for this research domain, we evaluate
the effectiveness of the proposed approach on key tasks including pose
estimation and visual place recognition. Our system outperforms a range of
baseline and state-of-the-art techniques, including enabling higher levels of
place recognition precision across opposing place viewpoints, and achieves
practically useful performance levels even under extreme viewpoint changes. We
reduce pose estimation error by $86.72\%$ relative to state of the art.

Figure 1: Local feature matches using RoRD. Our method RoRD finds precise
local feature correspondences under extreme viewpoint ($\sim$180∘) changes for
both indoor and outdoor sequences.

##  I INTRODUCTION

Figure 2: RoRD pipeline. Our approach takes a pair of perspective view images
with significant viewpoint as input to the Orthographic View Generator, which
aligns the camera to the plane-surface-normal to generate the top views, as
explained in Section III-B. These top views are then passed to an ensemble of
Vanilla D2-Net[1] and RoRD techniques (ours). This approach creates precise
feature correspondences that are robust to variations in viewpoints.

The use of local detectors and descriptors is widespread in computer and
robotic vision problems that include descriptor matching between an image
pair[2, 1, 3, 4], relative pose estimation[4, 2], front-end pipelines for
SLAM[5], and image retrievals[5, 6]. Traditional local descriptors [7, 8]
typically employ a detector-descriptor pipeline; the detector comprises sparse
or dense keypoints, while the descriptor characterizes a patch around the
detector typically using a high dimensional vector. However, the performance
of tasks like image retrieval, and pose estimation that rely on these
classical descriptors degrades in the presence of large and disparate
viewpoint changes.

The rising popularity of Deep Architectures has resulted in deep local
detectors and descriptors [3, 1, 9], which bypass the need to hand-craft
features for the task at hand. Nonetheless, even popular implementations such
as the SuperPoint Detector and Descriptor[3] and D2-Net [1] do not handle very
large viewpoint changes.

In this paper we address the challenge of extreme viewpoint change by making
the following contributions:

  1. 1.

We extend the operating range of deep descriptors such as D2-Net [1] to
function under extreme viewpoint change, by augmenting data with a succession
of homographic transforms and learning rotation-robust descriptors trained on
this augmented data, achieving better generalization across both different
environment types and perspective transformations;

  2. 2.

We present a new dual-headed D2-Net model with a correspondence ensemble which
combines vanilla and rotation-robust feature correspondences, leading to
superior performance under all rotation variations;

  3. 3.

We introduce a new dataset that enables development and testing in this
domain, DiverseView Dataset, comprising 3 sequences of various outdoor and
indoor images captured under extreme viewpoint variations;

  4. 4.

We evaluate the performance of these rotation-robust descriptors on a variety
of tasks, datasets, and appropriate metrics, namely,

    1. (a)

Descriptor Matching, evaluated using Mean Matching Accuracy (MMA) metric on
the HPatches [10] dataset, demonstrating improved Mutual Nearest Neighbor
(MNN) based matches compared to [7, 3, 1],

    2. (b)

Pose Estimation, evaluated by the relative angular error between actual and
estimated poses on the DiverseView dataset, demonstrating improved pose
precision (orientation and translation) estimates compared to baselines [1, 2,
7],

    3. (c)

Visual Place Recognition (VPR), evaluated using the recall metric on the
Oxford RobotCar dataset [11], demonstrating a four fold improvement in
opposite-view place matching percision (7 m) compared to the 30m achieved by
previous state-of-the-art [12, 7, 1, 2], even under different camera
intrinsics and illumination conditions.

##  II RELATED WORKS

Here we briefly cover relevant research on local feature correspondences and
orthographic view generation.

###  II-A Local Feature Correspondences

Traditional hand-crafted feature methods such as SIFT[7], SURF[8], ORB[13],
and BRISK[14] have been successfully used in various computer vision
applications for over a decade. However, these methods degrade significantly
when exposed to extreme variations in scene appearance and camera viewpoint.
More recently, the focus has shifted to Convolutional Neural Networks (CNNs)
for the task of feature extraction, which offer promising advances in
capability with respect to these challenges.

Recent research such as D2-Net[1] and SuperPoint[3] employ a CNN-based method
to obtain both keypoints and descriptors. D2-Net[1] uses a detect-and-describe
approach for sparse feature extraction, rather than performing early detection
using the low-level features. This approach performs well or even better under
challenging conditions such as illumination changes and weakly textured
scenes. Sparse-to-Dense[4] uses a siamese architecture to learn its local
descriptors. Similar to D2-Net[1], it uses VGG-16 as its backbone.

LBP-HF [15] introduces a method to obtain rotation-invariant image descriptors
based on [16]. Its rotation-invariant features are based on local binary
pattern histograms. RI-LBD [17] employs a learning-based method to obtain
local binary descriptors. It jointly learns the projection matrix and each
pattern’s orientation to obtain rotation-invariant local binary descriptors.
Other CNN-based alternatives developed for rotation-equivariance [18, 19, 20]
require re-designing of CNN architectures, whereas our proposed method
achieves rotation-robustness by training with in-plane rotated images and
performing correspondence ensemble.

###  II-B Orthographic View Generation

Past research has shown that orthographic view generation improves performance
in tasks such as image retrieval [5], and feature matching [6]. Monolayout
[21] predicts an amodal scene layout using an encoder to extract features at
multiple scales. Rectified features [6] performs perspective unwarping via the
help of 3D information. It uses [22] to first predict a dense depth map and
eventually clusters the surface normals for each point. After clustering, it
then computes a rectifying homography to align the camera with the resulting
surface normal. [23] estimates the homography that maps the perspective view
to the bird’s eye view by predicting vertical and horizontal vanishing lines
using a CNN. Earlybird [5] performs visual place recognition for indoor
environments by using floor features, estimating a fixed homography via the
RANSAC + 4 point algorithm [24]. [25] introduced a spatial transformer GAN for
generating boosted Inverse Perspective Mapping (IPM), enabling improved
detection of road markings and semantic interpretation of road scenes.
Approaches like [5] and [23] obtain the necessary orthographic views through
known a-priori information. In this work, we show that orthographic views
alone are not sufficient when used with existing local features under
significant viewpoint variations, and that rotation-robust descriptors are
necessary to better leverage such views to achieve superior performance.

##  III Methodology

The main objective of this work is to determine accurate pixel-level
correspondences between images under extreme viewpoint variations. Here we
introduce our proposed method for performing precise local feature matching by
learning rotation robust descriptors invariant to large viewpoint changes. We
further extend the applicability of the proposed descriptors through a
correspondence ensemble and orthographic view generation approach, as
discussed in subsequent sections. An ensemble architecture allows us to have
the illumination and scale invariance of the original D2Net, while RoRD makes
the network robust to high viewpoint changes.

###  III-A Feature correspondences

Figure 3: Example images of the training data used by RoRD. First row shows
training pair for PhotoTourism Dataset and second row shows it for the road
patches on the Oxford RobotCar Dataset.

####  III-A1 Rotation-Robust Descriptors

Given an input image $I_{1}$, we generate a corresponding image pair $I_{2}$
using a uniformly distributed discrete random homography transformation
consisting of $0^{\circ}-360^{\circ}$ rotations, as depicted in Figure 3. This
enables the generation of a large number of images with ground-truth pixel-
level annotations in a self-supervised manner. Along with a series of in-plane
rotations, we also incorporate perspectivity, scaling, and skewness
transformations to further simulate realistic viewpoint changes. We use the
PhotoTourism dataset for this training as detailed in the dataset section.

Our approach uses VGG-16 as the base architecture, similar to D2-Net[1], while
fine-tuning the last layer and keeping the earlier layers’ weights frozen.
Input to the network is a cropped $400\times 400\times 3$ image region and a
random rotation homography $H_{R}(\theta)$ is used to rotate the image in-
plane by $\theta$ degrees to create the training pair. We follow the same
training procedure of detect and describe as defined in [1], with primary
difference being the type of training data (and augmentation) used which in
our case is based on in-plane rotations. The local descriptors thus obtained
through this training procedure are referred to as RoRD (Rotation Robust
Descriptors).

####  III-A2 Correspondence Ensemble

While the above procedure leads to significant performance gains in local
feature matching under extreme viewpoint variations, we observe that
catastrophic forgetting could occur when viewpoint variations are rather
moderate or none. To overcome this limitation, we consider two solutions: one
based on training a single model and other based on training two separate
heads of the same model. Note that the first solution is merely chosen as a
baseline to justify the use of the second solution, as later demonstrated in
Section V. For the single model training, referred to as RoRD-Combined, we use
the combined training data from 3D depth and pose information (as was used in
the original D2-Net) and in-plane rotation and random homographies (as
proposed in this work).

For the second solution, we propose a Correspondence Ensemble (CE) method
where we use a dual-headed D2Net as shown in Fig. 2. The first head
corresponds to the output of vanilla D2-Net (trained with data as per the
original method), while the second head corresponds to RoRD (trained using in-
plane rotations data). This dual-headed approach improves model efficiency due
to a common backbone shared across both the models since we only train the
final layer. Once we have keypoints and descriptors from the two heads, mutual
nearest-neighbor based feature correspondences are obtained between an image
pair independently for each head. Using the descriptor distances of the
combined set of correspondences obtained from the two heads, we select the top
$50\%$ of the correspondences which have the lowest distance values. These
correspondences are then filtered by RANSAC-based geometric verification to
obtain a final set of feature correspondences.

Figure 4: Orthographic View Generation. We use 3D geometry to obtain our
necessary ROI patches in the form of orthographic projection. Depth
information is used to generate surface normals of our desired ROI. The camera
is aligned such that its optical axis is antiparallel to the surface normal
and obtains the desired orthographic view.

###  III-B Orthographic view generation

Most existing [2, 3, 1] feature matching methodologies use perspective view
images for detection, description and matching of high-confidence keypoints.
These methods work reliably on image pairs with smaller changes in viewpoint
or rotation but degrade as camera viewpoint becomes more extreme. As
previously demonstrated in [6, 5], orthographic views can help improve local
feature matching due to an increased visual overlap through such projections.
However, such views alone may not be sufficient to achieve this goal. In this
work, we show that rotation-robust features are additionally required to
better leverage orthographic views, especially when dealing with extreme
viewpoint variations. Here, we describe our approach to generating
orthographic views.

Our orthographic view generation process utilizes 3D information and has two
possible operating modes. In scenes where the layout is varied and the camera
is moving in 6-DoF, it utilizes ROI planar regions and surface normals. In
relatively consistent scene layout scenarios like self-driving cars, we use a
fast, lightweight IPM based approach to obtain top views of the road.

####  III-B1 Surface normal based orthographic view generation

For the DiverseView dataset, we use 3D geometry to obtain orthographic views
as shown in Fig. 4. Given a dominant planar region, we use depth information
(obtained through Intel Realsense D455 camera) to generate the 3D point cloud
of the desired ROI and calculate the surface normal of this plane using [26].
Further, we align a virtual camera such that its optical axis becomes
antiparallel to the surface normal of the plane and its location lies directly
above the centroid of the point cloud. Images taken using the virtual camera
location and the original camera location are related by a rectifying
homography, $H_{ortho}$. This matrix enables us to obtain the desired
orthographic view $x^{{}^{\prime}}$ from the perspective view $x$.

We use the rotation matrix $R$ to align the camera optical axis, $\vec{o}$,
with the semantic-plane-specific surface normal, $\vec{n}$, having distance
$d$ apart from the camera. To account for the translation between the
perspective view and the orthographic view, we also incorporate a fixed
translation $\vec{t}$ to ensure that our camera is placed at the right
location.

$\displaystyle\vec{v}$ $\displaystyle=\vec{o}\times\vec{n}$ (1) $\displaystyle
R$
$\displaystyle=I+[\vec{v}]_{x}+[\vec{v}]_{x}^{2}\times\frac{1-\vec{o}\cdot\vec{n}}{\lVert\vec{o}\times\vec{n}\rVert^{2}}$
(2)

$\vec{v}$ is the vector perpendicular to $\vec{o}$ and $\vec{n}$ and $[v]_{x}$
is its skew-symmetric matrix. We then use this rotation matrix to obtain the
rectifying homography matrix, $H_{ortho}$, which enables us to obtain the
desired orthographic view.

$\displaystyle x^{{}^{\prime}}$
$\displaystyle=K\>(R-\frac{\vec{t}\vec{n}^{T}}{d})\>K^{-1}\>x$ (3)
$\displaystyle x^{{}^{\prime}}$ $\displaystyle=H_{ortho}\>x$ (4)

$K$ is the intrinsic matrix and $K(R-\frac{\vec{t}\vec{n}^{T}}{d})K^{-1}$
corresponds to the rectifying homography matrix, $H_{ortho}$.

####  III-B2 Inverse Perspective Mapping (IPM) based orthographic view
generation

Considering an image pair in autonomous driving scenarios, with high-viewpoint
change and relative translation, the overlapping region between the pair
consists largely of the road. While seemingly aliased, in practice, roads
contain many discriminative features. We leverage the cracks and deformities
on the road surfaces to generate deep features which are robust to $180^{0}$
rotations, different intrinsics, and illumination change. We focus on the
benchmark Oxford RobotCar Dataset [11]. Due to the fixed camera geometry with
respect to the road, a single homography can rectify the front camera image,
and similarly, another homography can rectify the rear camera image. These
homographies are generated by mapping parallel lane markers on the road to an
appropriate sized polygonal region, and the best fit homography is calculated.

As shown in Table II and III, both the aforementioned methods boost the
performance of local feature matching for their respective datasets.

##  IV EXPERIMENTAL SETUP

In this section we describe the key benchmarks and newly contributed datasets
used to evaluate performance, as well as the range of evaluation metrics used.

###  IV-A Datasets

We report our results on two standard benchmark datasets and the new
DiverseView dataset.

####  IV-A1 PhotoTourism [27]

This dataset was only used for training. We used the following seven scenes
for this purpose: the Brandenburg Gate, Buckingham Palace, Grand Place
Brussels, Hagia Sophia Interior, Taj Mahal, Temple Nara Japan and Westminster
Abbey.

####  IV-A2 HPatches [10]

We use this dataset for evaluating local feature matching. For this purpose,
we use the original dataset as Standard HPatches (108 out of 116 scenes) as
done in [1]. We additionally proposed a variant which based on random in-plane
rotation of images from 0 to $360^{0}$, referred to as Rotated HPatches.

####  IV-A3 Oxford RobotCar Dataset [28]

We evaluate the Visual Place Recognition (VPR) performance enabled by our
underlying methods using two different traversals of the benchmark Oxford
RobotCar Dataset: Training Route ($2014/07/14-14:49:50$) and Testing Route
($2014/06/26-09:24:58$). Our training data comprises $6200$ images obtained
from both the front (Bumblebee) and rear (Grasshopper) facing cameras. Using
the same self-supervised training protocol as described in Section III-A and
orthographic view generation process described in Section III-B, top-views of
images are used and training image pairs are formed by in-plane rotations.
Note that front image is never compared to the rear image during training, and
training and test data belong to geographically distinct locations.

####  IV-A4 DiverseView dataset

Due to the lack of publicly available datasets with high viewpoint changes and
camera rotations, we introduce a new dataset comprising both outdoor and
indoor scenarios. The DiverseView dataset consists of 3 scenes of indoor and
outdoor locations, with images captured at high viewpoint changes and camera
rotations (up to 180∘). For the data collection, we have used the Intel
RealSense D455 camera with RGB, Depth, and IMU sensors. The DiverseView
dataset along with all sensor information for ground truth and orthographic
view generation purposes will be released publicly.

The three sequences of this dataset are as follows: Sequence 1 consists of
$1138$ images of an office-desk scene captured by moving around it in a semi-
circular path, thus exhibiting extreme rotation variations. Sequence 2
consists of $2534$ images obtained from a $360^{\circ}$ traversal about a
table-top. Sequence 3 consists of $3931$ images captured around a building
with graffiti-art with varied camera viewpoints and rotations in low-lighting
dusk conditions. For this dataset, evaluation is done using the model trained
on the PhotoTourism dataset, thus showcasing the generalizability of proposed
RoRD.

###  IV-B Evaluation

The versatility of our method is evaluated by showing results using three
different metrics on three different tasks. For the first task, we evaluate
the performance through an NN-based matching using MMA, as was done in prior
work [1], on the HPatches dataset. We then evaluate our methods and the
baselines on the task of Pose Estimation, as performed in [2], on the
DiverseView dataset. Finally, we evaluate the quality of local feature matches
on the Oxford RobotCar dataset for opposing-viewpoint visual place
recognition.

####  IV-B1 Local feature matching using MMA

We follow the standard protocol [1] for evaluating nearest neighbor feature
matching on HPatches dataset. A match is considered to be correct if the
backprojection of the keypoint using the ground truth homography matrix falls
within a specific pixel threshold. Computing the mean matching accuracy (MMA)
of all pairs, over varying thresholds, all the models are then compared on the
two variants of HPatches - Standard HPatches and Rotated HPatches. We use the
following baseline comparison methods: D2-Net [1] and SuperPoint [3], using
the trained model provided by the respective original authors; SIFT [7], using
its OpenCV implementation. From the proposed methods, we include RoRD, RoRD +
CE (Correspondence Ensemble) and RoRD-Combined, as described in Section III-A.

####  IV-B2 Pose Estimation using Rotation and Translation error

We use DiverseView dataset for this task. To generate precise ground truth
poses for the relative pose (R/t) estimates, we used RTABMAP SLAM [29] with
sensor fusion. Relative angular error,
$||\;log(\hat{R}R_{gt}^{T})\;||/\sqrt{2}$ in degrees is used to calculate the
error between the predicted, $\hat{R}$ and ground-truth $R_{gt}$ rotations. A
logarithmic mapping converts the difference rotation matrix,
$\hat{R}R_{gt}^{T}$, to an axis-angle representation in degrees. The Euclidean
norm is used between the ground-truth translation and predicted translation.
For evaluation, we sample $10$ representative images across the individual
dataset sequences. Each representative image is then matched with $100$ images
evenly distributed across the dataset sequence and then the extracted $R/t$ is
compared against the ground-truth. Finally, $10\times 100$ $R/t$ residual
errors are averaged to obtain final error for that dataset. We compare our
method against D2-Net [1], SuperPoint+SuperGlue [2] and SIFT [7] on both
perspective and orthographic views in R/t estimation.

####  IV-B3 Visual Place Recognition using Local Feature Matching

We evaluate the effectiveness of using rotation robust features and
orthographic images by performing visual place recognition on the Oxford
RobotCar dataset using GPS based priors [30]. $835$ front camera images are
used as queries and each query image is compared against all the rear images
(considered as reference database) within a $52$ m radius using GPS. The
reference image having the maximum number of inlier correspondences is
retrieved. We consider a match to be correct if the retrieved match lies
within a $7$ m localization radius of the car’s reverse traversal (visual
overlap).

TABLE I: Quantitative results for MMA on the HPatches dataset using pixel
thresholds 6/8/10. We highlight the first, second and third best MMA values.
We also provide averages over the results obtained by the Standard and Rotated
HPatches dataset. Our Ensemble method RoRD + CE outperforms all other methods
with RoRD and SIFT* being the second and the third best performer for the
averaged HPatches datasets, respectively.

Model | Standard | Rotated | Average
---|---|---|---
SIFT[7]* | 0.52/0.54/0.54 | 0.51/0.51/0.52 | 0.52/0.53/0.53
SuperPoint[3] | 0.69/0.71/0.73 | 0.21/0.22/0.22 | 0.45/0.46/0.48
D2-Net[1] | 0.73/0.81/0.84 | 0.17/0.20/0.22 | 0.45/0.50/0.53
(Ours) RoRD | 0.68/0.75/0.78 | 0.46/0.57/0.62 | 0.57/0.66/0.70
(Ours) RoRD Comb. | 0.71/0.78/0.81 | 0.44/0.54/0.59 | 0.58/0.66/0.70
(Ours) RoRD + CE | 0.79/0.84/0.86 | 0.48/0.59/0.64 | 0.64/0.72/0.75

##  V RESULTS

In this section, we show the performance comparisons of different models on
various tasks as explained in Section IV-B.

###  V-A Local Feature Matching

The quantitative performance of local feature matching on the HPatches dataset
is presented in Table I for three settings: standard, rotated, and the average
of two. As mentioned in Section IV-B, we follow the standard protocol of [1]
and use mutual NN matching, evaluating the results using MMA for pixel
thresholds 6, 8, and 10. Our proposed method RoRD + CE (correspondence
ensemble) consistently outperforms other methods including SIFT [7], ORB [13]
and SuperPoint [3].

For the Rotated HPatches dataset, SIFT achieves the best
results*00footnotetext: *In the IEEE’s published version of this paper (DOI:
10.1109/IROS51168.2021.9636619), SIFT’s results in the ‘Rotated’ and ‘Average’
columns of Table I are incorrect, which we have now corrected in this arXiv
version and over GitHub: https://github.com/UditSinghParihar/RoRD. This change
only affects the performance ranking of MMA evaluation on the HPatches
dataset, where our proposed method still ranks the best when considering
standard and average settings. Other results in the paper which benchmark
against SIFT, that is, Pose Estimation and Visual Place Recognition, are not
affected. and RoRD + CE achieves the second best, closely followed by RoRD.
Overall, considering the average performance across standard and rotated
HPatches, our ensemble method RoRD + CE achieves the best performance, while
also outperforming RoRD-Combined which shows the effectiveness of dual-headed
D2Net and correspondence ensemble over the combined data based training of a
single model.

Since SuperGlue [2] is a matcher with inherent outlier rejection based on
scene geometry, we have not included it in Table I where no geometric
verification (via RANSAC) is performed. For a fair comparison against
SuperGlue, we consider RoRD+RANSAC which achieves an average MMA of 0.78/0.95
using a pixel threshold of 6/10, as compared to 0.71/0.73 achieved by
SuperGlue.

###  V-B Pose estimation

Table II shows results on the DiverseView dataset for rotation and translation
estimation, relevant for downstream tasks like point cloud registration or
6-DoF localization. To evaluate the accuracy of the rotation and translation
estimates, we follow the evaluation metric as described in Section IV-B. In
this evaluation, we consider two additional aspects on top of the previous
analysis on HPatches: we use RANSAC-based geometric verification, and employ
orthographic views. We show that our proposed methods (RoRD and RoRD + CE),
although trained on the PhotoTourism dataset, can generalize well when
inferring on the DiverseView dataset. This suggests that our methods are
robust to changes in the data distribution due to changes in environment type
and characteristics.

TABLE II: Quantitative results for the Pose Estimation(R/t) task on the
DiverseView dataset. First and Second best values are highlighted. For the
perspective view setting, on sequences 1 and 3, RoRD performs the best with
our ensemble method RoRD + CE being the second best. On sequence 2, SIFT
obtains the best orientation (rotation) estimates with RoRD performing the
second best. For the translation estimates, RoRD and RoRD + CE perform the
best and second best respectively. For the orthographic view setting, on
sequences 1 and 2, RoRD performs the best with RoRD + CE being the second
best. For sequence 3, SIFT and RoRD perform the best and the second best.

Model | Sequence 1 | Sequence 2 | Sequence 3
---|---|---|---
Perspective View
SuperPoint+SuperGlue[2] | 58.71/1.18 | 79.03/1.47 | 49.27/4.45
D2-Net[1] | 68.76/1.02 | 80.28/1.44 | 48.33/4.55
SIFT[7] | 48.43/1.12 |  59.81/1.30 | 12.74/1.47
(Ours) RoRD | 18.12/0.42 |  61.90/1.15 | 10.17/1.38
(Ours) RoRD + CE | 20.69/0.44 | 65.55/1.21 | 10.37/1.41
Proposed Orthographic View (OV)
SuperPoint+SuperGlue[2] | 54.73/0.80 | 77.85/1.13 | 44.61/4.37
D2-Net[1] | 63.69/0.87 | 80.71/1.29 | 51.04/4.80
SIFT[7] | 13.04/0.26 | 17.84/0.34 | 5.11/0.76
(Ours) RoRD | 7.71/0.18 | 8.58/0.20 | 7.79/1.03
(Ours) RoRD + CE | 8.66/0.18 | 16.32/0.32 | 8.33/1.07

####  V-B1 Perspective View

From Table II, we observe that with perspective images as input to match local
features for challenging viewpoints, for sequences 1 and 3 RoRD performs the
best with our ensemble method RoRD + CE being a close second. For sequence 2,
we observe that SIFT [7] can obtain the best rotation estimates with RoRD
obtaining the best translation estimate. RoRD and our ensemble method RoRD +
CE obtain second-best estimates for rotation and translation respectively. We
notice that across all sequences, our method either performs best or second-
best for both rotation and translation estimates. This bolsters the fact that
for both indoor and outdoor sequences, to match local features for challenging
viewpoints we need features that are invariant to rotation.

####  V-B2 Orthographic View

We further improve upon the aforementioned results by using orthographic views
instead of perspective views. We attribute this to the fact that when using
orthographic views, the visual overlap increases and makes the task of feature
matching easier, supporting the findings of [6, 5]. However, here we also
demonstrate that the orthographic view alone is not sufficient and rotation-
robust descriptors are also required in combination to be able to fully
leverage such planar views based on performance enhancement. This is evident
from Table II that orthographic views lead to proportionally and consistently
larger performance gains for RoRD than for other existing local feature
methods. In Table II, for sequences 1 and 2, our method RoRD outperforms other
baselines. For sequence 3, we observe that SIFT has the best performance with
RoRD being the second-best. We notice in the absence of rotation-invariant
specific training regime, deep methods such as Superglue [2] and D2-Net [1]
struggle to consistently find accurate feature correspondences in such extreme
cases. On average, across all datasets and both types of views, our proposed
rotation-robust descriptors achieve superior performance.

For the DiverseView dataset, the dominant planes were assumed to be available
through manual annotation (to be released along with the the dataset). Our
initial investigation revealed that most of the existing methods suitable for
planes and surface normal extraction such as PlaneRCNN [31], Multi-task
RefineNet [32] and DenseDepth [33] catastrophically failed, which can be
attributed to their lack of generalization or untypical viewpoints. This shows
that image based 3D scene understanding methods are far from perfect.
Nevertheless, we evaluated PlaneRCNN based dominant plane extraction for
Sequence 3 and found that for the cases where plane extraction worked
correctly, orthographic-view based pose estimation performed similar to that
presented in Table II.

|  |
---|---|---
|  |
|  |
D2-Net | RoRD | RoRD + OV
Figure 5: Qualitative results from the DiverseView dataset. Each row
corresponds to a different sequence from the dataset. The top, middle and the
bottom rows correspond to Sequence-1, Sequence-2 and Sequence 3 respectively.
RoRD leverages its learnt rotation-robust features to obtain precise feature
correspondences, outperforming vanilla D2-net[1]. Incorporating orthographic
views with RoRD, (RoRD + OV) further improves performance, and outperforms
both D2-Net and RoRD for all three sequences.
---
Figure 6: Qualitative results on the Oxford Robotcar Dataset. We show the
feature correspondence results for D2-Net (top) and RoRD (bottom) when
operating on orthographic views (In these figures we backproject the obtained
correspondences to the perspective view). RoRD + OV outperforms D2-Net + OV.
---
Figure 7: Qualitative results on the HPatches Dataset. Our method RoRD
(bottom) outperforms D2-Net (top) by a significant margin.

###  V-C Visual Place recognition using local feature matching

Table III shows the quantitative results for visual place recognition using
local feature matching for forward and rear view sequences (Opposite
viewpoints) on the Oxford RobotCar Dataset, using orthographic views. We
observe that our method RoRD has the best performance as compared to the
baselines D2-Net [1], SuperPoint+SuperGlue (SP+SG) [2], SIFT (Rectified
Features) [6], and LoST [12]. LoST is the only perspective-view based global
descriptor method included in this study for the sake of completion, as it has
been established to work well for opposing viewpoints but for relatively
larger localization radii. Matching images from opposing viewpoints is
challenging due to occlusion and lack of visual overlap [12]. This can be
mitigated through local feature matching on orthographic views instead of
perspective views. This results in localization within a significantly smaller
radius of 7m, which compares to the use of a lower bound of 30m localization
radius in LoST [12] due to lack of sufficient visual overlap in perspective
views.

The poor performance of deep descriptors like SuperGlue and D2-Net here can be
attributed to a lack of rotation invariance.

Methods such as SIFT (Rectified Features) [6], although performing better than
D2-Net [1] and Superglue [2], are worse than RoRD. We also compare the
performance of our approach against Rectified Features [6], which is the
conceptually closest approach to ours that does visual place recognition based
on road surfaces. Due to the lack of open source code, we implement the
orthographic view projection of road surfaces using our IPM approach as
explained in section III-B2. Following this, as mentioned in their work, we
perform feature matching using the OpenCV implementation of SIFT [7]. They use
manual verification to decide if the retrieved database image in perspective
view is a True Positive based on the visual overlap. This approach is
susceptible to errors, thus we resort to the same automated evaluation
protocol as used for other methods in this study.

TABLE III: Quantitative results for the VPR task. We highlight First and
Second best recall values. We compare a multitude of methods for the VPR task
on the Oxford RobotCar Dataset[11]. All methods except LoST[12] operate on
proposed orthographic view (OV). Our method RoRD outperforms other baseline
methods significantly.

Method | LoST[12] | D2-Net[1] | SP+SG[2] | SIFT[6] | RoRD (Ours)
---|---|---|---|---|---
Recall | 10.30 | 22.16 | 25.47 | 34.97 | 70.31

###  V-D Qualitative Results

We show qualitative results on three sequences for the DiverseView dataset in
Figure 5. These sequences comprise both indoor and outdoor scenes. Each row
represents a particular sequence from the DiverseView dataset. We conduct a
three-way comparison between vanilla D2-Net, RoRD, and RoRD + OV (orthographic
view correspondences projected back to perspective views). We observe that on
all sequences, RoRD + OV and RoRD both outperform D2-Net. Even though the
sequences contain rich features in the form of tables, laptops, graffiti walls
and books, D2-Net fails to find precise correspondences. When operating on
orthographic views, our method (RoRD + OV) finds even more correspondences,
leading to reduced rotation and translation errors.

Figure 6 and 7 show qualitative results for the Oxford Robotcar [11] and the
HPatches dataset [10] respectively. For the task of finding precise pixel
level correspondences between opposing viewpoint sequences, our method RoRD
outperforms D2-Net where both the methods use orthographic views. As shown in
Figure 7, as a consequence of inplane-rotation training regime of RoRD, we are
able to obtain high quality feature correspondences and perform notably better
than vanilla D2-Net [1].

##  VI CONCLUSION

In this work, we present a system for creating rotation-robust features to
match local features between images under challenging viewpoint conditions.
Our method via a training pipeline uses simple geometric operations such as
in-plane rotations to achieve such features. Through these features, we show
that we are able to obtain precise feature correspondences between image pairs
despite significant viewpoint variations. We further extend the potential of
such features by introducing an ensemble approach RoRD + CE which combines the
original information learnt from D2-Net[1] and the newly learnt rotation-
robust features (RoRD). On the HPatches dataset, we show that having this
combination outperforms our other methods as well as existing baselines. For
the Oxford RobotCar and the DiverseView Datasets, we further boost the
performance of our method by using orthographic transformed images in lieu of
perspective images. We show that orthographic views alone are not sufficient
when used with existing local features, and rotation-robust descriptors are
necessary to achieve superior performance as demonstrated on pose estimation
and opposite-viewpoint place recognition tasks.

## References

  * [1] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler, “D2-Net: A Trainable CNN for Joint Detection and Description of Local Features,” in _Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [2] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperGlue: Learning feature matching with graph neural networks,” in _CVPR_ , 2020. [Online]. Available: https://arxiv.org/abs/1911.11763
  * [3] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” _CoRR_ , vol. abs/1712.07629, 2017\. [Online]. Available: http://arxiv.org/abs/1712.07629
  * [4] G. Hugo, B. Guillaume, and L. Vincent, “S2dnet: Learning accurate correspondences for sparse-to-dense feature matching,” 2020.
  * [5] S. Tourani, D. Desai, U. S. Parihar, S. Garg, R. K. Sarvadevabhatla, and K. M. Krishna, “Early bird: Loop closures from opposing viewpoints for perceptually-aliased indoor environments,” _CoRR_ , vol. abs/2010.01421, 2020\. [Online]. Available: https://arxiv.org/abs/2010.01421
  * [6] C. Toft, D. Turmukhambetov, T. Sattler, F. Kahl, and G. J. Brostow, “Single-image depth prediction makes feature matching easier,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [7] D. Lowe, “Distinctive image features from scale-invariant keypoints,” _International Journal of Computer Vision_ , vol. 60, pp. 91–, 11 2004.
  * [8] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” in _Computer Vision – ECCV 2006_ , A. Leonardis, H. Bischof, and A. Pinz, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 404–417.
  * [9] J. Revaud, P. Weinzaepfel, C. R. de Souza, N. Pion, G. Csurka, Y. Cabon, and M. Humenberger, “R2D2: repeatable and reliable detector and descriptor,” _CoRR_ , vol. abs/1906.06195, 2019. [Online]. Available: http://arxiv.org/abs/1906.06195
  * [10] V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk, “Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors,” in _CVPR_ , 2017.
  * [11] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000km: The Oxford RobotCar Dataset,” _The International Journal of Robotics Research (IJRR)_ , vol. 36, no. 1, pp. 3–15, 2017. [Online]. Available: http://dx.doi.org/10.1177/0278364916679498
  * [12] S. Garg, N. Suenderhauf, and M. Milford, “Lost? appearance-invariant place recognition for opposite viewpoints using visual semantics,” _Proceedings of Robotics: Science and Systems XIV_ , 2018.
  * [13] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “Orb: an efficient alternative to sift or surf,” 11 2011, pp. 2564–2571.
  * [14] S. Leutenegger, M. Chli, and R. Y. Siegwart, “Brisk: Binary robust invariant scalable keypoints,” in _2011 International Conference on Computer Vision_ , 2011, pp. 2548–2555.
  * [15] T. Ahonen, J. Matas, C. He, and M. Pietikäinen, “Rotation invariant image description with local binary pattern histogram fourier features,” 06 2009, pp. 61–70.
  * [16] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 24, no. 7, pp. 971–987, 2002.
  * [17] Y. Duan, J. Lu, J. Feng, and J. Zhou, “Learning rotation-invariant local binary descriptor,” _IEEE Transactions on Image Processing_ , vol. 26, no. 8, pp. 3636–3651, 2017.
  * [18] T. S. Cohen and M. Welling, “Group equivariant convolutional networks,” in _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ser. ICML’16. JMLR.org, 2016, p. 2990–2999.
  * [19] M. Weiler and G. Cesa, “General E(2)-Equivariant Steerable CNNs,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [20] J. Kim, W. Jung, H. Kim, and J. Lee, “Cycnn: A rotation invariant CNN using polar mapping and cylindrical convolution layers,” _CoRR_ , vol. abs/2007.10588, 2020. [Online]. Available: https://arxiv.org/abs/2007.10588
  * [21] K. Mani, S. Daga, S. Garg, S. S. Narasimhan, M. Krishna, and K. M. Jatavallabhula, “Monolayout: Amodal scene layout from a single image,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 1689–1697.
  * [22] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self-supervised monocular depth prediction,” October 2019.
  * [23] S. A. Abbas and A. Zisserman, “A geometric approach to obtain a bird’s eye view from an image,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_. IEEE, 2019, pp. 4095–4104.
  * [24] R. Hartley and A. Zisserman, _Multiple View Geometry in Computer Vision_ , 2nd ed. New York, NY, USA: Cambridge University Press, 2003.
  * [25] T. Bruls, H. Porav, L. Kunze, and P. Newman, “The right (angled) perspective: Improving the understanding of road scenes using boosted inverse perspective mapping,” _2019 IEEE Intelligent Vehicles Symposium (IV)_ , pp. 302–309, 2019.
  * [26] Q.-Y. Zhou, J. Park, and V. Koltun, “Open3D: A modern library for 3D data processing,” _arXiv:1801.09847_ , 2018.
  * [27] N. Snavely, S. M. Seitz, and R. Szeliski, “Photo tourism: Exploring photo collections in 3d,” in _SIGGRAPH Conference Proceedings_. New York, NY, USA: ACM Press, 2006, pp. 835–846.
  * [28] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000km: The Oxford RobotCar Dataset,” _The International Journal of Robotics Research (IJRR)_ , vol. 36, no. 1, pp. 3–15, 2017. [Online]. Available: http://dx.doi.org/10.1177/0278364916679498
  * [29] S. Das, “Simultaneous localization and mapping (SLAM) using RTAB-MAP,” _CoRR_ , vol. abs/1809.02989, 2018. [Online]. Available: http://arxiv.org/abs/1809.02989
  * [30] O. Vysotska, T. Naseer, L. Spinello, W. Burgard, and C. Stachniss, “Efficient and effective matching of image sequences under substantial appearance changes exploiting gps priors,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 2774–2779.
  * [31] C. Liu, K. Kim, J. Gu, Y. Furukawa, and J. Kautz, “Planercnn: 3d plane detection and reconstruction from a single image,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [32] V. Nekrasov, T. Dharmasiri, A. Spek, T. Drummond, C. Shen, and I. Reid, “Real-time joint semantic segmentation and depth estimation using asymmetric annotations,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 7101–7107.
  * [33] I. Alhashim and P. Wonka, “High quality monocular depth estimation via transfer learning,” _arXiv e-prints_ , vol. abs/1812.11941, 2018. [Online]. Available: https://arxiv.org/abs/1812.11941