Unleashing the Power of Depth and Pose Estimation Neural Networks by
  Designing Compatible Endoscopic Images
Deep learning models have witnessed depth and pose estimation framework on
unannotated datasets as a effective pathway to succeed in endoscopic
navigation. Most current techniques are dedicated to developing more advanced
neural networks to improve the accuracy. However, existing methods ignore the
special properties of endoscopic images, resulting in an inability to fully
unleash the power of neural networks. In this study, we conduct a detail
analysis of the properties of endoscopic images and improve the compatibility
of images and neural networks, to unleash the power of current neural networks.
First, we introcude the Mask Image Modelling (MIM) module, which inputs partial
image information instead of complete image information, allowing the network
to recover global information from partial pixel information. This enhances the
network' s ability to perceive global information and alleviates the phenomenon
of local overfitting in convolutional neural networks due to local artifacts.
Second, we propose a lightweight neural network to enhance the endoscopic
images, to explicitly improve the compatibility between images and neural
networks. Extensive experiments are conducted on the three public datasets and
one inhouse dataset, and the proposed modules improve baselines by a large
margin. Furthermore, the enhanced images we proposed, which have higher network
compatibility, can serve as an effective data augmentation method and they are
able to extract more stable feature points in traditional feature point
matching tasks and achieve outstanding performance.
Junyang Wu, Yun Gu
Unleashing the Power of Depth and Pose Estimation Neural Networks by Designing Compatible Endoscopic Images

###### Abstract

Deep learning models have witnessed depth and pose estimation framework on unannotated datasets as a effective pathway to succeed in endoscopic navigation. Most current techniques are dedicated to developing more advanced neural networks to improve the accuracy. However, existing methods ignore the special properties of endoscopic images, resulting in an inability to fully unleash the power of neural networks. In this study, we conduct a detail analysis of the properties of endoscopic images and improve the compatibility of images and neural networks, to unleash the power of current neural networks. First, we introduce the Mask Image Modelling (MIM) module, which inputs partial image information instead of complete image information, allowing the network to recover global information from partial pixel information. This enhances the network's ability to perceive global information and alleviates the phenomenon of local overfitting in convolutional neural networks due to local artifacts. Second, we propose a lightweight neural network to enhance the endoscopic images, to explicitly improve the compatibility between images and neural networks. Extensive experiments are conducted on the three public datasets and one inhouse dataset, and the proposed modules improve baselines by a large margin. Furthermore, the enhanced images we proposed, which have higher network compatibility, can serve as an effective data augmentation method and they are able to extract more stable feature points in traditional feature point matching tasks and achieve outstanding performance.

## 1 Introduction

With development of precision medicine, minimally invasive surgery has become a major development direction in medicine. Endoluminal intervention aims to reach a lesion through the body's cavity or lumen for biopsy or treatment. As a non-invasive intervention tool, endoscopes play an important role in endoluminal treatment. However, the pathways of lumen structure can be complex. How to design a suitable navigation algorithm to guide doctors to manipulate endoscopes to reach the target area is a clinical challenge.

For visual navigation, depth and pose estimation is an important challenge. The depth information of endoscopic images can measure the distance between the endoscope and the organ wall to avoid collision damage between the instrument and the patient's organs; pose information can perceive the position of the endoscope in real time and guide doctors to reach the target area according to the preoperative planned path. Moreover, robust pose and depth estimation techniques enable advanced applications like augmented reality and automated medical interventions, revolutionizing the field of endoscopy and pushing the boundaries of medical innovation.

Traditional multiview stereo method is a fascinating area of computer vision research that aims to reconstruct the three-dimensional structure of a scene from multiple images, e.g., structure from motion(SfM) [1] and simultaneous localization and mapping(SLAM) [2]. It leverage the power of multiple viewpoints to infer the depths and positions of objects in the scene. By analyzing the correspondences between points in different views and incorporating geometric constraints, traditional multiview stereo methods can estimate accurate depth maps and camera poses. However, due to the sparse and uneven distribution of feature points in endoscopy, traditional methods cannot achieve satisfactory results.

In recent years, with the development of deep learning, learning-based methods to estimate pose and depth have explored rapidly. These methods [3, 4, 5, 6, 7, 8] fully take advantage of the fundamental principles of SfM and train the depth and pose neural network by minimizing the appearance difference between consecutive target and frames.

Although many works focus on designing more advanced networks for depth and pose estimation tasks, in endoscopic scenarios, we believe that in addition to considering more sophisticated networks, it is crucial to address the question of whether the original endoscopic images are compatible with current convolution neural network. Due to the presence of artifacts and sparse feature inherent to endoscopic scenes, the original endoscopic images may introduce much noise to the neural network. In addition, within