CenterLoc3D: Monocular 3D Vehicle Localization Network for Roadside Surveillance Cameras]CenterLoc3D: Monocular 3D Vehicle Localization Network for Roadside Surveillance Cameras

1]Xinyao<EMAIL_ADDRESS>
[1]Wei<EMAIL_ADDRESS>
1]Huansheng<EMAIL_ADDRESS>
1]Chunhui<EMAIL_ADDRESS>
Monocular 3D vehicle localization is an important task for vehicle behaviour analysis, traffic flow parameter estimation and autonomous driving in Intelligent Transportation System (ITS) and Cooperative Vehicle Infrastructure System (CVIS), which is usually achieved by monocular 3D vehicle detection. However, monocular cameras cannot obtain depth information directly due to the inherent imaging mechanism, resulting in more challenging monocular 3D tasks. Currently, most of the monocular 3D vehicle detection methods still rely on 2D detectors and additional geometric constraint modules to recover 3D vehicle information, which reduces the efficiency. At the same time, most of the research is based on datasets of onboard scenes, instead of roadside perspective, which is limited in large-scale 3D perception. Therefore, we focus on 3D vehicle detection without 2D detectors in roadside scenes. We propose a 3D vehicle localization network CenterLoc3D for roadside monocular cameras, which directly predicts centroid and eight vertexes in image space, and the dimension of 3D bounding boxes without 2D detectors. To improve the precision of 3D vehicle localization, we propose a multi-scale weighted-fusion module and a loss with spatial constraints embedded in CenterLoc3D. Firstly, the transformation matrix between 2D image space and 3D world space is solved by camera calibration. Secondly, vehicle type, centroid, eight vertexes, and the dimension of 3D vehicle bounding boxes are obtained by CenterLoc3D. Finally, centroid in 3D world space can be obtained by camera calibration and CenterLoc3D for 3D vehicle localization. To the best of our knowledge, this is the first application of 3D vehicle localization for roadside monocular cameras. Hence, we also propose a benchmark for this application including a dataset (SVLD-3D), an annotation tool (LabelImg-3D), and evaluation metrics. Through experimental validation, the proposed method achieves high accuracy with $A{P_{3D}}$ of 51.30%, average 3D localization precision of 98%, average 3D dimension precision of 85% and real-time performance with FPS of 41.18.

§ INTRODUCTION

In recent years, 3D vehicle localization is becoming an essential component in CVIS and ITS. Vehicle behaviors [1, 2] and traffic flow statistics [3] can be analyzed and used for traffic state estimation [4] and traffic management [5], which is of great research significance and practical value. In practical applications, obtaining vehicle localization and dimension in 3D space is more important than vehicle information in 2D images. Different from sensors in autonomous vehicles, roadside sensors are usually installed on higher to obtain full-road vehicle information easily. In CVIS, by estimating 3D vehicle localization, vehicle locations can be obtained and sent to each vehicle for accurate path planning to avoid collisions.

The most common sensors currently used for 3D vehicle localization are lidar [6, 7, 8, 9] and stereo vision systems represented by binocular and RGB-D depth cameras [10, 11, 12]. We can obtain accurate 3D location provided by point cloud data directly. However, they are usually expensive and environmentally demanding. In contrast, monocular RGB cameras [13, 14, 15, 16, 17, 18] are cost-effective due to widespread deployment and fast data processing speed, which is more suitable for large-scale traffic scenes. Currently, monocular 3D vehicle localization is achieved by monocular 3D vehicle detection methods, which are focused on the onboard view. Differently, the roadside view is fixed with more geometric priors. At the same time, the roadside view is higher and wider than the onboard, which is more suitable for large-area perception. However, due to missing depth information, challenges like occlusion and congestion still exist in monocular roadside surveillance scenes.

Monocular 3D vehicle detection methods can be divided into the following two categories: (1) geometric constraint-based methods [19, 20, 21, 22]. (2) 3D feature estimation based methods [23, 24, 25, 26, 27, 28]. Methods of the first category are implemented based on mature 2D detectors, which use 3D bounding boxes or CAD models to represent vehicles. Then, 3D vehicle models are solved by establishing constraints of 3D bounding boxes fitting closely to 2D boxes. However, this geometric constraint is not enough for generating unambiguous results. Sub-networks will also reduce the processing efficiency. Compared with the first category, 3D vehicle information like keypoint, orientation, and dimension is directly extracted without 2D detectors. Due to missing depth information in monocular images, additional geometric inference modules are needed, which will also increase processing time to some extent.

To solve the above problems, we propose a monocular 3D vehicle localization method for surveillance cameras in traffic scenes. Firstly, the transformation matrix between 2D image space and 3D world space is solved by camera calibration, which is further used for 3D vehicle localization. Secondly, a one-stage 3D vehicle localization network CenterLoc3D is proposed, which contains three modules: backbone, multi-scale feature fusion, and multi-task detection head. In multi-scale feature fusion, we propose a weighted-fusion module to fuse five feature maps containing multi-scale information with different weights for multi-scale vehicle detection. In multi-task detection head, the multi-scale feature map is used as the input to obtain outputs. Outputs contain four branches: vehicle type, centroid, eight vertexes, and the dimension of 3D bounding boxes. To improve the precision of 3D vehicle localization without sacrificing efficiency, a loss with spatial constraints embedding is proposed, including reprojection constraint of 2D-3D transformation obtained by camera calibration and IoU constraint of 3D box projection. Finally, we also propose a benchmark including a dataset, an annotation tool, and evaluation metrics for 3D vehicle localization in roadside monocular traffic scenes. Through experimental validation, it can be proved that our method is efficient and robust.

The main contributions of this paper are summarized as follows:

* A monocular 3D vehicle localization network CenterLoc3D for roadside surveillance cameras in traffic scenes is proposed, which directly predicts accurate 3D vehicle projection vertexes and dimensions.

* A weighted-fusion module is proposed in multi-scale feature fusion, which further enhances feature extraction capability.

* A loss with spatial constraints embedding is proposed, which can effectively improve the accuracy of 3D vehicle localization.

* A benchmark including a dataset, an annotation tool, and evaluation metrics is proposed for experimental validation, which is helpful for the development of monocular 3D vehicle localization in roadside monocular traffic scenes.

§ RELATED WORK

Deep learning-based monocular 3D vehicle detection methods can be divided into the following two categories: geometric constraint-based and 3D feature estimation-based.

Geometric Constraint-Based Methods.
Due to the continuous development of convolutional neural networks (CNNs), many excellent 2D object detection methods [29, 30, 31, 32, 33] have emerged. Some methods [34, 35] directly apply the regional proposal in 2D object detection to 3D object detection. Mono3D [34] introduces sliding windows for 3D vehicle detection. Firstly, a series of candidate bounding boxes in 3D space are generated and projected to the image plane by camera calibration. Secondly, priori information like vehicle segmentation contours is combined to further obtain vehicle areas and 3D bounding boxes. The best results are finally selected by non-maximum suppression (NMS). However, the search range in 3D space is much larger than that in 2D space while obtaining priori information is time-consuming, which greatly reduces the efficiency. Currently, 2D object detection methods are still used in most 3D vehicle detection methods [13, 14, 19, 20, 21, 22] to obtain vehicle location, dimension and orientation. To improve efficiency, generating candidate boxes in 3D space is replaced by using geometric constraints of 3D bounding boxes fitting closely to 2D boxes. Deep3DBox [19] obtains 3D vehicle bounding boxes and orientation by constructing 2D-3D box constraints and multi-bin loss function. Deep MANTA [20], a multi-task network, uses 3D CAD models and region proposal network (RPN) to obtain vehicle type, 2D bounding boxes, location, visibility, and similarity to CAD model, which is used to select the best 3D CAD model. CubeSLAM [21] obtains three orthogonal vanishing points by extracting straight line segments of vehicles and constructs 3D vehicle bounding boxes by geometric constraints between vanishing points and 2D bounding boxes. GS3D [22] is able to correct 3D bounding boxes, which contains two sub-networks: 2D+O and 3D sub-network. The 2D+O sub-network adds a vehicle orientation regression branch to Faster R-CNN [29], which is used to obtain 2D vehicle bounding boxes and orientation simultaneously. With camera calibration and the above information, a coarse 3D bounding box called guidance can be obtained. The 3D sub-network is used to extract visible 3D features of the guidance and complete the correction to obtain more accurate detection results. The above methods based on geometric constraints usually require 2D bounding boxes and time-consuming priori information extraction. The constraints that 2D bounding boxes can provide are limited, which leads to ambiguous results. At the same time, sub-networks will additionally increase the processing time.

3D Feature Estimation-Based Methods.
To avoid ambiguous results from geometric constraints and further improve accuracy and efficiency, many new methods [23, 24, 25, 26, 15, 17, 18, 27, 28] focus on direct 3D information extraction in images which can be used for 3D bounding box regression by CNNs. Mono3DBox [23] proposes an end-to-end detector that can directly predict the center point of the vehicle bottom in the image. Then, the point is converted into 3D space by a look-up table to realize 3D vehicle detection and localization. MonoGRK [24] proposes an end-to-end keypoint-based 3D vehicle detection and localization network, using ResNet-101 [36] as the backbone. The detection head contains three sub-networks: 2D vehicle detection, 2D keypoint regression, and 3D vehicle dimension regression. Besides, 3D CAD models are also used like DeepMANTA [20], but only 14 keypoints are labelled for each model, which describes the CAD model coarsely. 2D-3D space is linked in the geometric inference module, which enables the model end-to-end training. Transformer3D [26] proposes a roadside 3D vehicle detection method using CNNs. Automatic camera calibration [37] is firstly used to obtain three orthogonal vanishing points and scale factors for perspective transformation. Then, the transformed image is fed into RetinaNet [32] to obtain 2D bounding boxes and conversion parameters for 3D boxes. Finally, 3D bounding boxes are recovered by camera calibration results and the conversion parameters for speed measurement. The pipeline is simple, but 3D bounding boxes are not directly evaluated in the experimental results. RTM3D [25] views 3D vehicle detection as keypoint detection without leveraging 2D detection results. The network is constructed based on ResNet-18 [36] and DLA-34 [38], which directly predicts center points and eight vertexes of 3D bounding boxes in the image. Then, vehicle location, dimension, and orientation can be solved by minimizing the reprojection error of 2D-3D bounding boxes. Lite-FPN [27] proposes a light-weight keypoint-based feature pyramid network, which contains three parts: backbone, detection head and post-processing. Top-k operation is used in the detection head to connect keypoints and regression sub-network. The attention module is added in keypoint detection loss to effectively solve the problem of mismatch between keypoint confidence and location. KM3D [28] proposes a 3D vehicle detection network that contains keypoint detection and geometric inference modules. Outputs include center point, eight vertexes, orientation, dimension, and confidence. In the network, keypoints are used to further guide vehicle orientation regression and 3D IoU strategy is used to help confidence branch training. The above method based on 3D feature estimation can improve the accuracy of 3D vehicle detection by designing a geometric inference module in the network. However, this module will increase model complexity and inference time.

Among the reviewed literature, geometric constraint-based methods leverage 2D detectors and additional geometric constraint modules to obtain 3D vehicle information, which reduces the efficiency. 3D feature estimation-based methods embed geometric inference modules into networks without 2D detectors, which increases the complexity. Moreover, almost all the reviewed research uses datasets of onboard scenes, instead of roadside perspective, which is limited in large-scale 3D perception.

In this paper, we focus on an efficient 3D vehicle detection structure for roadside surveillance cameras to directly obtain 3D vehicle information without 2D detectors and geometry inference modules, to tackle the above-mentioned research gaps and weaknesses.

§ CENTERLOC3D FOR 3D VEHICLE LOCALIZATION

§.§ Framework

The overall framework of the proposed method is shown in Figure <ref>, which consists of two components: camera calibration and 3D vehicle localization. Firstly, the transformation matrix between 2D image space and 3D world space is solved by camera calibration. Secondly, vehicle type, centroid, eight vertexes, and dimensions of 3D bounding boxes are obtained by CenterLoc3D. Finally, 3D vehicle centroids can be obtained by camera calibration for 3D vehicle localization.

=0pt =0pt plus 0cm The overall framework of the proposed method. The method consists of two parts: camera calibration and 3D vehicle localization. A single image is the input of CenterLoc3D to obtain vehicle type, centroid, eight vertexes, and dimensions of 3D bounding boxes. Combined with camera calibration, 3D vehicle localization results can be further obtained.

§.§ Camera Calibration

To complete 3D vehicle localization in traffic scenes, the transformation matrix between 2D image space and 3D world space must be derived through camera calibration. We refer to the previous work [39, 40] to define coordinate systems, establish the camera calibration model, and choose the single vanishing point-based calibration method VWL (One Vanishing Point, Known Width and Length) [39] to complete camera calibration.

=0pt =0pt plus 0cm Schematic diagram of coordinate systems and camera calibration model. The world coordinate system is defined as ${O_w} - x,{\rm{ }}y,{\rm{ }}z$, camera coordinate system ${O_c} - {x_c},{\rm{ }}{y_c},{\rm{ }}{z_c}$, image coordinate system ${O_i} - u,{\rm{ }}v$. Parameters of calibration model include camera focal length $f$, camera height from the ground $h$, camera tilt angle $\phi$ and camera pan angle $\theta$.

Schematic diagram of coordinate systems and camera calibration model is shown in Figure <ref>. Three coordinate systems are defined, all of which are right-handed. The world coordinate system is defined by $x,{\rm{ }}y,{\rm{ }}z$ axis. The origin ${O_w}$ is located at the projection point of the camera on the road plane, whereas $z$ is perpendicular to the road plane upwards. The camera coordinate system is defined by ${x_{\rm{c}}},{\rm{ }}{y_{\rm{c}}},{\rm{ }}{z_{\rm{c}}}$ axis. The origin ${O_c}$ is located at the camera optical center. ${x_{\rm{c}}}$ is parallel to $x$. ${z_{\rm{c}}}$ points to the ground along the camera optical axis. ${y_{\rm{c}}}$ is perpendicular to the plane ${x_c}{O_c}{z_c}$. The image coordinate system is defined by $u,{\rm{ }}v$ axis. The origin ${O_i}$ is located at image center. In the image coordinate system, $u$ is horizontal right and $v$ is vertical downward. ${z_{\rm{c}}}$ intersects the road plane at $r = ({c_x},{c_y})$ in the image coordinate system, which is called the principal point and the default location is at image center. ${c_x}$ and ${c_y}$ represent half of the image width and height, respectively.

In addition to the above parameters, another parameter is the roll angle, which can be expressed by a simple image rotation and has no effect on calibration results. Therefore, it is not considered. In this paper, VWL and road marks [40] are used to solve and optimize calibration parameters $f,h,\phi ,\theta $. The vanishing point $VP = ({u_0},{v_0})$ along the direction of traffic flow is extracted by road edge lines. To illustrate the road space in a straightforward way, ${O_i}$ and $y$ axis are adjusted. Firstly, ${O_i}$ is moved to the upper left corner of the image, corresponding to the intrinsic parameter matrix $K$:

\begin{equation}
\label{equation_K}
K = \left[ {\begin{array}{*{20}{c}}
\end{array}} \right]
\end{equation}

Then, $y$ axis is adjusted to the direction along traffic flow. The rotation matrix $R$ contains a rotation of $\phi  + {\pi  \mathord{\left/
{\vphantom {\pi  2}} \right.
\kern-\nulldelimiterspace} 2}$ about $x$ axis and $\theta $ about $z$ axis, which can be defined as:

\begin{equation}
\label{equation_R}
\begin{aligned}
R &= {R_x}(\phi  + {\pi  \mathord{\left/
{\vphantom {\pi  2}} \right.
\kern-\nulldelimiterspace} 2}){R_z}(\theta )\\
&= \left[ {\begin{array}{*{20}{r}}
{\cos \theta }&{ - \sin \theta }&0\\
{ - \sin \phi \sin \theta }&{ - \sin \phi \cos \theta }&{ - \cos \phi }\\
{\cos \phi \sin \theta }&{\cos \phi \cos \theta }&{ - \sin \phi }
\end{array}} \right]
\end{aligned}
\end{equation}

The translation matrix $T$ is:

\begin{equation}
\label{equation_T}
T = \left[ {\begin{array}{*{20}{c}}
0&0&1&{ - h}
\end{array}} \right]
\end{equation}

The adjusted transformation formula from world point $(x,y,z)$ to image point $(u,v)$ is:

\begin{equation}
\label{equation_H}
s\left[ {\begin{array}{*{20}{c}}
\end{array}} \right] = KRT\left[ {\begin{array}{*{20}{c}}
\end{array}} \right] = H\left[ {\begin{array}{*{20}{c}}
\end{array}} \right]
\end{equation}

where $H = \left[ {{h_{ij}}} \right],i = 1,2,3;j = 1,2,3,4$ is the $3 \times 4$ projection matrix from world to image coordinate. $s$ is the scale factor.

Finally, according to the above derivation, the adjusted transformation between world and image can be described as follows:

\begin{equation}
\label{equa_xyz2uv}
\left\{ \begin{array}{l}
u = \frac{{{h_{11}}x + {h_{12}}y + {h_{13}}z + {h_{14}}}}{{{h_{31}}x + {h_{32}}y + {h_{33}}z + {h_{34}}}}\\
v = \frac{{{h_{21}}x + {h_{22}}y + {h_{23}}z + {h_{24}}}}{{{h_{31}}x + {h_{32}}y + {h_{33}}z + {h_{34}}}}
\end{array} \right.
\end{equation}

\begin{equation}
\label{equa_uv2xyz}
\left\{ \begin{array}{l}
x = \frac{{{b_1}({h_{22}} - {h_{32}}v) - {b_2}({h_{12}} - {h_{32}}u)}}{{({h_{11}} - {h_{31}}u)({h_{22}} - {h_{32}}v) - ({h_{12}} - {h_{32}}u)({h_{21}} - {h_{31}}v)}}\\
y = \frac{{ - {b_1}({h_{21}} - {h_{31}}v) + {b_2}({h_{11}} - {h_{31}}u)}}{{({h_{11}} - {h_{31}}u)({h_{22}} - {h_{32}}v) - ({h_{12}} - {h_{32}}u)({h_{21}} - {h_{31}}v)}}
\end{array} \right.
\end{equation}

where $\left\{ \begin{array}{l}
{b_1} = u({h_{33}}z + {h_{34}}) - ({h_{13}}z + {h_{14}})\\
{b_2} = v({h_{33}}z + {h_{34}}) - ({h_{23}}z + {h_{24}})
\end{array} \right.$.

§.§ CenterLoc3D

In the roadside camera view, 3D vehicle localization can be described in terms of 3D vehicle detection. We propose a 3D vehicle localization network CenterLoc3D for roadside cameras, which uses a single RGB image as the input and the outputs include vehicle type, centroid, vertexes, and dimensions of 3D bounding boxes. Combined with camera calibration in Section <ref>, 3D vehicle localization results can be calculated.

§.§.§ Network Architecture

The overall architecture of the network is shown in Figure <ref>, which contains three parts: backbone, multi-scale feature fusion and multi-task detection head. Firstly, the RGB image is scaled to $I \in { \mathbb{R} ^{H \times W \times 3}}$ as input, where $H = W = 512$. Then, the image is downsampled with a factor of $S = 4$.

=0pt =0pt plus 0cm The architecture of CenterLoc3D. The network mainly consists of three parts: backbone, multi-scale feature fusion, and multi-task detection head. Multi-scale weighted-fusion feature map is obtained in the first two parts. 3D bounding box regression includes vehicle type, centroid, eight vertexes, and dimensions, which are decoded to final outputs.

Backbone. To make a trade-off between accuracy and efficiency, we use ResNet-50 [36] as our backbone, containing residual and inverted bottleneck structures, which have good feature extraction capability. We extract the last three features with channels of 512, 1024, and 2048 for feature fusion.

Multi-Scale Feature Fusion. The series of feature maps obtained by the backbone are hierarchical. The high-resolution feature map retains more accurate local features and is suitable for small object detection. The low-resolution feature map contains higher-level semantic information and is suitable for large object detection. In monocular roadside traffic scenes, vehicles are distributed in different locations of different sizes. To make the network adaptive to vehicles of different sizes, a weighted-fusion module is proposed in multi-scale feature fusion based on RetinaNet [32]. Figure <ref> shows the schematic diagram of weighted-fusion module. The feature maps ${C_3},{C_4},{C_5}$ of size $64 \times 64$, $32 \times 32$, and $16 \times 16$ are extracted by backbone and used to construct feature pyramid ${F_p} = \left\{ {{P_3},{P_4},{P_5},{P_6},{P_7}} \right\}$. Different from YOLOv4 [31], which directly retains the feature pyramid, we use deconvolution and unsample [41] to unify the five feature maps of different sizes in ${F_p}$ to the same size $\overline {{F_p}}  = \left\{ {\overline {{P_3}} ,\overline {{P_4}} ,\overline {{P_5}} ,\overline {{P_6}} ,\overline {{P_7}} } \right\}$. Then, weighted-fusion module is added by the fusion strategy shown in Equation <ref>. Finally, a fused feature map $F \in {\mathbb{R}^{\frac{H}{S} \times \frac{W}{S} \times 64}}$ containing multi-scale information of vehicles is obtained. This feature fusion module not only generates multi-scale feature maps, but also reduces computational effort in the prediction process and improve the overall efficiency.

\begin{equation}
\label{equa_feature_fusion}
F = \sum\limits_{i = 3}^7 {{w_i} \times \overline {{P_i}} }
\end{equation}

where ${w_i}$ denotes the weight of the feature map with $\sum\limits_{i = 3}^7 {{w_i}}  = 1$, which can be set according to the importance degree. We set 0.5, 0.2, 0.1, 0.1 and 0.1 respectively in the experiment.

Schematic diagram of weighted-fusion module.

Multi-Task Detection Head. Based on the multi-scale fusion feature map, a multi-task detection head is designed based on the actual demand for 3D vehicle localization in roadside scenes. The detection head is comprised of four branches: vehicle type, centroid, eight vertexes, and dimensions of 3D bounding boxes. To further improve the ability to distinguish different types of vehicles, the type branch is implemented by fully convolutional layers and attention module [42]. The remaining branches are implemented by fully convolutional layers only. Inspired by CenterNet [33], vehicle type is defined as centroid heatmap ${M_c} \in {[0,1]^{\frac{H}{S} \times \frac{W}{S} \times C}}$, where $C$ denotes the number of vehicle types. The remaining branches are defined as centroid offset ${M_{co}} \in {[0,1]^{\frac{H}{S} \times \frac{W}{S} \times 2}}$, vertexes ${M_v} \in {\mathbb{R}^{\frac{H}{S} \times \frac{W}{S} \times 16}}$ and dimension ${M_s} \in {\mathbb{R}^{\frac{H}{S} \times \frac{W}{S} \times 3}}$. To ensure the stability during training, we normalize centroids and vertexes to the fusion feature map with size ${H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S},{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}$.

§.§.§ Loss Function

In the training process, the loss function contains six components: vehicle classification, centroid offset, vertexes, dimension, and a loss with spatial constraints embedding for improving the precision of 3D vehicle localization. The spatial constraints include reprojection constraint of 2D-3D transformation obtained by camera calibration and IoU constraint of 3D box projection.

Vehicle Classification Loss. To solve the problem of imbalanced positive and negative samples, we use focal loss [32] as vehicle classification loss:

\begin{equation}
\label{equa_focal_loss}
{L_c} =  - \frac{1}{N}\sum\limits_{k = 1}^C {\sum\limits_{i = 1}^{{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}} {\sum\limits_{j = 1}^{{H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S}} {\left\{ {\begin{array}{*{20}{c}}
{{{(1 - {{\hat p}_{cij}})}^\alpha }\log ({{\hat p}_{cij}})}&{if{\rm{ }}{p_{cij}} = 1}\\
{\hat p_{cij}^\alpha {{(1 - {p_{cij}})}^\beta }\log (1 - {{\hat p}_{cij}})}&{if{\rm{ }}{p_{cij}} < 1}
\end{array}} \right.} } }
\end{equation}

where $N$ is the number of positive samples, $\alpha $ and $\beta $ are hyper-parameters used to adjust loss weights of positive and negative samples, which are usually set to 2 and 4. ${p_{cij}}$ is the response value of each ground truth vehicle in the heatmap described by the Gaussian kernel function ${e^{ - \frac{{{{(x - {p_{cijx}})}^2} + {{(y - {p_{cijy}})}^2}}}{{2{\sigma ^2}}}}}$. $\sigma $ is the standard deviation calculated from the ground truth vehicle dimension in image space [33].

3D Vehicle Information Regression Loss. We use L1 regression loss for vehicle centroid offset, vertex, and dimension loss:

\begin{equation}
\label{equa_lco}
{L_{co}} = \frac{1}{N}\sum\limits_{i = 1}^{{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}} {\sum\limits_{j = 1}^{{H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S}} {\mathbbm{1}_{ij}^{obj}\lvert {M_{co}^{ij} - ({{{p_{center}}} \mathord{\left/
{\vphantom {{{p_{center}}} S}} \right.
\kern-\nulldelimiterspace} S} - {{\tilde p}_{center}})} \rvert} }
\end{equation}

\begin{equation}
\label{equa_lv}
{L_v} = \frac{1}{N}\sum\limits_{i = 1}^{{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}} {\sum\limits_{j = 1}^{{H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S}} {\mathbbm{1}_{ij}^{obj}\lvert {M_v^{ij} - {{{p_{vertex}}} \mathord{\left/
{\vphantom {{{p_{vertex}}} S}} \right.
\kern-\nulldelimiterspace} S}} \rvert} }
\end{equation}

\begin{equation}
\label{equa_ls}
{L_s} = \frac{1}{N}\sum\limits_{i = 1}^{{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}} {\sum\limits_{j = 1}^{{H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S}} {\mathbbm{1}_{ij}^{obj}\lvert {M_s^{ij} - \tilde M_s^{ij}} \rvert} }
\end{equation}

where $\mathbbm{1}_{ij}^{obj}$ denotes whether centroid appears at $i,j$. ${p_{center}}$ and ${p_{vertex}}$ denote the ground truth centroids and vertexes of 3D bounding boxes in input image with size $H,W$. $\tilde p_{center}^{}$ denotes the ground truth centroids in the fusion feature map with size ${H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S},{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}$. ${\tilde M_s}$ denotes the ground truth feature map of 3D vehicle dimension.

Loss with spatial constraints embedding. To further improve the precision of 3D vehicle localization, loss functions with spatial constraints embedding are designed, including reprojection constraints using camera calibration and vehicle IoU constraints.

Reprojection constraint

IoU constraint

Schematic diagram of spatial constraints.

The schematic diagram of reprojection constraint and vehicle IoU constraint is shown in Figure <ref>. We define the predicted and ground truth vertexes in image as $p_i^{pred} = (u_i^{pred},v_i^{pred})$ and $p_i^{gt} = (u_i^{gt},v_i^{gt})$. The corresponding vertexes in world are $P_i^{pred} = (x_i^{pred},y_i^{pred},z_i^{pred})$ and $P_i^{gt} = (x_i^{gt},y_i^{gt},z_i^{gt})$. The projection of predicted vertexes in world and image are defined as $P_i^{proj} = (x_i^{proj},y_i^{proj},z_i^{proj})$ and $p_i^{proj} = (u_i^{proj},v_i^{proj})$, $i = 1,2, \cdots ,8$. The predicted dimension is $D_v^{pred} = (l_v^{pred},w_v^{pred},h_v^{pred})$. Based on $p_i^{pred}$ from CenterLoc3D, $P_i^{proj}$ can be obtained by Equation <ref> and Table <ref>. Then, $p_i^{proj}$ can be obtained by Equation <ref>. As shown in Figure <ref>, $p_i^{proj}$ and $p_i^{pred}$ constitute the projection constraints, where predicted, ground truth, and projection boxes are represented in blue, red, and green, respectively.

The minimum external rectangles of predicted and ground truth vertexes are $v_{rec}^{pred}$ and $v_{rec}^{gt}$. As shown in Figure <ref>, blue indicates the predicted box while red is the ground truth. The green area indicates the overlap between the two boxes.

Calculation of 3D bounding box projection vertexes in world space.

Vertex     World coordinate

$P_1^{proj}$    $(x_2^{gt} + w_v^{pred},y_2^{gt},z_2^{gt})$

$P_2^{proj}$    $(x_2^{gt},y_2^{gt},z_2^{gt})$

$P_3^{proj}$    $(x_2^{gt},y_2^{gt} + l_v^{pred},z_2^{gt})$

$P_4^{proj}$    $(x_2^{gt} + w_v^{pred},y_2^{gt} + l_v^{pred},z_2^{gt})$

$P_5^{proj}$    $(x_2^{gt} + w_v^{pred},y_2^{gt},z_2^{gt} + h_v^{pred})$

$P_6^{proj}$    $(x_2^{gt},y_2^{gt},z_2^{gt} + h_v^{pred})$

$P_7^{proj}$    $(x_2^{gt},y_2^{gt} + l_v^{pred},z_2^{gt} + h_v^{pred})$

$P_8^{proj}$    $(x_2^{gt} + w_v^{pred},y_2^{gt} + l_v^{pred},z_2^{gt} + h_v^{pred})$

The loss comprised of two constraints shown in Figure <ref> is defined as follows:

\begin{equation}
\label{equa_lproj}
{L_{proj}} = \frac{1}{N}\sum\limits_{i = 1}^{{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}} {\sum\limits_{j = 1}^{{H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S}} {\mathbbm{1}_{ij}^{obj}\lvert {M_{proj}^{ij}(H,M_s^{ij}) - {\bar M_v^{ij}} } \rvert} }
\end{equation}

\begin{equation}
\label{equa_liou}
{L_{iou}} = \frac{1}{N}\sum\limits_{i = 1}^{{W \mathord{\left/
{\vphantom {W S}} \right.
\kern-\nulldelimiterspace} S}} {\sum\limits_{j = 1}^{{H \mathord{\left/
{\vphantom {H S}} \right.
\kern-\nulldelimiterspace} S}} {\mathbbm{1}_{ij}^{obj} \cdot IoU(R_{bbox}^{ij} - \tilde R_{bbox}^{ij})} }
\end{equation}

where $M_{proj}^{}(H,M_s^{ij})$ denotes the feature map of projection vertexes calculated by camera calibration matrix $H$ in Section <ref> and vehicle dimension feature map ${M_s}$ predicted by the network. $\bar {M_v^{}} $ denotes predicted 3D bounding boxes vertexes in the original image. $R_{bbox}^{ij}$ and $\tilde R_{bbox}^{ij}$ denote the predicted and ground truth minimum external rectangles. $IoU$ denotes the IoU loss strategy and we use CIoU loss [43] in the experiment.

With the above six loss functions, the multi-task loss function can be defined as follows:

\begin{equation}
\label{equa_lossfunc}
L = {\lambda _c}{L_c} + {\lambda _{co}}{L_{co}} + {\lambda _v}{L_v} + {\lambda _s}{L_s} + {\lambda _{proj}}{L_{proj}} + {\lambda _{iou}}{L_{iou}}
\end{equation}

where $\lambda $ is the weight to balance the loss of each component. We set ${\lambda _c} = 1$, ${\lambda _{co}} = 1$ , ${\lambda _v} = 0.1$ , ${\lambda _s} = 0.1$ , ${\lambda _{proj}} = 0.1$ and ${\lambda _{iou}} = 1$ in the experiment.

§ DATASET OF 3D VEHICLE LOCALIZATION

Most currently available 3D vehicle localization datasets are based on onboard views [44, 45, 46] instead of roadside, which is difficult for large-scale 3D perception and validation of roadside vehicle perception methods. Therefore, we propose a 3D vehicle localization dataset (SVLD-3D) for roadside surveillance cameras and an annotation tool (LabelImg-3D) for experimental validation.

§.§ Dataset Composition

Scene A

Scene B

Scene C

=0pt =0pt plus 0cm Samples in SVLD-3D. Five different scenes are listed from the first to the fifth row.

Scenes in SVLD-3D dataset are from BrnoCompSpeed [47] and self-collected urban scenes with resolution of ${\rm{1920}} \times {\rm{1080}}$ and ${\rm{1080}} \times {\rm{720}}$, respectively. BrnoCompSpeed is public and provided by Brno University of Technology, which contains six highway scenes with three views (left, center, and right). Each frame contains ground truth road marks and vanishing points (used for camera calibration (section <ref>)). Each vehicle contains ground truth location and speed collected by Lidar and GPS (used for vehicle dimension annotation (section <ref>) and training loss function (section <ref>)). However, only highway scenes with low traffic volumes and single-vehicle types exist in BrnoCompSpeed. To further expand dataset diversity, urban scenes with more vehicle types, congestion, and occlusion are also included in SVLD-3D.

SVLD-3D contains five typical scenes with three vehicle types (car, truck, and bus), with a total of 14593 images in the training and validation dataset and 2273 images in the test dataset. Some samples in SVLD-3D are shown in Figure <ref>. Table <ref> shows detailed information of different scenes in the dataset, where the effective field of view ${D_r} = ({D_{ry}},{D_{rx}})$ is the maximum distance that the roadside camera can perceive along and perpendicular to the road direction.

Details of different scenes in SVLD-3D.

2*Scene     2*${D_{ry}}$     2*${D_{rx}}$     4cCamera Calibration Parameters
$f$          $\phi /rad$      $\theta /rad$      $h/mm$
A                          120                             25                              2878.13      0.17874          0.26604            10119.08

B                          120                             25                              3994.17      0.15717          0.35346            8071.00

C                          60                              15                              3384.25      0.26295          -0.24869            8126.49

D                          80                              10                              3743.78      0.11225          -0.07516            7353.40

E                          60                              10                              1142.26      0.33372          0.14387            7166.44

§.§ Label Process

Since the original data in SVLD-3D only contains images and 3D vehicle location provided by Lidar, we develop an annotation tool LabelImg-3D and relabel vehicles to obtain ground truth vehicle type, centroid, vertexes, and dimensions of 3D bounding boxes. The ground truth 3D and 2D vehicle centroid, and dimension are denoted as $P_{cen}^{gt} = (x_{cen}^{gt},y_{cen}^{gt},z_{cen}^{gt})$, $p_{cen}^{gt} = (u_{cen}^{gt},v_{cen}^{gt})$ and $D_v^{gt} = (l_v^{gt},w_v^{gt},h_v^{gt})$.

Compared with variable views of autonomous vehicles, the roadside camera usually has certain installation standards and the pan angle is relatively fixed. Therefore, only the scenes with typical pan angles are selected. To reduce label effort, previous work [48] and 2D bounding boxes provided by YOLOv4 [31] are used as guidance. The specific label steps are as follows:

* Vehicle type. If the vehicle type obtained by YOLOv4 is consistent with the ground truth, label this type as ground truth. Otherwise, adjust to the correct type in LabelImg-3D.

* Vehicle dimension. Firstly, YOLOv4 is used to obtain 2D vehicle bounding boxes as guidance. Secondly, the geometric constraint of 3D boxes fitting closely to 2D is used to adjust vehicle dimension by camera calibration. At the same time, vehicle dimensions can also be obtained by observing vehicle models and referring to relevant documents during the labeling process. These two sub-steps can be helpful to obtain more accurate vehicle dimensions $D_v^{gt}$.

* Vehicle centroid. $p_{cen}^{gt}$ can be obtained by Equation <ref> when $P_{cen}^{gt}$ is known by lidar.

* Vehicle vertexes. Based on $P_{cen}^{gt}$ and $D_v^{gt}$ in step (2), $p_i^{gt}$ can be obtained by Table <ref> and Equation <ref>.

Calculation of 3D bounding box ground truth vertexes in world space.

Vertex     World coordinate

$P_1^{gt}$    $(x_{cen}^{gt} + {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} - {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} - {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

$P_2^{gt}$    $(x_{cen}^{gt} - {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} - {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} - {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

$P_3^{gt}$    $(x_{cen}^{gt} - {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} + {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} - {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

$P_4^{gt}$    $(x_{cen}^{gt} + {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} + {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} - {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

$P_5^{gt}$    $(x_{cen}^{gt} + {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} - {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} + {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

$P_6^{gt}$    $(x_{cen}^{gt} - {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} - {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} + {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

$P_7^{gt}$    $(x_{cen}^{gt} - {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} + {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} + {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

$P_8^{gt}$    $(x_{cen}^{gt} + {{w_v^{gt}} \mathord{\left/
{\vphantom {{w_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},y_{cen}^{gt} + {{l_v^{gt}} \mathord{\left/
{\vphantom {{l_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2},z_{cen}^{gt} + {{h_v^{gt}} \mathord{\left/
{\vphantom {{h_v^{gt}} 2}} \right.
\kern-\nulldelimiterspace} 2})$

Finally, vehicle type, centroid, vertexes, and dimensions are recorded in the annotation file. Figure <ref> shows the flowchart of label process and visualization of annotations.

Flowchart of label process

Visualization of annotations

Label process and visualization.

§ EXPERIMENTAL PROTOCOLS

In this section, we introduce implementation details and evaluation metrics for our experiments.

§.§ Implementation Details

We implement our network using PyTorch platform with Core i7-8700 CPU and one GTX 1080Ti GPU. The original image is scaled to $512 \times 512$ for training and testing. We split the dataset into the training set and validation set with a ratio of 9:1. We use Adam optimizer with a base learning rate of 0.001 for 100 epochs. The learning rate reduces by a factor of 10 when validation loss no longer decreases for three continuous epochs. The pretrained model on ImageNet is used for fine-tuning. We train our network using weights freezing in the first 60 epochs with a batch size of 16. The batch size drops $2 \times$ in the rest epochs. When validation loss no longer decreases for 7 continuous epochs, the training process will be stopped by early stopping.

We use random color jitter, horizontal flip, and perspective transformation as image augmentation. As shown in Figure <ref>, perspective transformation is used as the simulation of roadside view change according to the camera imaging principle. After data augmentation, the training and validation set can be expanded to 58372 images, which quadruples the original volume.

=0pt =0pt plus 0cm Schematic diagram of data augmentation. (a) No augmentation. (b) Color jitter (CJ). (c) Horizontal flip (HF). (d) Perspective transformation (PT). (e) CJ + HF. (f) CJ + PT. (g) HF + PT. (h) CJ + HF + PT.

§.§ Evaluation Metrics

Evaluation metrics include average precision (AP), frame per second (FPS), precision and error of 3D vehicle localization and dimension prediction.

§.§.§ Average Precision and Speed

Referring to evaluation metrics of existing 3D vehicle detection datasets, we use $A{P_{3D}}$ [34] to evaluate 3D average precision and FPS to evaluate speed.

$A{P_{3D}}$ is similar to $A{P_{2D}}$ in 2D detection provided by VOC dataset [49]. Differently, 3D IoU is used instead of 2D in calculating $A{P_{3D}}$. The equation of $AP$ can be expressed as:

\begin{equation}
\label{equa_ap}
\begin{array}{l}
AP = \frac{1}{{11}}\sum\limits_{r \in \{ 0,0.1, \ldots ,1\} } {{p_{{\rm{interp}}}}(r)} \\
{p_{{\rm{interp}}}}(r) = \mathop {\max }\limits_{\tilde r:\tilde r \ge r} p(\tilde r)
\end{array}
\end{equation}

The precision at each recall level $r$ is interpolated by taking the maximum precision measured for a method for which the corresponding recall exceeds $r$, $p(\tilde r)$ is the measured precision at recall $\tilde r$.

The equation of FPS is defined as:

\begin{equation}
\label{equa_fps}
FPS = {1 \mathord{\left/
{\vphantom {1 {{t_{proc}}}}} \right.
\kern-\nulldelimiterspace} {{t_{proc}}}}
\end{equation}

where $t_{proc}$ represents the processing time (measured in second) of the network for a single frame.

§.§.§ 3D Vehicle Localization Precision and Error

Combined with camera calibration in Section <ref>, 3D vehicle centroids can be further obtained by Equation <ref> and used for 3D vehicle localization.

The predicted and ground-truth 3D vehicle centroid are denoted as $P_{cen}^{pred} = (x_{cen}^{pred},y_{cen}^{pred},z_{cen}^{pred})$ and $P_{cen}^{gt} = (x_{cen}^{gt},y_{cen}^{gt},z_{cen}^{gt})$.

3D vehicle localization precision and error can be defined as follows:

\begin{equation}
\label{equa_ploc}
{P_{loc}} = (1 - \sum\limits_{k \in \{ x,y\} } {\frac{{\lvert {k_{cen}^{pred} - k_{cen}^{gt}} \rvert}}{{{{{D_{rk}}} \mathord{\left/
{\vphantom {{{D_{rk}}} 2}} \right.
\kern-\nulldelimiterspace} 2}}}} ) \times 100\%
\end{equation}

\begin{equation}
\label{equa_eloc}
{E_{loc}} = \sum\limits_{k \in \{ x,y\} } {\lvert {k_{cen}^{pred} - k_{cen}^{gt}} \rvert}
\end{equation}

where ${D_r}$ can be found in Table <ref>.

§.§.§ 3D Vehicle Dimension Precision and Error

We design a 3D vehicle dimension regression branch in the network, which can be used for 3D vehicle dimension prediction, including length, width, and height in meters.

The predicted and ground-truth 3D dimension are denoted as $D_v^{pred} = (l_v^{pred},w_v^{pred},h_v^{pred})$ and $D_v^{gt} = (l_v^{gt},w_v^{gt},h_v^{gt})$.

3D vehicle dimension precision and error can be defined as follows:

\begin{equation}
\label{equa_pdim}
{P_{dim}} = (1 - \sum\limits_{k \in \{ {l_v},{w_v},{h_w}\} } {\frac{{\lvert {{k^{pred}} - {k^{gt}}} \rvert}}{{{k^{gt}}}}} ) \times 100\%
\end{equation}

\begin{equation}
\label{equa_edim}
{E_{dim}} = \sum\limits_{k \in \{ {l_v},{w_v},{h_w}\} } {\lvert {{k^{pred}} - {k^{gt}}} \rvert}
\end{equation}

§ RESULTS AND DISCUSSIONS

In this section, we provide experimental results, an ablation study and discussions to demonstrate the effectiveness of CenterLoc3D.

§.§ Average Precision and Speed of CenterLoc3D

$A{P_{3D}}$ and FPS of different monocular 3D vehicle detection methods on validation and test set are compared in Table <ref>, which are calculated by Equation <ref> and Equation <ref>. In addition, FLOPS and parameters are also important metrics, with 28.61GFlops and 34.95M for our network. KITTI validation and test set are used in onboard scenes at easy, moderate, and hard settings, which are determined by vehicle size in image space. We use the proposed SVLD-3D dataset for experimental validation. The IoU thresholds are 0.5 and 0.7 respectively. SVLD-3D dataset has only one validation set without difficulty settings.

Comparison of $A{P_{3D}}$ and FPS of different monocular 3D vehicle detection methods.

2*Method      2*Scene        2*Backbone     2*GPU              3cc]@c@$A{P_{3D}}(IOU > 0.5)$
$[va{l_1}/va{l_2}]$      3cc]@c@$A{P_{3D}}(IOU > 0.7)$
$[va{l_1}/va{l_2}/test]$     2*FPS
Easy                                             Moderate                                         Hard                                             Easy                                               Moderate                                           Hard

MonoGRNet [13]                    onboard                       VGG-16                        GTX Titan X                       50.51/54.21                                      36.97/39.69                                      30.82/33.06                                      13.88 / 24.97 / -                                  10.19 / 19.44 / -                                  7.62 / 16.30 / -                                  16.7

Deep3DBox [19]                    onboard                       VGG-16                        -                                 27.04 / -                                        20.55 / -                                        15.88 / -                                        5.85 / - / -                                       4.10 / - / -                                       3.84 / - / -                                      -

GS3D [22]                        onboard                       VGG-16                        -                                 32.15/30.60                                      29.89/26.40                                      26.19/22.89                                      13.46/11.63/7.69                                   10.97/10.51/6.29                                   10.38/10.51/6.16                                  0.4

2*RTM3D [25]       2*onboard      ResNet-18                      2*GTX 1080Ti×2     47.43/46.52                                      33.86/32.61                                      31.04/30.95                                      18.13 / 18.38 / -                                  14.14 / 14.66 / -                                  13.33 / 12.35 / -                                 28.6

DLA-34                                                           54.36/52.59                                      41.90/40.96                                      35.84/34.95                                      20.77/19.47/13.61                                  16.86/16.29/10.09                                  16.63/15.57/8.18                                  18.2

SMOKE [15]                       onboard                       DLA-34                         GTX TITAN X×4                     3c-                                                                                14.76/19.99/14.03                                  12.85/15.61/9.76                                   11.50/15.28/7.84                                  33.3

2*KM3D [28]        2*onboard      ResNet-18                      2*GTX 1080Ti       47.23/47.13                                      34.12/33.31                                      31.51/25.84                                      19.48/18.34/12.65                                  15.32/14.91/8.39                                   13.88/12.58/7.12                                  47.6

DLA-34                                                           56.02/54.09                                      43.13/43.07                                      36.77/37.56                                      22.50/22.71/16.73                                  19.60/17.71/11.45                                  17.12/16.15/9.92                                  25.0

3*Lite-FPN [27]     3*onboard      ResNet-18                      3*GTX 2080Ti       3c3*-                                                                                17.04 / - / -                                      14.02 / - / -                                      12.23 / - / -                                     88.57

ResNet-34                                                        3c                                                                                18.01 / - / 15.32                                  15.29 / - / 10.64                                  14.28 / - / 8.59                                  71.32

DLA-34                                                           3c                                                                                19.31 / - / -                                      16.19 / - / -                                      15.47 / - / -                                     42.37

Ours        roadside     ResNet-50                      GTX 1080Ti       3c91.34 / -                                                                                3c79.36 / - / 51.30                                                                                41.18

Figure <ref> illustrates visualization results on SVLD-3D test set. Different views and types of vehicles in SVLD-3D test set are tested, with occlusion of environment and other vehicles. Vehicles are widely distributed in the scene. From Table <ref> and Figure <ref>, detected vehicles all response with red circle in the heatmaps. It can be seen that our network achieves real-time 3D vehicle detection and is adaptive to occlusion and small vehicles.

Scene A

Scene B

Scene C

Scene D

Scene E

=0pt =0pt plus 0cm Visualization results on SVLD-3D test set. 3D vehicle detection results (left) are represented by color bounding boxes. Centroids of heatmaps (right) are represented in red circles.

3D vehicle information which can be obtained by different methods is compared in Table <ref>. It can be seen that our method obtains not only 3D bounding boxes but also 3D centroids, dimensions, and 3D locations, with real-time performance.

Comparison of 3D vehicle information obtained by different methods.

Method                                              BBox                       Centroid                   Dimension                  Location

MonoGRNet [13]

Deep3DBox [19]

GS3D [22]

RTM3D [25]

SMOKE [15]

KM3D [28]

Lite-FPN [27]

§.§ 3D Vehicle Localization Precision and Error of CenterLoc3D

Table <ref> and Figure <ref> show 3D vehicle localization results and precision of different scenes and types in SVLD-3D test set, which is calculated by Equation <ref>. It can be seen that our network can also achieve good results in 3D vehicle localization, with an average precision of 98%.

Top views of 3D vehicle localization of different frames in SVLD-3D test set are shown in Figure <ref>, which also shows that our network has high precision in 3D vehicle localization. Each sub-figure in Figure <ref> represents different frames in different scenes. At the same time, vehicles far from the roadside camera can also be detected and localized.

3D vehicle localization results and precision on SVLD-3D test set.

Vehicle     Type     ${P_{centroid}}$                ${\widetilde P_{centorid}}$     Precision

1          Car     24.927, 88.430, 0.673     24.952, 88.519, 0.760     0.996

2          Car     8.611, 68.119, 0.717     8.585, 67.729, 0.770      0.991

3          Car     -0.015, 82.595, 0.755      -0.015, 82.595, 0.755      0.999

4          Car     8.188, 105.823, 0.780     8.217, 105.714, 0.825     0.996

5          Car     11.508, 72.225, 0.731     11.434, 71.608, 0.785     0.984

6          Car     18.829, 62.322, 0.727     18.791, 61.883, 0.790     0.990

7          Truck     21.315, 43.322, 0.958      21.219, 43.156, 0.875      0.990

8          Car     8.538, 38.425, 0.735     8.451, 38.100, 0.730     0.988

9          Car     18.144, 67.893, 0.674      18.165, 68.067, 0.700      0.995

10         Car     0.336, 43.784, 0.730     0.382, 43.959, 0.700     0.993

11         Car     3.812, 46.123, 0.697      3.794, 46.043, 0.710      0.997

12         Car     11.035, 65.584, 0.730     11.261, 65.935, 0.740     0.976

13         Car     0.142, 81.101, 0.721     0.100, 80.344, 0.770     0.984

14         Car     -14.298, 59.759, 0.662     -14.297, 59.758, 0.680     0.999

15         Car     -6.232, 39.097, 0.672     -6.193, 39.138, 0.750     0.993

16          Car     -9.671, 38.371, 0.705     -9.754, 38.703, 0.665     0.978

17          Car     -6.249, 56.957, 0.681     -6.275, 56.747, 0.690      0.989

18          Car     -10.033, 64.324, 0.740      -10.032, 64.321, 0.740      0.999

19          Car     -1.770, 53.300, 0.756     -1.820, 52.683, 0.860     0.975

20          Bus     -5.174, 71.789, 1.452     -5.341, 73.016, 1.410     0.936

21          Car     -7.645, 57.673, 0.800     -7.593, 57.247, 0.800     0.975

22          Car     0.356, 22.313, 0.739      0.340, 22.388, 0.670      0.994

23          Car     0.862, 37.990, 0.735     0.804, 37.053, 0.765     0.957

24          Car     1.376, 40.059, 0.825      1.412, 40.053, 0.860      0.993

[Scene A-4093]

[Scene A-4172]

[Scene A-4391]

[Scene B-6647]

[Scene B-6751]

[Scene B-8001]

[Scene C-8241]

[Scene C-8287]

[Scene C-8351]

[Scene D-4752]

[Scene D-5743]

[Scene D-7930]

[Scene E-4060]

[Scene E-5116]

[Scene E-6712]

=0pt =0pt plus 0cm Top views of 3D vehicle localization of different frames in SVLD-3D test set. The red star indicates ground truth 3D vehicle locations. The blue square indicates predicted 3D vehicle locations. The number below each sub-figure indicates the frame number.

=0pt =0pt plus 0cm Visualization results of 3D vehicle localization on SVLD-3D test set. Color boxes indicate the predicted results. Pink boxes indicate the ground truth results. Numbers on the arrow pointing to vehicles are consistent with numbers in Table <ref>.

[Scene A]

[Scene B]

[Scene C]

[Scene D]

[Scene E]

=0pt =0pt plus 0cm 3D vehicle localization error of X- (green), Y- (red), Z- (blue) axis, and total error (orange) according to the distances between vehicles and roadside cameras in different scenes of SVLD-3D test set.

3D vehicle localization error is calculated by Equation <ref>. In Figure <ref>, we can see that 3D vehicle localization error increases as the distance grows between vehicles and roadside cameras of different scenes in SVLD-3D test set. The Y-axis error is the largest among the X-, Y-, and Z-axis. The largest error in scene B is larger than that in other scenes. This is due to the fact that the camera pan angle in scene B is closer to 0° than in the other scenes, which leads to incomplete vehicle feature learning along the vehicle length direction.

§.§ 3D Vehicle Dimension Precision and Error of CenterLoc3D

Table <ref> and Figure <ref> show 3D vehicle dimension prediction results and precision of different scenes and types in SVLD-3D test set, which is calculated by Equation <ref>. It can be seen that our network can also achieve good results in 3D vehicle dimension prediction, with an average precision of 85%.

3D vehicle dimension prediction results and precision on SVLD-3D test set.

Vehicle     Type     ${D_v}$              ${\widetilde D_v}$     Precision

1         Car     3.60, 1.71, 1.37     3.79, 1.70, 1.27       0.860

2         Car      3.26, 1.67, 1.31     3.18, 1.61, 1.25       0.890

3         Car      4.05, 1.76, 1.40     3.92, 1.80, 1.40       0.942

4         Car      4.51, 1.81, 1.47     4.42, 1.88, 1.46       0.935

5         Car      4.43, 1.78, 1.37     4.40, 1.78, 1.48       0.915

6         Car      4.74, 1.80, 1.46     4.33, 1.77, 1.40       0.843

7         Car      4.58, 1.82, 1.45     4.96, 1.86, 1.54       0.845

8         Car      4.50, 1.79, 1.40     4.50, 1.70, 1.36       0.912

9         Car      3.74, 1.64, 1.27     4.07, 1.68, 1.30       0.872

10        Car      4.55, 1.80, 1.42     4.58, 1.68, 1.40       0.910

11         Car     3.57, 1.80, 1.35     4.11, 1.80, 1.38       0.844

12         Car      3.71, 1.76, 1.36     3.90, 1.80, 1.33       0.912

13         Car      3.34, 1.77, 1.32     3.70, 1.76, 1.25       0.838

14         Bus      12.83, 2.71, 2.75     12.00, 2.76, 2.82       0.886

15         Car      4.74, 1.87, 1.48     4.77, 1.83, 1.53       0.939

16         Car      5.00, 1.89, 1.48     4.75, 1.86, 1.56       0.880

17         Car      4.69, 1.84, 1.44     4.60, 1.81, 1.37       0.914

18         Car      4.68, 1.85, 1.43     4.56, 1.81, 1.34       0.885

19         Car      4.64, 1.84, 1.45     4.68, 1.82, 1.50       0.947

20        Bus      12.74, 2.68, 2.62     12.00, 2.76, 2.82       0.838

=0pt =0pt plus 0cm Visualization results of 3D vehicle dimension prediction on SVLD-3D test set. Color boxes indicate the predicted results. Pink boxes indicate the ground truth results. Numbers on the arrow pointing to vehicles are consistent with numbers in Table <ref>.

[Scene A]

[Scene B]

[Scene C]

[Scene D]

[Scene E]

=0pt =0pt plus 0cm 3D vehicle dimension prediction error of length (green), width (red), height (blue), and total error (orange) according to the distances between vehicles and roadside cameras in three different scenes of SVLD-3D test set.

In Figure <ref>, we can see that 3D vehicle dimension error increases continuously as the distance grows between vehicles and roadside cameras of different scenes in SVLD-3D test set. The vehicle length error is the largest among length, width, and height. Since the driving directions of vehicles are usually parallel to the road direction, partial feature occlusion exists along the vehicle length direction. Therefore, the vehicle dimension prediction error in the length direction is larger than width and height.

3D vehicle dimension prediction error is calculated by Equation <ref>. The errors of different methods are shown in Table <ref>, from which it can be seen that our network has certain advantages in 3D vehicle dimension prediction.

Comparison of 3D vehicle dimension prediction errors of different methods.

Method            Length/m     Width/m     Height/m

3DOP [10]        0.504        0.094       0.107

Mono3D [34]     0.582        0.103       0.172

MonoGRNet [13]     0.412        0.084       0.084

MonoGRK [24]       0.403        0.091       0.101

Ours              0.137        0.031       0.030

§.§ Ablation Study of CenterLoc3D

To further validate the effect of the weighted-fusion module and loss with spatial constraint embedding in our network, ablation experiments are conducted on SVLD-3D validation and test set. CenterNet [33] with ResNet-50 [36] is used as our baseline. We added the proposed modules one by one for validation. Ablation results are shown in Table <ref>. For the improvement column, only $A{P_{3D}}$ of IoU threshold 0.7 on the test set is listed for comparison. In Table <ref>, we can see that the results of adding three modules one by one outperform the former model by 6.64%, 2.48%, and 5.98% in $A{P_{3D}}$ respectively. Therefore, conclusions can be summarized as follows: 1) Weighted-fusion module not only enables the network adaptive to different vehicle sizes but also increases network generalization capability. 2) Spatial constraints of camera calibration and vehicle IoU in loss help accurate 3D bounding box learning.

Ablation study with different modules in CenterLoc3D on SVLD-3D validation and test set.

2*Model     3cModules                                                     2*$A{P_{3D}}(IOU > 0.7)$     2*FPS     2*Improvement of $A{P_{3D}}$
Weighted-Fusion     Reprojection     IoU

${M_{base}}$                                                                                52.52 / 36.20                                                46.73                     -

${M_1}$                                                                               57.07 / 42.84                                                43.23                     6.64

${M_2}$                                                 68.38 / 45.32                                                41.31                     2.48

${M_3}$                                79.36 / 51.30                                                41.18                     5.98

Figure <ref> shows 3D vehicle localization error of ablation study on SVLD-3D test set. Only three views (left, middle and right) of BrnoCompSpeed scenes in SVLD-3D test set are included in this study. In this figure, it can be seen that 3D vehicle localization error from ${M_{base}}$ to ${M_3}$ in the same scene decreases gradually, which indicates that the designed modules can effectively reduce 3D vehicle localization error.

[${M_{base}}$-Scene A-0.34236]

[${M_{base}}$-Scene B-0.35048]

[${M_{base}}$-Scene C-0.29607]

[${M_1}$-Scene A-0.26307]

[${M_1}$-Scene B-0.26794]

[${M_1}$-Scene C-0.25254]

[${M_2}$-Scene A-0.29456]

[${M_2}$-Scene B-0.30740]

[${M_2}$-Scene C-0.24080]

[${M_3}$-Scene A-0.29415]

[${M_3}$-Scene B-0.29399]

[${M_3}$-Scene C-0.22648]

=0pt =0pt plus 0cm 3D vehicle localization error of ablation study on SVLD-3D test set. The results of ${M_{base}}$, ${M_1}$, ${M_2}$ and ${M_3}$ of different scenes are shown from the first to fourth row. 3D vehicle localization error of X-, Y-, Z-axis, and total error according to the distances between vehicles and roadside cameras on SVLD-3D test set are represented by green, red, blue, and orange lines. The number below each sub-figure indicates the average 3D vehicle localization error in meters of the corresponding scene.

§ CONCLUSION

Through experimental validation, CenterLoc3D achieves good performance on 3D vehicle detection, localization, and dimension prediction for roadside surveillance cameras, with $A{P_{3D}}$ of 51.30%, average 3D localization precision of 98%, average 3D dimension precision of 85% and real-time performance with FPS of 41.18. Our contributions are as follows: 1) A 3D vehicle localization network CenterLoc3D for roadside monocular cameras is proposed, which can directly obtain 3D bounding boxes and 3D dimensions without leveraging 2D detectors. 2) A weighted-fusion strategy is proposed, which can effectively enhance feature extraction and improve generalization. 3) Loss function with constraints of camera calibration and vehicle IoU are embedded in CenterLoc3D, which reduces 3D vehicle localization error. In addition, we also propose a benchmark including a dataset, an annotation tool, and evaluation metrics, which provides a data basis for experimental validation.

However, CenterLoc3D is still needed to be improved for practical and advanced applications. When the camera pan angle is close to 0°, the features along the vehicle length direction are incomplete, leading to an increase in 3D vehicle localization error. In future work, the dataset needs to be further expanded to contain scenes with more views and more types of vehicles. In the meanwhile, more effective feature extraction modules and loss functions are needed to be designed to further improve 3D vehicle localization precision in roadside monocular traffic scenes.

#1ISBN #1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<https://doi.org/#1>et al.#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<><#>1#1#1#1#1#1#1#1#1#1#1#1#1#1PreBibitemsHook

[1]

Wang, C.,
Musaev, A.,
Sheinidashtegol, P.,
Atkison, T.:
Towards detection of abnormal vehicle behavior using traffic cameras.
In: International Conference on Big Data,
pp. 125–136

[2]

Huang, C.,
Hu, Z.,
Xu, Y.,
Wang, Y.:
Vision-based abnormal vehicle behavior detection: A survey.
Pattern Recognition and Artificial Intelligence

[3]

Liang, H.,
Song, H.,
Li, H.,
Dai, Z.:
Vehicle counting system using deep learning and multi-object
Transportation Research Record

[4]

Tasgaonkar, P.P.,
Garg, R.D.,
Garg, P.K.:
Vehicle detection and traffic estimation with sensors technologies for
intelligent transportation systems.
Sensing and Imaging

[5]

Dai, Z.,
Song, H.,
Liang, H.,
Wu, F.,
Wang, X.,
Jia, J.,
Fang, Y.:
Traffic parameter estimation and control system based on machine vision.
Journal of Ambient Intelligence and Humanized Computing,

[6]

Shi, S.,
Wang, X.,
Li, H.:
Pointrcnn: 3d object proposal generation and detection from point
In: 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 770–779

[7]

Shi, S.,
Guo, C.,
Jiang, L.,
Wang, Z.,
Shi, J.,
Wang, X.,
Li, H.:
Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.
In: 2020 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 10526–10535

[8]

Lang, A.H.,
Vora, S.,
Caesar, H.,
Zhou, L.,
Yang, J.,
Beijbom, O.:
Pointpillars: Fast encoders for object detection from point clouds.
In: 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 12689–12697

[9]

Zhou, Y.,
Tuzel, O.:
Voxelnet: End-to-end learning for point cloud based 3d object
In: 2018 IEEE/CVF Conference on Computer Vision and Pattern
pp. 4490–4499

[10]

Chen, X.,
Kundu, K.,
Zhu, Y.,
Ma, H.,
Fidler, S.,
Urtasun, R.:
3d object proposals using stereo imagery for accurate object class
IEEE Transactions on Pattern Analysis and Machine Intelligence

[11]

Qi, C.R.,
Liu, W.,
Wu, C.,
Su, H.,
Guibas, L.J.:
Frustum pointnets for 3d object detection from rgb-d data.
In: 2018 IEEE/CVF Conference on Computer Vision and Pattern
pp. 918–927

[12]

Li, P.,
Chen, X.,
Shen, S.:
Stereo r-cnn based 3d object detection for autonomous driving.
In: 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 7636–7644

[13]

Qin, Z.,
Wang, J.,
Lu, Y.:
Monogrnet: A geometric reasoning network for 3d object localization.
The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19)

[14]

Ku, J.,
Pon, A.D.,
Waslander, S.L.:
Monocular 3d object detection leveraging accurate proposals and shape
In: 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 11859–11868

[15]

Liu, Z.,
Wu, Z.,
Tóth, R.:
SMOKE: Single-stage monocular 3d object detection via keypoint estimation.
arXiv preprint arXiv:2002.10111

[16]

Ding, M.,
Huo, Y.,
Yi, H.,
Wang, Z.,
Shi, J.,
Lu, Z.,
Luo, P.:
Learning depth-guided convolutions for monocular 3d object detection.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition,
pp. 11672–11681

[17]

Zhang, Y.,
Lu, J.,
Zhou, J.:
Objects are different: Flexible monocular 3d object detection.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR),
pp. 3289–3298

[18]

Wang, T.,
Zhu, X.,
Pang, J.,
Lin, D.:
Fcos3d: Fully convolutional one-stage monocular 3d object detection.
arXiv preprint arXiv:2104.10956

[19]

Mousavian, A.,
Anguelov, D.,
Flynn, J.,
Košecká, J.:
3d bounding box estimation using deep learning and geometry.
In: 2017 IEEE Conference on Computer Vision and Pattern Recognition
pp. 5632–5640

[20]

Chabot, F.,
Chaouch, M.,
Rabarisoa, J.,
Teulière, C.,
Chateau, T.:
Deep manta: A coarse-to-fine many-task network for joint 2d and 3d
vehicle analysis from monocular image.
In: 2017 IEEE Conference on Computer Vision and Pattern Recognition
pp. 1827–1836

[21]

Yang, S.,
Scherer, S.:
Cubeslam: Monocular 3-d object slam.
IEEE Transactions on Robotics

[22]

Li, B.,
Ouyang, W.,
Sheng, L.,
Zeng, X.,
Wang, X.:
Gs3d: An efficient 3d object detection framework for autonomous
In: 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 1019–1028

[23]

Gupta, I.,
Rangesh, A.,
Trivedi, M.:
3d bounding boxes for road vehicles: A one-stage, localization
prioritized approach using single monocular images.
In: 2018 European Conference on Computer Vision Workshops (ECCV),
pp. 626–641

[24]

Barabanau, I.,
Artemov, A.,
Burnaev, E.,
Murashkin, V.:
Monocular 3d object detection via geometric reasoning on keypoints.
In: 15th International Conference on Computer Vision Theory and

[25]

Li, P.,
Zhao, H.,
Liu, P.,
Cao, F.:
Rtm3d: Real-time monocular 3d detection from object keypoints
for autonomous driving.
In: 2020 European Conference on Computer Vision Workshops (ECCV),
pp. 644–660

[26]

Kocur, V.,
Ftánik, M.:
Detection of 3d bounding boxes of vehicles using perspective
transformation for accurate speed measurement.
Machine Vision and Applications

[27]

Yang, L.,
Zhang, X.,
Wang, L.,
Zhu, M.,
Li, J.:
Lite-FPN for Keypoint-based Monocular 3D Object Detection

[28]

Li, P.,
Zhao, H.:
Monocular 3d detection with geometric constraint embedding and
semi-supervised training.
IEEE Robotics and Automation Letters

[29]

Ren, S.,
He, K.,
Girshick, R.,
Sun, J.:
Faster r-cnn: Towards real-time object detection with region proposal
IEEE Transactions on Pattern Analysis and Machine Intelligence

[30]

Liu, W.,
Anguelov, D.,
Erhan, D.,
Szegedy, C.,
Reed, S.,
Fu, C.-Y.,
Berg, A.C.:
Ssd: Single shot multibox detector.
In: 2016 European Conference on Computer Vision (ECCV),
pp. 21–37

[31]

Bochkovskiy, A.,
Wang, C.-Y.,
Liao, H.-Y.M.:
YOLOv4: Optimal Speed and Accuracy of Object Detection

[32]

Lin, T.-Y.,
Goyal, P.,
Girshick, R.,
He, K.,
Dollár, P.:
Focal loss for dense object detection.
In: 2017 IEEE International Conference on Computer Vision (ICCV),
pp. 2999–3007

[33]

Zhou, X.,
Wang, D.,
Krähenbühl, P.:
Objects as points.
In: arXiv Preprint arXiv:1904.07850

[34]

Chen, X.,
Kundu, K.,
Zhang, Z.,
Ma, H.,
Fidler, S.,
Urtasun, R.:
Monocular 3d object detection for autonomous driving.
In: 2016 IEEE Conference on Computer Vision and Pattern Recognition
pp. 2147–2156

[35]

Liu, L.,
Lu, J.,
Xu, C.,
Tian, Q.,
Zhou, J.:
Deep fitting degree scoring network for monocular 3d object
In: 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 1057–1066

[36]

He, K.,
Zhang, X.,
Ren, S.,
Sun, J.:
Deep residual learning for image recognition.
In: 2016 IEEE Conference on Computer Vision and Pattern Recognition
pp. 770–778

[37]

Sochor, J.,
Juránek, R.,
Herout, A.:
Traffic surveillance camera calibration by 3d model bounding box
alignment for accurate vehicle speed measurement.
Computer Vision and Image Understanding

[38]

Yu, F.,
Wang, D.,
Shelhamer, E.,
Darrell, T.:
Deep layer aggregation.
In: 2018 IEEE/CVF Conference on Computer Vision and Pattern
pp. 2403–2412

[39]

Kanhere, N.K.,
Birchfield, S.T.:
A taxonomy and analysis of camera calibration methods for traffic
monitoring applications.
IEEE Trans. Intell. Transp. Syst.

[40]

Wang, W.,
Zhang, C.,
Tang, X.,
Song, H.,
Cui, H.:
Automatic self-calibration and optimization algorithm of traffic
camera in road scene.
Journal of Computer-Aided Design and Computer Graphics

[41]

Lin, T.-Y.,
Dollár, P.,
Girshick, R.,
He, K.,
Hariharan, B.,
Belongie, S.:
Feature pyramid networks for object detection.
In: 2017 IEEE Conference on Computer Vision and Pattern Recognition
pp. 936–944

[42]

Hu, J.,
Shen, L.,
Sun, G.:
Squeeze-and-excitation networks.
In: 2018 IEEE/CVF Conference on Computer Vision and Pattern
pp. 7132–7141

[43]

Zheng, Z.,
Wang, P.,
Liu, W.,
Li, J.,
Ye, R.,
Ren, D.:
Distance-iou loss: Faster and better learning for bounding box
In: Proceedings of the AAAI Conference on Artificial Intelligence,
vol. 34,
pp. 12993–13000

[44]

Geiger, A.,
Lenz, P.,
Urtasun, R.:
Are we ready for autonomous driving? the kitti vision benchmark
In: 2012 IEEE Conference on Computer Vision and Pattern Recognition,
pp. 3354–3361

[45]

Sun, P.,
Kretzschmar, H.,
Dotiwalla, X.,
Chouard, A.,
Patnaik, V.,
Tsui, P.,
Guo, J.,
Zhou, Y.,
Chai, Y.,
Caine, B.,
Vasudevan, V.,
Han, W.,
Ngiam, J.,
Zhao, H.,
Timofeev, A.,
Ettinger, S.,
Krivokon, M.,
Gao, A.,
Joshi, A.,
Zhang, Y.,
Shlens, J.,
Chen, Z.,
Anguelov, D.:
Scalability in perception for autonomous driving: Waymo open dataset.
In: 2020 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 2443–2451

[46]

Caesar, H.,
Bankiti, V.,
Lang, A.H.,
Vora, S.,
Liong, V.E.,
Xu, Q.,
Krishnan, A.,
Pan, Y.,
Baldan, G.,
Beijbom, O.:
nuscenes: A multimodal dataset for autonomous driving.
In: 2020 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR),
pp. 11618–11628

[47]

Sochor, J.,
Juránek, R.,
Špaňhel, J.,
Maršík, L.,
Široký, A.,
Herout, A.,
Zemčík, P.:
Comprehensive data set for automatic single camera visual speed
IEEE Transactions on Intelligent Transportation Systems

[48]

Wang, W.,
Tang, X.,
Zhang, C.,
Song, H.,
Cui, H.:
Spatial positioning method of vehicle in cross-camera traffic scene.
Journal of Computer-Aided Design and Computer Graphics

[49]

Everingham, M.,
Van Gool, L.,
Williams, C.,
Winn, J.,
Zisserman, A.:
The pascal visual object classes (voc) challenge.
International Journal of Computer Vision


# Geometric Constraints Enable Self-Supervised Sinogram
Inpainting in Sparse-View Tomography

Fabian Wagner  Pattern Recognition Lab, Friedrich-Alexander-Universität
Erlangen-Nürnberg, Germany  Mareike Thies  Pattern Recognition Lab, Friedrich-
Alexander-Universität Erlangen-Nürnberg, Germany  Noah Maul  Pattern
Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
Laura Pfaff  Pattern Recognition Lab, Friedrich-Alexander-Universität
Erlangen-Nürnberg, Germany  Oliver Aust  Department of Rheumatology and
Immunology, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
Sabrina Pechmann  Fraunhofer Institute for Ceramic Technologies and Systems
IKTS, Forchheim, Germany  Christopher Syben  Fraunhofer Development Center
X-Ray Technology EZRT, Erlangen, Germany  Andreas Maier  Pattern Recognition
Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany

Abstract The diagnostic quality of computed tomography (CT) scans is usually
restricted by the induced patient dose, scan speed, and image quality. Sparse-
angle tomographic scans reduce radiation exposure and accelerate data
acquisition, but suffer from image artifacts and noise. Existing image
processing algorithms can restore CT reconstruction quality but often require
large training data sets or can not be used for truncated objects. This work
presents a self-supervised projection inpainting method that allows optimizing
missing projective views via gradient-based optimization. By reconstructing
independent stacks of projection data, a self-supervised loss is calculated in
the CT image domain and used to directly optimize projection image intensities
to match the missing tomographic views constrained by the projection geometry.
Our experiments on real X-ray microscope (XRM) tomographic mouse tibia bone
scans show that our method improves reconstructions by
$3.1$–$7.4\,\%$/$7.7$–$17.6\,\%$ in terms of PSNR/SSIM with respect to the
interpolation baseline. Our approach is applicable as a flexible self-
supervised projection inpainting tool for tomographic applications.
---

##  1 Introduction

Computed tomography (CT) scanners allow the reconstruction of unknown 3D
object density distributions from a set of acquired X-ray projection images.
Most clinical applications use filtered back projection (FBP)-based
reconstruction algorithms that require a dense angular sampling to meet the
precondition of the analytical algorithm. Although decreasing the number of
measured projection images can be beneficial to reduce patient dose, improve
acquisition speed, and reduce motion effects [1], it introduces noise and
image artifacts that can impair diagnostic value.
Different image processing algorithms were proposed to restore the image
quality of scans acquired with reduced angular sampling and dose, intervening
at different stages of the CT reconstruction pipeline. A first set of
algorithms operates directly on the acquired sinogram data with the goal to
upsample the number of projection images [2, 3]. Consistency conditions on CT
projection data were applied to limited-angle acquisitions to improve image
quality while preserving consistency with the measured data [4, 5]. However,
such methods are often insensitive to in-plane artifacts and only work for
non-truncated data. A second group of methods apply pure image post-processing
algorithms to improve the overall quality of noisy reconstructions acquired
with reduced dose [6, 7, 8]. To make CT reconstruction pipelines compatible
with gradient-based data-driven training, differentiable FBP operators were
presented that allow propagating a loss calculated on the reconstructed image
back to the raw sinogram data [9, 10]. Such known operators [11] allow
training CT pipelines employing neural networks on the sinogram and the
reconstruction simultaneously [12]. Existing approaches use neural networks or
other denoising operators to restore image quality [13, 14] or make use of
conventional inpainting techniques to increase the angular projection sampling
artificially [15].
So far most learning-based models are trained supervisedly on large-scale low
and high-quality paired data sets [2]. However, multiple self-supervised
training approaches exist that circumvent the need for paired training data.
Noise2Noise [16] calculates a loss metric from two independent noisy image
representations. Noise2Inverse [17] and other works [18] extend this principle
to tomographic CT settings by splitting projection data into two independent
sets. The resulting reconstructions are regarded as image representations with
independent noise realizations and allow for deriving a self-supervised
denoising loss. Only a few self-supervised inpainting techniques exist and
even fewer are applied to CT problems [3, 19].
In this work, we present a self-supervised CT projection inpainting method. We
regard missing projection images as trainable data tensors that are updated to
be consistent with the measured data. Reconstructing them with a
differentiable FBP operator allows deriving a self-supervised loss in the
image domain with the real measured sparse-angle reconstruction.
Backpropagating that loss through our fully differentiable reconstruction
pipeline allows directly optimizing the missing projection data tensor to
generate additional projection views. Our contributions are the following:

  * •

We present a self-supervised projection inpainting method based on a
differentiable FBP pipeline.

  * •

We propose to directly optimize missing CT projection information through the
fixed projective geometry of a differentiable FBP operator.

  * •

We evaluate our method on high-resolution cone-beam X-ray microscope (XRM) ex-
vivo mouse tibia bone data.

Figure 1: Illustration of the proposed self-supervised projection inpainting
scheme. A target image is reconstructed from a sparsely sampled CT scan, e.g.,
one-third of the usual number of projections from a high-resolution scan
(orange data). The intensities of intermediate missing projections are
regarded as trainable parameters (green projections). Further, a self-
supervised loss is calculated between the two reconstructed projection stacks
and backpropagated through a differentiable FBP operator to the inpainted
projections to update them consistently with the measured data.

##  2 Methods

###  2.1 Differentiable Reconstruction Pipeline

The acquisition of CT projection images $\mathbf{p}_{j}$ with
$j\in\\{0,\dots,J\\}$ can be written as

$\displaystyle\mathbf{p}_{j}=\mathbf{A}[G_{j}]\mathbf{y}+\mathbf{n}_{j}$ (1)

with the true scanned object $\mathbf{y}$, noise vector $\mathbf{n}_{j}$, and
the forward operator $\mathbf{A}$ conditioned to the projection geometry $G$.
The inverse CT problem aims to reconstruct the unknown object from the
acquired set of $J$ projections using the adjoint system matrix
$\mathbf{A}^{\top}$ and the filter operator $\mathbf{K}$

$\displaystyle\mathbf{\hat{y}}_{J}=\frac{1}{J}\sum_{j}\mathbf{A}[G_{j}]^{\top}\mathbf{K}\mathbf{p}_{j}\enspace.$
(2)

Using a differentiable backprojection operator [9] allows backpropagating a
loss metric $L$ derived on the reconstructed image $\mathbf{\hat{y}}$ to the
raw CT projection data using the chain rule

$\displaystyle\frac{\partial L}{\partial\mathbf{p}_{j}}=\frac{\partial
L}{\partial\mathbf{\hat{y}}}\frac{\partial\mathbf{\hat{y}}}{\partial\mathbf{p}_{j}}\enspace.$
(3)

Hence, models working in the projection domain or the projection data itself
can be directly modified using gradient-based optimization.

###  2.2 Optimizing Consistent Sinogram Information

Following the Noise2Inverse [17] pipeline, a self-supervised loss $L$ can be
derived from the reconstructions of two independent stacks of CT projection
images $\mathbf{p}_{j}$ ($j\in\\{0,\dots,J\\}$) and $\mathbf{p}_{k}$
($k\in\\{0,\dots,K\\}$). In the reconstruction domain, only their image
content but not their noise realization is correlated which allows deriving a
loss between both reconstructed images to assess the consistency of the image
contents. Other works [17] minimize that loss to train the weights of
denoising models $f(\cdot,w)$ in the projection and the reconstruction domain
self-supervisedly

$\displaystyle\underset{w}{\text{argmin
}}L\left(\mathbf{\hat{y}}_{J},\mathbf{\hat{y}}_{K}\right)\enspace.$ (4)

In the sparse-angle tomographic setting where only a limited number of $J$
projection images was measured, we propose to minimize the following
expression to optimize the missing set of $K$ noise-free projection images
$\mathbf{p}_{k}$

$\displaystyle\underset{\mathbf{p}_{k}}{\text{argmin
}}L\left(\mathbf{\hat{y}}_{J},\mathbf{\hat{y}}_{K}\right)=\underset{\mathbf{p}_{k}}{\text{argmin
}}L\left(\mathbf{\hat{y}}_{J}[G_{0:J}],\mathbf{\hat{y}}_{K}[G_{0:K}]\right)\enspace.$
(5)

We regard the pixel intensities of the inpainted projection images
$\mathbf{p}_{k}$ as trainable weights and optimize them in a self-supervised
and data-driven way as illustrated in Fig. 1. To enable backpropagating the
gradient to the projection images, we use a differentiable FBP operator [9]
within our XRM reconstruction pipeline [10]. Due to the well-defined
projection geometries $G_{j}$ and $G_{k}$ for the individual projection images
$\mathbf{p}_{j}$ and $\mathbf{p}_{k}$, the proposed pipeline is constrained to
optimize missing angular projection data in between the measured sparse-angle
projections to predict realistic reconstructions $\mathbf{\hat{y}}_{K}$ close
to $\mathbf{\hat{y}}_{J}$. Although the pipeline can in theory converge to an
identity solution where inpainted projections equal the true measured data,
this solution is not favored by the derived pixel-wise loss as both
independent reconstructions would be slightly misregistered through the
different sets of view geometries $G_{0:J}$ and $G_{0:K}$.

##  3 Experiments

###  3.1 Data

The used data set consists of five high-resolution X-ray microscope (XRM)
cone-beam scans of ex-vivo mouse tibia bones. Investigating bone structures on
the micrometer scale is instructive for understanding the cause and
progression of bone-related diseases on the cellular level as well as for
developing adapted therapies. Lacunar bone structures, visible in Fig. 2 as
tiny holes in the bone, are in particular of interest as they contain
osteocyte cells that are heavily involved in the bone-remodeling process [20].
To resolve Lacunae reasonably well, $1401$ projection images are acquired,
which accumulates to a total acquisition time of around $14\,\text{h}$. Here,
neither acquisition time nor induced sample dose allow for desired in-vivo
investigations [1]. Sparse-angle CT acquisitions in combination with self-
supervised projection inpainting algorithms can enable faster low-dose
acquisition while preserving a high reconstruction quality. The used XRM scans
image the tibia bone of mice close to the knee joint and contain truncated
information of the proximal fibula bone in some projective views.
The study was performed in line with the principles of the Declaration of
Helsinki. Approval was granted by the Ethics Committee of FAU Erlangen-
Nürnberg (license TS-10/2017).

###  3.2 Training

We evaluate the effectiveness of our method on five XRM mouse tibia scans
using half ($50\,\%$ dose) and one-third ($33\,\%$ dose) of the available
$1401$ projections for all compared inpainting strategies. Projection images
are reconstructed using the public cone-beam XRM reconstruction framework of
Thies et al. [10]. Nearest neighbor interpolation and trilinear interpolation
operators are taken from the PyTorch framework and compared to our proposed
self-supervised projection inpainting method. We initialize the missing
projection intensities with interpolated projections to accelerate the
optimization and start from a reasonable reconstruction. Subsequently,
projections $\mathbf{p}_{k}$ are registered in the PyTorch graph as trainable
parameters and are updated using a stochastic gradient descent optimizer with
learning rate $0.1$ without momentum. Further, we use the mean absolute error
as loss function $L$. Quantitative performance metrics are calculated with
respect to the high-resolution XRM images reconstructed from all available
projections.

##  4 Results and Discussion

$50\,\%$ dose | PSNR | SSIM
---|---|---
Nearest neigh. int. | $27.0\pm 0.4$ | $0.620\pm 0.008$
Trilinear int. | $27.8\pm 0.4$ | $0.656\pm 0.008$
Optimized projections | $\bm{29.0\pm 0.5}$ | $\bm{0.729\pm 0.009}$
$33\,\%$ dose | PSNR | SSIM
Nearest neigh. int. | $25.8\pm 0.5$ | $0.571\pm 0.011$
Trilinear int. | $26.2\pm 0.5$ | $0.585\pm 0.011$
Optimized projections | $\bm{26.6\pm 0.5}$ | $\bm{0.615\pm 0.013}$
Table 1: Quantitative reconstruction results (mean $\pm$ std) calculated
between the prediction starting from one half and one third of the full number
of projections and the high-resolution reconstruction that is regarded as
ground truth.

We compute the commonly used quantitative image quality metrics peak signal-
to-noise ratio (PSNR) and structural similarity index measure (SSIM) to
evaluate reconstructions with respect to the high-resolution ground truth
images reconstructed from all available projections. Mean $\pm$ standard
deviation of the used five XRM scans is provided in Tab. 1 for both
investigated sparse-angle settings. The calculated metrics indicate improved
reconstruction quality of the optimized XRM projections across both sparse-
angle tomographic settings and over the compared reference methods. In our
experiments, optimized projection images improve the PSNR by $3.1$–$7.4\,\%$
and the SSIM by $7.7$–$17.6\,\%$ relative to the nearest neighbor
interpolation baseline.

Figure 2: Reconstructions of mouse tibia bone XRM scans. The magnified region
of interest is highlighted in the overview slice. The ground truth (GT) and
low-resolution (LR) images are calculated from all and one-third of the
available projection images respectively. Nearest, trilinear, and optimized
denote the different reconstructions from the interpolated and optimized CT
projection images.

Qualitative results are compared on the reconstructions of the inpainted
projection images in magnified regions of interest in Fig. 2. Biologically
interesting lacunar structures are visible as small dark holes within the bone
tissue. Whereas nearest neighbor interpolation only slightly improved the
noise level within the bone region, trilinear interpolated and optimized
projections further reduce reconstruction noise. Although the bone
reconstructed from the optimized projections appears a bit smoother compared
to trilinear interpolation, only small improvements are visible.
None of the compared methods contain deep neural networks or require any form
of pre-training on additional data, reference data, or a learned prior. In our
proposed method, projection intensities are directly changed using gradient-
based optimization within the fully differentiable reconstruction pipeline. In
contrast to other techniques employing view consistency, we believe that our
self-supervised pipeline is not limited to circular CT trajectories but can be
applied to more difficult acquisition geometries as long as there is a
differentiable reconstruction operator at hand. In addition, three of the five
used bone scans contain truncated residual parts of the proximal fibula bone
which are only visible in some projection views which would severely disturb
most projection consistency-based algorithms. Our experiments show that the
present data truncation can be handled well by our proposed projection
optimization method. Further work is required to fully evaluate the
generalizability of end-to-end optimization of CT projections and compare it
to existing projection consistency-based methods or deep learning-based
models.

##  5 Conclusion

In this work, we present a truly self-supervised projection inpainting
technique to improve the reconstruction quality of sparsely acquired CT
projection data. Our method allows directly optimizing projection image
intensities through a differentiable FBP operator and a self-supervised loss
metric calculated from two independently reconstructed projection stacks. The
proposed pipeline requires further evaluation to prove its clinical
effectiveness. Nevertheless, it has great potential over existing self-
supervised algorithms as additional regularization can be applied and data
consistency is enforced.

##  6 Acknowledgments

This work was supported by the European Research Council (ERC Grant No.
810316) and a GPU donation through the NVIDIA Hardware Grant Program.

## References

  * [1] Fabian Wagner et al.  “Monte Carlo dose simulation for in-vivo X-ray nanoscopy”  In _Proc. BVM_ Springer, 2022, pp. 107–112
  * [2] Haoyu Wei et al.  “2-step sparse-view CT reconstruction with a domain-specific perceptual network”  In _arXiv preprint arXiv:2012.04743_ , 2020
  * [3] Guangming Zang et al.  “IntraTomo: Self-supervised learning-based tomography via sinogram synthesis and prediction”  In _Proc. ICCV_ , 2021, pp. 1960–1970
  * [4] Yixing Huang et al.  “Restoration of missing data in limited angle tomography based on Helgason–Ludwig consistency conditions”  In _Biomed Phys Eng Express_ 3.3 IOP Publishing, 2017, pp. 035015
  * [5] André Aichert et al.  “Epipolar consistency in transmission imaging”  In _IEEE Trans Med Imaging_ 34.11 IEEE, 2015, pp. 2205–2219
  * [6] Fabian Wagner et al.  “Ultralow-parameter denoising: Trainable bilateral filter layers in computed tomography”  In _Med Phys_ 49.8 Wiley Online Library, 2022, pp. 5107–5120
  * [7] Zhicheng Zhang, Xiaokun Liang, Wei Zhao and Lei Xing  “Noise2Context: Context-assisted learning 3D thin-layer for low-dose CT”  In _Med Phys_ 48.10 Wiley Online Library, 2021, pp. 5794–5803
  * [8] Sun-Young Jeon, Wonjin Kim and Jang-Hwan Choi  “MM-Net: Multi-frame and multi-mask-based unsupervised deep denoising for low-dose computed tomography”  In _IEEE TRPMS_ IEEE, 2022, pp. 1–12
  * [9] Christopher Syben et al.  “PYRO-NN: Python reconstruction operators in neural networks”  In _Med Phys_ 46.11 Wiley Online Library, 2019, pp. 5110–5115
  * [10] Mareike Thies et al.  “Calibration by differentiation–Self-supervised calibration for X-ray microscopy using a differentiable cone-beam reconstruction operator”  In _J Microsc_ 287.2 Wiley Online Library, 2022, pp. 81–92
  * [11] Andreas K Maier et al.  “Learning with known operators reduces maximum error bounds”  In _Nat Mach Intell_ 1.8 Nature Publishing Group, 2019, pp. 373–380
  * [12] Fabian Wagner et al.  “On the benefit of dual-domain denoising in a self-supervised low-dose CT setting”  In _arXiv preprint arXiv:2211.01111_ , 2022
  * [13] Mareike Thies et al.  “Learned cone-beam CT reconstruction using neural ordinary differential equations”  In _Proc. CT Meeting_ 12304 SPIE, 2022, pp. 1230409
  * [14] Yixing Huang et al.  “Data consistent artifact reduction for limited angle tomography with deep learning prior”  In _MLMIR_ , 2019, pp. 101–112  Springer
  * [15] Yinsheng Li et al.  “Strategy of computed tomography sinogram inpainting based on sinusoid-like curve decomposition and eigenvector-guided interpolation”  In _JOSA A_ 29.1 Optica Publishing Group, 2012, pp. 153–163
  * [16] Jaakko Lehtinen et al.  “Noise2Noise: Learning image restoration without clean data”  In _arXiv preprint arXiv:1803.04189_ , 2018
  * [17] Allard Adriaan Hendriksen, Daniël Maria Pelt and K Joost Batenburg  “Noise2Inverse: Self-supervised deep convolutional denoising for tomography”  In _IEEE Trans Comput Imaging_ 6 IEEE, 2020, pp. 1320–1335
  * [18] Dufan Wu, Kyungsang Kim and Quanzheng Li  “Low-dose CT reconstruction with Noise2Noise network and testing-time fine-tuning”  In _Med Phys_ 48.12 Wiley Online Library, 2021, pp. 7657–7672
  * [19] Byeongjoon Kim, Hyunjung Shim and Jongduk Baek  “A streak artifact reduction algorithm in sparse-view CT using a self-supervised neural representation”  In _Med Phys_ 49.12 Wiley Online Library, 2022, pp. 7497–7515
  * [20] Anika Grüneboom et al.  “Next-generation imaging of the skeletal system and its blood supply”  In _Nat Rev Rheumatol_ 15.9 Nature Publishing Group, 2019, pp. 533–549