# CN-RMA: Combined Network with Ray Marching Aggregation for 3D Indoor Object
Detection from Multi-view Images

Guanlin Shen1 Jingwei Huang2 Zhihua Hu3 Bin Wang1111*Corresponding author,
email address<EMAIL_ADDRESS>This work was supported by the
National Natural Science Foundation of China under Grant 62072271.
1School of Software, Tsinghua University, China 2Tencent, China
3Nanjing University of Information Science and Technology, China

###### Abstract

This paper introduces CN-RMA, a novel approach for 3D indoor object detection
from multi-view images. We observe the key challenge as the ambiguity of image
and 3D correspondence without explicit geometry to provide occlusion
information. To address this issue, CN-RMA leverages the synergy of 3D
reconstruction networks and 3D object detection networks, where the
reconstruction network provides a rough Truncated Signed Distance Function
(TSDF) and guides image features to vote to 3D space correctly in an end-to-
end manner. Specifically, we associate weights to sampled points of each ray
through ray marching, representing the contribution of a pixel in an image to
corresponding 3D locations. Such weights are determined by the predicted
signed distances so that image features vote only to regions near the
reconstructed surface. Our method achieves state-of-the-art performance in 3D
object detection from multi-view images, as measured by<EMAIL_ADDRESS>and<EMAIL_ADDRESS>on the ScanNet and ARKitScenes datasets. The code and models are released at
https://github.com/SerCharles/CN-RMA.

##  1 Introduction

Figure 1: The Comparison of Our CN-RMA, the two-stage method, and
ImVoxelNet[25]. Our CN-RMA is an end-to-end object detection method that
incorporates an occlusion-aware 2D to 3D aggregation technique. In contrast,
the two-stage method lacks end-to-end trainability, while ImVoxelNet employs a
heuristic aggregation method that disregards occlusion considerations.

3D object detection from multi-view images is a fundamental problem in various
fields, including robotics, autonomous driving, and augmented reality (AR).
However, since explicit scene geometry is unavailable to detect occlusion, it
is an ill-posed problem to identify correspondences between image regions and
3D locations. Therefore, image features can be wrongly projected to 3D,
leading to inaccurate detection. While occlusion is not a critical issue in
open space and is ignored for autonomous driving scenarios [15, 32], it
commonly exists among objects in complex environments like indoor scenes.

One straightforward solution for 3D object detection from multi-view images is
to perform 3D scene reconstruction from multi-view images [19, 26, 22, 30, 33,
3] followed by 3D object detection from reconstructed point clouds [40, 24,
9]. However, such a solution is not ideal due to the lack of connectivity
between the two stages. The first stage usually introduces noises and
incompletion in the reconstructed 3D geometry with limited power of 3D
reconstruction techniques, and such geometry loss is not resolved in the
second stage. Moreover, aggregated color signals to inaccurate geometry cannot
fully exploit rich image features and further harm the performance. An
alternative method proposed by ImVoxelNet [25] involves aggregating 2D
features extracted from multi-view images into 3D voxel volumes through
unprojection in an end-to-end manner. However, due to the lack of scene
geometry information, this exploratory aggregation approach struggles to
effectively address complex occlusion issues, leading to feature voting from
images to unrelated 3D locations.

In this paper, we present CN-RMA, an end-to-end novel 3D object detection
method from multi-view images that seamlessly combines the reconstruction and
detection networks with occlusion-aware feature aggregation. Our network
mainly consists of a Multi-View Stereo (MVS) module [19] and a novel
occlusion-aware aggregation module followed by a 3D detection module [24]. In
the MVS module, we aim to reconstruct the rough scene geometry. We extract 2D
features from the input images and feed them into the reconstruction network
to generate a rough Truncated Signed Distance Function (TSDF) [7] as a 3D
representation. Our key contribution is the occlusion-aware aggregation module
called Ray Marching Aggregation (RMA), which leverages the reconstructed TSDF
to detect occlusion based on ray marching. In comparison with conventional 3D
detection methods that vote image features equally along rays to the 3D space,
we associate different weights according to the signed distance values.
Specifically, RMA incorporates the idea of volume density given TSDF inspired
by NeuS [30] and accumulates transmittance through ray marching to calculate
the weight of each point along a ray, effectively addressing the occlusion
issues encountered in complex environments. Then, we can aggregate image
features by weights in the 3D space aware of occlusion. Finally, we extract
points with aggregated features near the reconstructed surface and pass the
point cloud to the 3D detection module for object detection. Given the
challenging task, we propose a pre-training and fine-tuning method to train
the entire network, making the components cooperate to achieve the best
performance. Figure 1 illustrates the comparison between our proposed CN-RMA,
ImVoxelNet [25] and the two-stage method.

We evaluate our method on the ScanNet [8] and ARKitScenes [1] datasets to
assess its performance and compare it with existing methods. Our approach
outperforms other methods, achieving significant improvements in<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>including 3.2 and 3.0 in ScanNet, and 7.4 and 13.1 in ARKitScenes,
respectively.

In summary, our contributions are three-fold:

  * $\bullet$

We establish a seamless connection between the multi-view 3D reconstruction
network and 3D object detection network, enabling better exploitation of image
features in 3D space for improved performance.

  * $\bullet$

We propose an innovative occlusion-aware aggregation method, RMA, which
leverages the reconstructed scene TSDF to address the complex occlusion
issues.

  * $\bullet$

We adopt a pretraining and finetuning scheme, and achieve the state-of-the-art
performance for indoor 3D object detection from multi-view images.

##  2 Related Work

###  2.1 3D Object Detection from Multi-View Images

3D object detection from multi-view images has been a hot topic in the vision
community for many years. It aims to estimate the classes, poses, and sizes of
objects from images. For outdoor scenes, a lot of methods project the features
to Bird’s Eye View (BEV) for the sake of memory and better performance [10,
13, 34, 12, 29]. However, the BEV representation is not suitable for 3D object
detection in indoor scenes due to object stacking and occlusion. In recent
years, several methods have tried to aggregate 2D features in 3D space [25,
32, 15]. For instance, ImVoxelNet [25] projects the 2D features from images
into 3D space and aggregates the features with 3D CNN in the voxel form.
DETR3D [32] detects objects by generating random 3D object queries and linking
3D positions to images with camera transformation. PETR [15] produces the 3D
position-aware features by encoding the position information of 3D coordinates
into image features. However, these exploratory aggregation methods have not
taken full advantage of the scene geometry. ImGeoNet [28] introduces geometry
implicit, while NeRF-Det [36] incorporates NeRF [18]. However, these methods
have not considered the occlusion during the feature aggregation process,
leading to inaccurate detection results.

###  2.2 Neural Implicit Reconstruction

To recover the 3D geometry from multi-view images, neural implicit
representations are often adopted, such as the Signed Distance Function (SDF)
[7, 11, 19, 26, 3]. The scene mesh can be obtained from the SDF using
techniques like Marching Cubes [17]. For instance, Atlas [19] predicts the
Truncated Signed Distance Function (TSDF) of the scene with 3D CNN. And many
subsequent methods have made improvements based on the Atlas network.
NeuralRecon [26] splits one complete scene into fragments and uses a Gated
Recurrent Unit (GRU) [5] network to fuse the 3D features of the fragments to
save time and memory. VolumeFusion [3] employs deep MVS [37] techniques to
predict the TSDF. However, these improvements, while enhancing the
effectiveness of 3D reconstruction, have made the network more complex and
challenging to combine with other networks.

In recent years, NeRF [18] based methods have utilized neural implicit fields
in novel view synthesis and 3D reconstruction [39, 2, 31, 16].

For example, NeuS [30] and VolSDF [38] incorporate the SDF into neural
radiance fields by integrating it into the density function, bridging the gap
between the SDF and the volume density of points along each ray. It shows the
possibility of sampling and weighting points in 3D space with the Truncated
Signed Distance Function (TSDF), which can be used to address the occlusion
issues.

###  2.3 3D Object Detection from Point Clouds

3D object detection from point clouds is much more straightforward. According
to the representation of point clouds, it can be divided into point cloud-
based and voxel-based methods. For point cloud-based methods, the voting
scheme introduced by VoteNet [21] is broadly adopted [31, 35], while
PointNet++ [20] is often used to extract the features of point clouds.
However, voting-based object detection methods require additional data
annotation on the point clouds, making them difficult to combine with
reconstruction networks. Voxel-based methods usually convert point clouds into
3D voxels and utilize 3D CNN for voxel processing [40, 12]. However, dense
volumetric representation and 3D CNN are memory-consuming. Thus, sparse
convolution based on sparse voxels has been introduced to improve the
performance of 3D object detection [9, 24].

Our approach focuses on enhancing the performance of 3D indoor object
detection from multi-view images by integrating 3D scene reconstruction
methods and 3D object detection methods from point clouds. To take full
advantage of the scene geometry and handle occlusions, we designed an
occlusion-aware aggregation method based on neural implicit representations.

##  3 Method

Figure 2: The overall architecture of our CN-RMA method. The purple blocks
represent neural networks, while the green blocks represent modules without
trainable neurons. Following Atlas [19], the 2D CNN backbone $\mathcal{F}$ is
a ResNet50-FPN network [14], and the 3D reconstruction network $\mathcal{R}$
is a 3D CNN network that features an encoder-decoder structure with skip
connections with a $1\times 1\times 1$ convolutional head. Following FCAF3D
[24], the object detection network $\mathcal{D}$ is a sparse 3D convolutional
network comprising a ResNet34 backbone [4] and a 4-layer decoder network.

###  3.1 Problem Formulation

We aim to achieve precise 3D object detection in a cluttered scene with
complex occlusions using multi-view images and their corresponding camera
parameters. To accomplish this, we propose a pipeline that combines a MVS
reconstruction module and a 3D detection network through our occlusion-aware
aggregation approach, as shown in Figure 2.

Given the input images $\\{\textbf{I}_{i}\in R^{h_{I}\times w_{I}\times
3}\\}$, along with corresponding camera intrinsics $\\{\textbf{K}_{i}\in
R^{3\times 3}\\}$ and extrinsics $\\{\textbf{R}_{i}\in R^{4\times 4}\\}$, our
goal is to predict the 3D bounding boxes $\\{\textbf{b}_{j}\\}$ and label
scores $\\{\textbf{s}_{j}\\}$ of objects in the scene. Our pipeline borrows
components from multi-view stereo (MVS) [19] and point cloud-based 3D
detection methods [24]. We first extract the $C$-channel 2D features
${\textbf{F}_{i}\in R^{h\times w\times C}}$ for each image with the 2D
backbone $\mathcal{F}$. Then we aggregate the image features
$\\{\textbf{F}_{i}\\}$ as volume feature $\textbf{V}\in R^{W\times H\times
D\times C}$ with the assistance of associated camera parameters
$\\{\textbf{K}_{i}\\}$ and $\\{\textbf{R}_{i}\\}$ using unprojection and
average pooling [19, 26, 25], which is denoted as $\mathcal{A}$.

Then we predict the rough scene TSDF $\textbf{S}\in R^{W\times H\times D}$
from the 3D volumes V using the 3D reconstruction network $\mathcal{R}$
(Section 3.2). Section 3.3 introduces a novel occlusion-aware aggregation
module $\mathcal{A}^{*}$ to extract the 3D geometry as a point cloud with
features $\textbf{P}\in R^{N_{pt}\times(3+C)}$, based on the rough scene TSDF
S and a ray-marching-based voting scheme. Finally, the point cloud with
features P are passed through the detection network $\mathcal{D}$ to obtain
the 3D bounding boxes $\\{\textbf{b}_{j}\\}$ and their corresponding label
scores $\\{\textbf{s}_{j}\\}$ (Section 3.4).

###  3.2 Multi-View Stereo Module

Complete reconstruction is important to avoid missing detection. While NeRF-
based methods [18, 30, 38] deliver complete results, they require fitting
model parameters for each specific scene. Since our task requires obtaining
model parameters that can be universally applicable to all validation scenes,
selecting them as our MVS module may lead to lower generalization ability or
increased complexity in network training. Among end-to-end 3D reconstruction
methods, we find Atlas [19] a proper choice as our MVS module since it can be
trained and used to predict the reconstruction in an end-to-end manner,
including a 2D backbone and a 3D reconstruction network.

Specifically, for an input image $I_{i}$, we first extract the 2D features
$\textbf{F}_{i}$ with $C$ channels using the ResNet50-FPN [14] backbone. We
then lift per-view 2D features into 3D via back projection given camera
parameters $\\{\textbf{K}_{i}\\}$ and $\\{\textbf{R}_{i}\\}$, and aggregate
them to generate 3D volume features V with the voxel size of $4cm^{3}$ by
average-pooling [19, 26, 25].

Next, we feed the 3D volume features into the 3D CNN reconstruction network in
Atlas [19], which features an encoder-decoder structure with skip connections
[23] with a $1\times 1\times 1$ convolutional head, to obtain the rough scene
TSDF S. As suggested by [19], we employ the L1 loss at three different scales
to boost the training:

$L_{recon}=\sum_{i=1}^{3}\|\mathrm{sgn}(\textbf{S}_{i})\mathrm{ln}(1+\lvert\textbf{S}_{i}\rvert)-\mathrm{sgn}(\hat{\textbf{S}}_{i})\mathrm{ln}(1+\lvert\hat{\textbf{S}}_{i}\rvert)\|_{1}$
(1)

where $\textbf{S}_{i}$ and $\hat{\textbf{S}}_{i}$ denotes the predicted and
ground truth TSDF values from coarse to fine, and we denote the predicted TSDF
with the finest scale $\textbf{S}_{3}$ as the rough scene TSDF S utilized in
the following sections.

Figure 3: The 1D illustration comparing our Ray Marching Aggregation (RMA)
method, with the Depth Aggregation method (DA) based on depth prediction, and
the Volume Aggregation method(VA) based on unprojection [25, 26, 19]. The
points depicted in the illustration represent sample points along a ray, with
their colors indicating their respective weights. The points enclosed within
one red square represent selected points.

###  3.3 Ray Marching Aggregation

Although we predict a 3D feature volume by directly averaging the lifted image
features similar to [25, 26, 19] in the reconstruction stage, volume features
are polluted from certain views since image features can vote to unobserved
space due to the lack of consideration for occlusion. An illustration of such
aggregation from view to volume is illustrated in Figure 3(c).

A straightforward solution to handle occlusion would be to directly render a
depth map and vote image features only to the surface specified by the depth
map (Figure 3(b)). However, such a voting scheme is sensitive to the quality
of the depth map produced from the reconstruction. In order to improve
robustness, we introduce a soft occlusion-aware aggregation scheme called Ray
Marching Aggregation (RMA), inspired by NeRF [18] and NeuS [30]. Specifically,
we compute the volume density given TSDF according to NeuS [30]. We sample
points on the ray of each pixel by ray marching and compute the opacity of
each point by accumulating transmittance according to NeRF [18]. As a result,
we can compute 3D features by averaging image features weighted by the
transmittance from different views. As illustrated in Figure 3(a), RMA can
vote image features into the scene by softly considering occlusions. Finally,
we extract points and aggregated features near the reconstructed surface and
pass the point cloud to the 3D detection module for object detection.

In detail, given a 2D feature map $\textbf{F}_{i}$ and corresponding camera
parameters $\textbf{K}_{i}$ and $\textbf{R}_{i}$, we assume that a ray
$\textbf{p}_{i,u,v}(t)=\textbf{o}_{i}+t\cdot\textbf{d}_{i,u,v}$ is emitted
from the camera towards the object represented by the pixel
$\textbf{F}_{i}(u,v,:)$. For convenience, we denote the ray as
$\textbf{p}(t)=\textbf{o}+t\cdot\textbf{d}$ in the following sections, where o
and d can be determined as the origin and direction of the ray given the
camera parameter and pixel location. We use ray marching to sample a set of
points $\\{\textbf{p}(t_{i})\\}$ along the ray with $t_{i}$ in increasing
order. Inspired by NeuS [30], the opacity value at the interval
$[t_{i},t_{i+1}]$ can be modeled as

$\alpha(\textbf{p}(t_{i}))=\max(\frac{\Phi(\textbf{S}(\textbf{p}(t_{i})))-\Phi(\textbf{S}(\textbf{p}(t_{i+1})))}{\Phi(\textbf{S}(\textbf{p}(t_{i})))},0),$
(2)

with $\Phi(x)$ as the sigmoid function, and $\textbf{S}(\textbf{p}(t_{i}))$,
which denotes the TSDF value of sample point $\textbf{p}(t_{i})$, is obtained
by querying the voxel closest to $\textbf{p}(t_{i})$. Then, the weight
$W(\textbf{p}(t_{i}))$ of each sampled point $\textbf{p}(t_{i})$ can be
computed according to NeRF [18] as

$W(\textbf{p}(t_{i}))=T(\textbf{p}(t_{i}))\cdot\alpha(\textbf{p}(t_{i})),$ (3)

where, $T(\textbf{p}(t_{i}))$ is the accumulated transmittance at the interval
$[0,t_{i}]$ as

$T(\textbf{p}(t_{i}))=\prod\limits_{j=0}^{i-1}(1-\alpha(\textbf{p}(t_{j})))$
(4)

We retain only those points with a weight greater than the threshold
$\theta_{rma}$.

To compute the 3D feature of each retained point, we average image features
$\textbf{F}_{i}(u,v,:)$ weighted by the opacity of the point for $i$-th image,
where $(u,v)$ is the projected location of the point to the image. As a
result, we obtain the point cloud with features P by concatenating 3D
coordinates and 3D features.

###  3.4 3D Object Detection Network

We feed the reconstructed point cloud with aggregated feature P into the 3D
detection network $\mathcal{D}$ for the final detection results. We apply
FCAF3D [24] as our detection network considering efficiency, memory
consumption, and performance.

Firstly, we transform P into sparse voxels [4] with a voxel size of $1cm^{3}$.
Then we pass these sparse voxels into FCAF3D [24] to predict the
classification scores s, bounding box regression parameters b, and 3D
centerness $c$ [27, 25, 24] of each voxel. The detection loss
$\mathcal{L}_{D}$ is the same as proposed in FCAF3D:

$L_{det}=\frac{1}{N_{pos}}\sum_{\hat{x},\hat{y},\hat{z}}(L_{cls}(\hat{\textbf{s}},\textbf{s})+m\cdot
L_{reg}(\hat{\textbf{b}},\textbf{b})+m\cdot L_{cntr}(\hat{c},c))$ (5)

Where $\\{(\hat{x},\hat{y},\hat{z})\\}$ represents the sparse voxel
coordinates, $m=\vmathbb{1}_{\\{\hat{\textbf{s}}_{\hat{x},\hat{y},\hat{z}}\neq
0\\}}$ indicates whether a sparse voxel matches an object,
$N_{pos}=\sum_{\hat{x},\hat{y},\hat{z}}m$ denotes the number of voxels
matching an object. $L_{cls}$ is the focal loss to supervise s, $L_{reg}$ is
the IOU loss for b, and $L_{cntr}$ is the binary cross-entropy loss for $c$.

###  3.5 Training Procedure

Due to the complexity of our architecture, which combines a MVS module and a
detection network, training the modules from scratch may lead to overfitting.
For example, to effectively train the detection network, it is essential to
feed the network with high-quality point clouds with features, which makes the
initialization of the reconstruction network important. Therefore, we employ a
pre-training and joint fine-tuning scheme in our training procedure to strike
a balance between the 3D reconstruction network and the 3D detection network.

Firstly, we pre-train the 2D backbone and 3D reconstruction network using only
the reconstruction loss $L_{recon}$, to fully leverage the 3D geometry.
Subsequently, we freeze the aforementioned networks and proceed to pre-train
the 3D detection network by utilizing the ray marching aggregation module and
solely considering the detection loss $L_{det}$. Finally, to obtain the
ultimate 3D detection results, we jointly fine-tune the entire network with
the total loss denoted as $L_{t}=\lambda\cdot L_{recon}+L_{det}$. Where
$\lambda$ is a constant to balance the reconstruction loss $L_{recon}$ and
detection loss $L_{det}$.

##  4 Experiments

Section 4.1 introduces the datasets, metrics, and baselines in detail. The
implementation details are introduced in Section 4.2. We compare our method
with the state-of-the-art methods in Section 4.3, where we show a clear
advantage in terms of<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Section 4.4 presents the ablation
studies, and the improvements in the detection results demonstrate the
effectiveness of our proposed ideas.

###  4.1 Datasets, Metrics, and Baselines

We evaluate our method, CN-RMA, using two indoor object detection datasets:
ScanNet [8] and ARKitScenes [1]. Following the settings of prior methods [25,
36, 28], we detect Axis-Aligned Bounding Boxes (AABB) for objects across 18
categories in ScanNet. The dataset is divided into 1201 training scans and 312
testing scans. For ARKitScenes, which comprises 4498 training scans and 549
testing scans, we detect Oriented Bounding Boxes (OBB) for objects across 17
categories. It is worth noting that the 3D geometric annotations in the
ARKitScenes dataset are relatively rough. The depth map resolution of
ARKitScenes is 192$\times$256, which is much lower compared to the
480$\times$640 resolution of ScanNet. For the evaluation metrics, we choose
the normally used mean average precision (mAP) with thresholds of 0.25 and 0.5
in both datasets. For a fair comparison, we choose the previous state-of-the-
art methods for indoor 3D object detection from multi-view images, namely
ImVoxelNet [25], NeRF-Det [36], and ImGeoNet [28], as well as the two-stage
baselines. As mentioned before, the two-stage baseline is a straightforward
combination of 3D reconstruction and 3D detection methods. Specifically, Atlas
[19] and NeuralRecon [26] are utilized to reconstruct the 3D point clouds,
while FCAF3D [24] is applied for the 3D detection.

###  4.2 Implementation Details

CN-RMA is implemented using the MMDetection3D [6] framework. We set the
feature channels $C$ to 32. The weight threshold of our aggregation approach,
$\theta_{rma}$, is set to 0.05. The loss weight $\lambda$ is set to 0.5. We
sample 300 points for each pixel in ray marching, and the maximum $t$ is set
as the diagonal length of the volume V. All experiments are conducted on 4
NVIDIA A6000 GPUs with a batch size of 1. More details are included in the
supplementary material.

Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---
ImVoxelNet [25] | 46.7 | 23.4
NeRF-Det [36] | 53.5 | 27.4
ImGeoNet [28] | 54.8 | 28.4
Atlas [19]+FCAF3D [24] | 55.4 | 33.8
NeuralRecon [26]+FCAF3D | 51.5 | 31.6
Ours (CN-RMA) | 58.6 | 36.8
Table 1<EMAIL_ADDRESS>and<EMAIL_ADDRESS>results of the ScanNet [8] dataset. We directly cite the experimental results from the ImGeoNet [28] paper. Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---
ImVoxelNet [25] | 27.3 | 4.3
NeRF-Det [36] | 39.5 | 21.9
ImGeoNet [28] | 60.2 | 43.4
Atlas [19]+FCAF3D [24] | 51.3 | 40.6
NeuralRecon [26]+FCAF3D | 36.3 | 24.9
Ours (CN-RMA) | 67.6 | 56.5
Table 2<EMAIL_ADDRESS>and<EMAIL_ADDRESS>results of the ARKitScenes [1] dataset. We directly cite the experimental results from the ImGeoNet [28] paper. Method | parameter |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---
VA | $-$ | 31.1 | 11.3
RMA | $\theta_{rma}=0.02$ | 58.6 | 37.0
RMA | $\theta_{rma}=0.05$ | 58.6 | 36.8
RMA | $\theta_{rma}=0.10$ | 57.3 | 35.4
DA | $k=1$ | 57.1 | 33.8
DA | $k=2$ | 56.9 | 34.1
DA | $k=3$ | 56.3 | 34.2
DA | $k=4$ | 57.9 | 34.7
Table 3: Ablation study results of different aggregation schemes and parameters. VA refers to the Volume Aggregation method based on unprojection [25, 19, 26]. DA denotes the Depth Aggregation method relying on depth prediction. $\theta_{rma}$ represents the weight threshold for selecting sample points in our RMA method, while $k$ denotes the number of point pairs selected in the DA method. All experiments are conducted in ScanNet [8] with our proposed parameters following our standard training steps. Training scheme |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---
Joint Train From Scratch | 48.2 | 28.8
P-MVS + JFT | 50.3 | 30.9
P-MVS + P-Det | 55.8 | 34.7
P-MVS + P-Det + JFT | 58.6 | 36.8
Table 4: Ablation study results of different training schemes. P-MVS denotes
pre-training the MVS module, P-Det denotes pre-training the detection network,
and JFT denotes jointly fine-tuning the entire network. All experiments are
conducted in ScanNet [8] with our proposed parameters.

###  4.3 Comparison

We compare our method CN-RMA with the previous state-of-the-art method and
two-stage baselines, as shown in Tables 1 and 2. Our method achieves superior
performance on both the ScanNet [8] and ARKitScenes [1] datasets,
outperforming other approaches in terms of<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Our method
surpasses the previous state-of-the-art method ImGeoNet [28] by 3.8 for
<EMAIL_ADDRESS>and 8.4 for<EMAIL_ADDRESS>in ScanNet, and 7.4 for<EMAIL_ADDRESS>and 13.1 for
<EMAIL_ADDRESS>in ARKitScenes. When compared to the two-stage baseline combining
Atlas [19] and FCAF3D [24], our method outperforms it by 3.2 for<EMAIL_ADDRESS>and
3.0 for<EMAIL_ADDRESS>in ScanNet, and 16.3 for<EMAIL_ADDRESS>and 15.9 for<EMAIL_ADDRESS>in
ARKitScenes. As shown in Figure LABEL:fig:vis_scannet_main, ImVoxelNet and
NeRF-Det often predict inaccurate bounding boxes due to insufficient
utilization of geometric information. As for two-stage baselines, it is easy
to predict inaccurate bounding boxes and miss some objects, due to noises and
incomplete scene geometry reconstructed with MVS methods. Figure
LABEL:fig:vis_arkit_main shows the detection results of the ARKitScenes
dataset. It is shown that the proposed method can also predict good results
even with relatively low-quality reconstructed geometry, demonstrating the
robustness of our method.

###  4.4 Ablation Study

In this section, we present the experimental results conducted on the ScanNet
dataset to compare various aggregation schemes with different hyper-parameters
and training schemes of our method.

####  4.4.1 Aggregation Schemes

We begin by comparing our occlusion-aware aggregation approach RMA, with the
other two schemes shown in Figure 3: Volume Aggregation (VA, Figure 3(c)) and
Depth Aggregation (DA, Figure 3(b)). The VA method lifts per-view 2D features
into 3D via back projection [26, 25, 19] directly. Specifically, we directly
convert the global feature volume V used in the 3D reconstruction network into
a point cloud with features for detection. The DA method directly lifts 2D
features to point clouds through depth maps of each view obtained from the
reconstruction results. In detail, for points along a ray, we define
$\textbf{p}(t_{i})$ as the First Hitting Point (FHP) if

$i=\mathop{\arg\min}\limits_{j}\\{j\mid\textbf{S}(\textbf{p}(t_{j}))\cdot\textbf{S}(\textbf{p}(t_{j+1}))\leq
0\\}$ (6)

which represents the first intersecting point. Considering the possible errors
in 3D reconstruction, we select $2\cdot k$ points from $\textbf{p}(t_{i-k+1})$
to $\textbf{p}(t_{i+k})$. The weight of a selected point decreases linearly as
it moves farther away from the FHP.

To study the sensitivity of the hyper-parameters, we explore different values
of $\theta_{rma}$ in our RMA module with 0.02, 0.05, and 0.10. We refrain from
testing smaller $\theta_{rma}$ values due to excessive GPU memory usage. For
DA, we experiment with different values of k, including 1, 2, 3, and 4.

Table 3 presents the results of the different aggregation schemes and hyper-
parameters. Our RMA method achieves the best performance in both<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>surpassing the VA method significantly by 27.5 in<EMAIL_ADDRESS>and 25.7
in<EMAIL_ADDRESS>The comparison reveals that integrating the rough scene TSDF
obtained from 3D reconstruction into the aggregation process effectively
enhances detection performance by providing valuable 3D geometry information
and considering occlusion. Additionally, our RMA method outperforms the best
results of DA by 0.7 in<EMAIL_ADDRESS>and 2.3 in<EMAIL_ADDRESS>It indicates that with
the possible errors in the reconstructed scene TSDF, the DA method that
directly chooses the first intersecting point along a ray may not always yield
optimal results. In contrast, our RMA method offers more flexibility by
combining local geometry information conveyed by $\alpha$ and accumulated ray
information conveyed by $T$. Moreover, the results do not change much with
different $\theta_{rma}$ indicating the robustness of our RMA method.

####  4.4.2 Training Schemes

We compare our training scheme, which involves subsequent pre-training of the
MVS module and the detection network followed by joint fine-tuning of the
entire network, with three other schemes to demonstrate the effectiveness of
our proposed approach. The straightforward training scheme is to train the
entire network from scratch without any pre-training. There are also several
possible schemes considering the pre-training and fine-tuning. Specifically,
the second scheme that we compare focuses on pre-training only the MVS module
and then jointly training the entire network. The last scheme involves pre-
training the MVS module and then freezing it to pre-train the detection
module, without joint fine-tuning of the entire network.

The comparison results presented in Table 4 demonstrate that our three-step
training scheme with pre-training and fine-tuning achieves the best
performance in both<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Our training scheme outperforms the
scheme without any pre-training by 10.4 in<EMAIL_ADDRESS>and 8.0 in<EMAIL_ADDRESS>Additionally, it outperforms the scheme without pre-training of the detection
network by 8.3 in<EMAIL_ADDRESS>and 5.9 in<EMAIL_ADDRESS>These comparisons highlight the
importance of both pre-training the MVS module and pre-training the
reconstruction network for optimal performance. This is necessary to avoid
potential overfitting caused by the complexity of our architecture and to
provide a solid geometry foundation for our RMA aggregation scheme, which
heavily relies on reliable geometry information from the reconstructed scene
TSDF. Furthermore, our training scheme outperforms the scheme without fine-
tuning by 2.8 in<EMAIL_ADDRESS>and 2.1 in<EMAIL_ADDRESS>demonstrating the effectiveness
of fine-tuning. Fine-tuning facilitates knowledge transfer and synergistic
interaction between the MVS module and the detection network, contributing to
improved performance.

Overall, our experimental results validate the efficacy of our training
scheme, emphasizing the significance of pre-training, fine-tuning, and the
interplay between the MVS module and the detection network in achieving
superior performance for our method.

##  5 Conclusion

In this paper, we introduced CN-RMA, a novel 3D indoor object detection method
from multi-view images. Our proposed approach surpasses previous state-of-the-
art methods and outperforms two-stage baselines. We also present an effective
occlusion-aware technique for aggregating 2D features into 3D point clouds
using rough scene TSDF, which holds potential for integration into other 3D
scene understanding tasks from multi-view images.

Future work should focus on exploring techniques for further improving the
performance of CN-RMA, such as investigating alternative aggregation schemes
or incorporating additional contextual information, which could be beneficial.
We anticipate continued advancements in 3D indoor object detection and related
research areas by addressing these limitations and building upon our findings.

## References

  * Baruch et al. [2021] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman.  ARKitscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_ , 2021.
  * Chen et al. [2021] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14124–14133, 2021.
  * Choe et al. [2021] Jaesung Choe, Sunghoon Im, Francois Rameau, Minjun Kang, and In So Kweon.  Volumefusion: Deep depth fusion for 3d scene reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 16086–16095, 2021.
  * Choy et al. [2019] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3075–3084, 2019.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ , 2014.
  * Contributors [2020] MMDetection3D Contributors.  MMDetection3D: OpenMMLab next-generation platform for general 3D object detection.  https://github.com/open-mmlab/mmdetection3d, 2020.
  * Curless and Levoy [1996] Brian Curless and Marc Levoy.  A volumetric method for building complex models from range images.  In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_ , pages 303–312, 1996.
  * Dai et al. [2017] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proc. Computer Vision and Pattern Recognition (CVPR), IEEE_ , 2017.
  * Gwak et al. [2020] JunYoung Gwak, Christopher Choy, and Silvio Savarese.  Generative sparse detection networks for 3d single-shot object detection.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 297–313. Springer, 2020.
  * Huang et al. [2021] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  _arXiv preprint arXiv:2112.11790_ , 2021.
  * Izadi et al. [2011] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al.  Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera.  In _Proceedings of the 24th annual ACM symposium on User interface software and technology_ , pages 559–568, 2011.
  * Lang et al. [2019] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12697–12705, 2019.
  * Li et al. [2022] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In _European conference on computer vision_ , pages 1–18. Springer, 2022.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017.
  * Liu et al. [2022] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petr: Position embedding transformation for multi-view 3d object detection.  In _European Conference on Computer Vision_ , pages 531–548. Springer, 2022.
  * Long et al. [2022] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang.  Sparseneus: Fast generalizable neural surface reconstruction from sparse views.  In _European Conference on Computer Vision_ , pages 210–227. Springer, 2022.
  * Lorensen and Cline [1998] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _Seminal graphics: pioneering efforts that shaped the field_ , pages 347–353. 1998.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Murez et al. [2020] Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich.  Atlas: End-to-end 3d scene reconstruction from posed images.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16_ , pages 414–431. Springer, 2020.
  * Qi et al. [2017] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ , 30, 2017.
  * Qi et al. [2019] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3d object detection in point clouds.  In _proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9277–9286, 2019.
  * Ren et al. [2023] Yufan Ren, Tong Zhang, Marc Pollefeys, Sabine Süsstrunk, and Fangjinhua Wang.  Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16685–16695, 2023.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Rukhovich et al. [2022a] Danila Rukhovich, Anna Vorontsova, and Anton Konushin.  Fcaf3d: Fully convolutional anchor-free 3d object detection.  In _European Conference on Computer Vision_ , pages 477–493. Springer, 2022a.
  * Rukhovich et al. [2022b] Danila Rukhovich, Anna Vorontsova, and Anton Konushin.  Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2397–2406, 2022b.
  * Sun et al. [2021] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao.  Neuralrecon: Real-time coherent 3d reconstruction from monocular video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15598–15607, 2021.
  * Tian et al. [2019] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9627–9636, 2019.
  * Tu et al. [2023] Tao Tu, Shun-Po Chuang, Yu-Lun Liu, Cheng Sun, Ke Zhang, Donna Roy, Cheng-Hao Kuo, and Min Sun.  Imgeonet: Image-induced geometry-aware voxel representation for multi-view 3d object detection.  In _Proceedings of the IEEE international conference on computer vision_ , 2023.
  * Wang et al. [2023a] Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang.  Dsvt: Dynamic sparse voxel transformer with rotated sets.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13520–13529, 2023a.
  * Wang et al. [2021a] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021a.
  * Wang et al. [2021b] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4690–4699, 2021b.
  * Wang et al. [2022a] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In _Conference on Robot Learning_ , pages 180–191. PMLR, 2022a.
  * Wang et al. [2022b] Yiqun Wang, Ivan Skorokhodov, and Peter Wonka.  Hf-neus: Improved surface reconstruction using high-frequency details.  _Advances in Neural Information Processing Systems_ , 35:1966–1978, 2022b.
  * Wang et al. [2023b] Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang.  Frustumformer: Adaptive instance-aware resampling for multi-view 3d detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5096–5105, 2023b.
  * Xie et al. [2021] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Dening Lu, Mingqiang Wei, and Jun Wang.  Venet: Voting enhancement network for 3d object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3712–3721, 2021.
  * Xu et al. [2023] Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, Ruilong Li, Jialiang Wang, Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka.  Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection.  In _ICCV_ , 2023.
  * Yao et al. [2018] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 767–783, 2018.
  * Yariv et al. [2021] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ , 34:4805–4815, 2021.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021.
  * Zhou and Tuzel [2018] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4490–4499, 2018.

Supplementary Material

##  6 Implementation Details

###  6.1 Training Details

In the pretraining stage of the Atlas [19] 2D backbone and reconstruction
network (Stage 1), we directly utilize the checkpoint provided by Atlas to
load our 2D backbone and 3D reconstruction network for ScanNet [8]. For
ARKitScenes [1], we randomly select 50 images as input for each scene and set
the voxel grid size to $(160\times 160\times 64)$. The network is trained for
80 epochs using the ADAM optimizer with a learning rate of 0.0005. In this
stage, we perform data augmentation by applying random translations and
rotations to the scenes following Atlas.

In the pretraining stage of the FCAF3D [24] detection network (Stage 2), we
generate point clouds with features for each scene using our RMA method and
use these point clouds to train the detection network. For ScanNet, we use 50
images and set the voxel grid size to $(256\times 256\times 96)$ for
generating the point clouds. For ARKitScenes, we use 40 images and set the
voxel grid size to $(192\times 192\times 80)$. We train the network for 12
epochs on both datasets, repeating the dataset 10 times for ScanNet and 3
times for ARKitScenes. The ADAM optimizer is employed with an initial learning
rate of 0.001 and weight decay of 0.0001. The learning rate decreases on the
8th and 11th epochs. In this stage, we random sample $N_{pt}=500000$ points
for each scene, and perform data augmentation by randomly translating,
rotating, flipping, and scaling the input point clouds, following FCAF3D.

In the joint fine-tuning stage (Stage 3), we use 40 images and set the voxel
grid size to $(192\times 192\times 80)$. The ADAM optimizer is used with an
initial learning rate of 0.001 and weight decay of 0.0001. We fine-tune the
ScanNet network for 100 epochs, with the learning rate decreasing on the 80th
epoch. For ARKitScenes, we fine-tune the network for 40 epochs, with the
learning rate decreasing on the 27th and 36th epochs. In this stage, we do not
conduct random data augmentation on input camera parameters and ground truth
TSDF. Instead, after obtaining point clouds with features, we random sample
$N_{pt}=500000$ points for each scene, and perform data augmentation by
randomly translating, rotating, flipping, and scaling the input point clouds,
following FCAF3D.

For validation, we employ 50 images and set the grid size to $(256\times
256\times 96)$ for ScanNet, and we use 40 images and set the grid size to
$(192\times 192\times 80)$ for ARKitScenes.

###  6.2 Experiment Details

Since the code has not been made publicly available, we directly reference the
experimental results from the ImGeoNet [28] paper.

For experiments on ScanNet, we utilize the provided pre-trained networks of
ImVoxelNet [25], NeRF-Det [36], Atlas [19], and NeuralRecon [26]. For
baselines with retraining, we retrain the FCAF3D [24] network for 12 epochs
using the reconstructed scene point clouds, repeating the dataset 10 times.

For experiments on ARKitScenes, we train the Atlas network for 80 epochs and
the NeuralRecon network for 15 epochs. We train the ImVoxelNet network for 12
epochs, repeating the dataset 9 times. We train the NeRF-Det network for 12
epochs without repeating the dataset. For baselines with retraining, we
retrain the FCAF3D network for 12 epochs using the reconstructed scene point
clouds, repeating the dataset 3 times.

##  7 Additional Experiment Results

The results of Table 5 illustrate that retraining the reconstruction network
of the two-stage baseline with reconstructed scene point clouds is better than
directly using the network trained by ground truth point clouds. The per-
category<EMAIL_ADDRESS>and<EMAIL_ADDRESS>scores of experiments on the ScanNet [8] and
ARKitScenes [1] datasets are shown in Tables 6, 7, 8, and 9. Additional
visualization results of ScanNet and ARKitScenes are presented in Figures
LABEL:fig:vis_scannet_supp and LABEL:fig:vis_arkit_supp.

Method | Dataset | Retrain |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---
Atlas [19]+FCAF3D [24] |  ScanNet[8] | ✓ | 55.4 | 33.8
Atlas+FCAF3D | ScanNet |  | 39.4 | 22.1
NeuralRecon [26]+FCAF3D | ScanNet | ✓ | 51.5 | 31.6
NeuralRecon+FCAF3D | ScanNet |  | 29.6 | 13.1
Atlas+FCAF3D |  ARKitScenes [1] | ✓ | 51.3 | 40.7
Atlas+FCAF3D | ARKitScenes |  | 34.1 | 25.8
NeuralRecon+FCAF3D | ARKitScenes | ✓ | 36.3 | 24.9
NeuralRecon+FCAF3D | ARKitScenes |  | 13.6 | 9.1
Table 5: Comparison of retraining for two-stage baselines. Method | cab | bed | chair | sofa | tabl | door | wind | bkshf | pic | cntr | desk | curt | fridg | showr | toil | sink | bath | ofurn | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ImVoxelNet [25] | 31.7 | 83.4 | 71.8 | 67.2 | 55.1 | 31.7 | 15.2 | 36.8 | 2.0 | 33.0 | 63.2 | 24.0 | 53.0 | 20.0 | 91.3 | 53.2 | 76.1 | 32.3 | 46.7
NeRF-Det [36] | 42.3 | 84.6 | 75.9 | 78.5 | 56.3 | 33.4 | 21.4 | 49.9 | 2.4 | 50.6 | 73.9 | 21.3 | 54.3 | 62.5 | 90.9 | 57.7 | 75.5 | 32.3 | 53.5
ImGeoNet [28] | 38.7 | 86.5 | 76.6 | 75.7 | 59.3 | 42.0 | 28.1 | 59.2 | 4.3 | 42.8 | 71.5 | 36.9 | 51.8 | 44.1 | 95.2 | 58.0 | 79.6 | 36.8 | 54.8
Atlas+FCAF | 41.6 | 85.4 | 80.2 | 81.6 | 54.7 | 38.3 | 27.3 | 50.1 | 7.6 | 58.9 | 73.3 | 16.8 | 36.6 | 61.9 | 94.4 | 58.8 | 92.3 | 37.4 | 55.4
Neucon+FCAF | 38.7 | 82.2 | 78.3 | 81.4 | 56.2 | 30.5 | 12.5 | 42.1 | 6.0 | 54.2 | 64.6 | 20.8 | 34.6 | 41.3 | 89.1 | 67.8 | 89.1 | 37.3 | 51.5
Ours(CN-RMA) | 42.3 | 80.0 | 79.4 | 83.1 | 55.2 | 44.0 | 30.6 | 53.6 | 8.8 | 65.0 | 70.0 | 44.9 | 44.0 | 55.2 | 95.4 | 68.1 | 86.1 | 49.7 | 58.6
Table 6: Per-category<EMAIL_ADDRESS>scores for 18 categories from the ScanNet [8] dataset. Atlas+FCAF denotes the two-stage baseline combining Atlas [19] and FCAF3D [24], and Neucon+FCAF denotes the two-stage baseline combining NeuralRecon [26] and FCAF3D. We directly cite the experimental results from the ImGeoNet [28] paper. Method | cab | bed | chair | sofa | tabl | door | wind | bkshf | pic | cntr | desk | curt | fridg | showr | toil | sink | bath | ofurn | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ImVoxelNet [25] | 10.2 | 71.5 | 37.2 | 32.8 | 33.0 | 4.3 | 0.8 | 11.7 | 0.2 | 4.8 | 34.0 | 5.9 | 16.1 | 2.1 | 73.0 | 20.5 | 50.8 | 11.9 | 23.4
NeRF-Det [36] | 15.8 | 73.1 | 45.3 | 40.6 | 39.5 | 8.1 | 2.0 | 20.3 | 0.2 | 13.8 | 42.5 | 5.3 | 25.3 | 10.0 | 63.0 | 26.0 | 49.1 | 12.7 | 27.4
ImGeoNet [28] | 14.3 | 74.2 | 47.4 | 46.9 | 41.0 | 8.1 | 2.0 | 26.9 | 0.5 | 6.6 | 44.7 | 4.4 | 28.2 | 3.9 | 71.0 | 25.9 | 48.3 | 17.2 | 28.4
Atlas+FCAF | 20.3 | 74.8 | 47.9 | 65.0 | 44.0 | 9.7 | 5.0 | 37.0 | 1.1 | 25.3 | 51.4 | 3.5 | 23.4 | 3.0 | 69.3 | 31.8 | 74.1 | 22.6 | 33.8
Neucon+FCAF | 15.8 | 74.7 | 45.6 | 68.8 | 43.2 | 8.0 | 3.5 | 26.1 | 1.2 | 15.8 | 40.4 | 1.3 | 21.9 | 1.6 | 74.4 | 28.3 | 77.8 | 21.3 | 31.6
Ours(CN-RMA) | 21.3 | 69.2 | 52.4 | 63.5 | 42.9 | 11.1 | 6.5 | 40.0 | 1.2 | 24.9 | 51.4 | 19.6 | 33.0 | 6.6 | 73.3 | 36.1 | 76.4 | 31.5 | 36.8
Table 7: Per-category<EMAIL_ADDRESS>scores for 18 categories from the ScanNet [8] dataset. Atlas+FCAF denotes the two-stage baseline combining Atlas [19] and FCAF3D [24], and Neucon+FCAF denotes the two-stage baseline combining NeuralRecon [26] and FCAF3D. We directly cite the experimental results from the ImGeoNet [28] paper. Method | cab | fridg | shlf | stove | bed | sink | wshr | tolt | bthtb | oven | dshwshr | frplce | stool | chr | tble | TV | sofa | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ImVoxelNet [25] | 20.7 | 33.3 | 13.5 | 4.3 | 57.7 | 25.8 | 53.8 | 65.2 | 66.0 | 25.5 | 2.2 | 0.2 | 2.5 | 26.7 | 24.5 | 0.0 | 41.6 | 27.3
NeRF-Det [36] | 34.7 | 61.1 | 30.7 | 9.4 | 73.2 | 29.9 | 62.6 | 77.2 | 86.4 | 45.0 | 7.4 | 2.1 | 12.1 | 46.4 | 38.3 | 0.1 | 55.5 | 39.5
ImGeoNet [28] | 55.8 | 82.6 | 48.4 | 20.4 | 89.3 | 52.8 | 80.0 | 92.5 | 94.7 | 66.0 | 18.1 | 68.8 | 30.6 | 72.3 | 70.3 | 2.2 | 79.0 | 60.2
Atlas+FCAF | 56.0 | 62.7 | 20.9 | 19.5 | 88.4 | 60.7 | 53.1 | 89.7 | 94.9 | 42.4 | 3.4 | 48.7 | 23.2 | 63.0 | 62.2 | 2.2 | 80.3 | 51.3
Neucon+FCAF | 51.2 | 79.5 | 19.6 | 14.8 | 62.5 | 46.0 | 39.6 | 41.2 | 56.5 | 29.0 | 7.1 | 46.9 | 10.4 | 37.1 | 33.2 | 2.8 | 40.6 | 36.3
Ours(CN-RMA) | 73.5 | 82.3 | 47.7 | 37.4 | 91.6 | 74.5 | 78.0 | 93.3 | 93.3 | 74.6 | 53.6 | 67.2 | 35.9 | 73.1 | 72.8 | 14.5 | 85.1 | 67.6
Table 8: Per-category<EMAIL_ADDRESS>scores for 17 categories from the ARKitScenes [1] dataset. Atlas+FCAF denotes the two-stage baseline combining Atlas [19] and FCAF3D [24], and Neucon+FCAF denotes the two-stage baseline combining NeuralRecon [26] and FCAF3D. We directly cite the experimental results from the ImGeoNet [28] paper. Method | cab | fridg | shlf | stove | bed | sink | wshr | tolt | bthtb | oven | dshwshr | frplce | stool | chr | tble | TV | sofa | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ImVoxelNet [25] | 3.3 | 14.3 | 0.7 | 0.0 | 20.5 | 5.2 | 27.5 | 36.3 | 20.4 | 4.9 | 2.2 | 0.0 | 0.4 | 4.8 | 4.1 | 0.0 | 5.1 | 8.8
NeRF-Det [36] | 10.8 | 48.0 | 5.7 | 0.6 | 36.1 | 7.9 | 46.3 | 60.8 | 64.9 | 21.0 | 5.6 | 0.0 | 2.9 | 18.8 | 14.1 | 0.0 | 28.2 | 21.9
ImGeoNet [28] | 31.8 | 72.5 | 21.7 | 3.9 | 83.3 | 19.9 | 71.2 | 84.8 | 91.0 | 44.4 | 15.9 | 23.1 | 13.3 | 49.3 | 45.1 | 0.1 | 67.2 | 43.4
Atlas+FCAF | 37.1 | 62.2 | 8.4 | 7.6 | 83.3 | 30.9 | 47.6 | 81.6 | 94.1 | 29.2 | 3.4 | 19.2 | 17.2 | 44.9 | 50.5 | 0.0 | 75.0 | 40.7
Neucon+FCAF | 30.3 | 74.2 | 8.4 | 6.0 | 50.4 | 15.5 | 30.1 | 28.8 | 49.5 | 17.9 | 5.4 | 16.4 | 6.6 | 24.2 | 26.9 | 0.0 | 32.1 | 24.9
Ours(CN-RMA) | 56.0 | 79.8 | 27.8 | 21.3 | 87.4 | 51.6 | 75.9 | 89.1 | 92.2 | 60.8 | 53.3 | 40.6 | 25.1 | 60.5 | 60.1 | 1.2 | 77.3 | 56.5
Table 9: Per-category<EMAIL_ADDRESS>scores for 17 categories from the ARKitScenes [1]
dataset. Atlas+FCAF denotes the two-stage baseline combining Atlas [19] and
FCAF3D [24], and Neucon+FCAF denotes the two-stage baseline combining
NeuralRecon [26] and FCAF3D. We directly cite the experimental results from
the ImGeoNet [28] paper.


# An Optimization of the FPGA Based Wavelet
Trigger in Radio Detection of Cosmic Rays

Zbigniew Szadkowski, for the Pierre Auger Collaboration Manuscript received
June 03, 2014.Zbigniew Szadkowski is with the University of Łódź, Department
of Physics and Applied Informatics, Faculty of High-Energy Astrophysics,
90-236 Łódź, Poland, (e-mail: zszadkow @kfd2.phys.uni.lodz.pl, phone: +48 42
635 56 59).

###### Abstract

Experiments that observe coherent radio emission from extensive air showers
induced by ultra-high energy cosmic rays are designed for a detailed study of
the development of the electromagnetic part of air showers. Radio detectors
can operate with 100% up time as e.g. surface detectors based on water-
Cherenkov tanks. They are being developed for ground-based experiments (e.g.
the Pierre Auger Observatory) as another type of air shower detector in
addition to the fluorescence detectors, which operate with only $\sim$10% of
duty in dark nights. The radio signals from air showers are caused by the
coherent emission due to geomagnetic radiation and charge excess processes.
Currently used self-triggers in radio detectors often generate a dense stream
of data, which is analyzed afterwards. Huge amounts of registered data
requires a significant man-power for the off-line analysis. An improvement of
the trigger efficiency becomes a relevant factor. The wavelet trigger, which
investigates on-line a power of radio signals ($\mathrm{\sim V^{2}/R}$) is
promising with respect to current designs.

In this work, Morlet wavelets with various scaling factors were used for an
analysis of real data from the Auger Engineering Radio Array (AERA) and for an
optimization of the utilization of the resources in an FPGA. The wavelet
analysis showed that the power of events is concentrated mostly in a limited
range of the frequency spectrum (consistent with a range imposed by the input
analog band-pass filter). However, we found several events with suspicious
spectral characteristics, where the signal power is spread over the full band-
width sampled by a 200 MHz digitizer with significant contribution of very
high and very low frequencies. These events may not origin from cosmic ray
showers but can be human-made contaminations. The engine of the wavelet
analysis can be implemented into the modern powerful FPGA and can remove
suspicious events on-line to reduce the trigger rate.

###### Index Terms:

Pierre Auger Observatory, trigger, FPGA, DCT.

##  I Introduction

Results from various cosmic rays experiments located on the ground level,
point to the need for very large aperture detection systems for ultra-high
energy cosmic rays. With its nearly 100% duty cycle, its high angular
resolution, and its sensitivity to the longitudinal air-shower evolution, the
radio technique is particularly well-suited for detection of Ultra High-Energy
Cosmic Rays (UHECRs) in large-scale arrays. The present challenges are to
understand the emission mechanisms and the features of the radio signal, and
to develop an adequate measuring instrument. Electron-positron pairs generated
in the shower development are separated and deflected by the Earth magnetic
field, hence introduce an electromagnetic emission [1][2]. During shower
development, charged particles are concentrated in a shower disk of a few
meters thickness. This results in a coherent radio emission up to about 100
MHz. Short but coherent radio pulses of 10 ns up to a few 100 ns duration are
generated with an electric field strength increasing approximately linearly
with the energy of the primary cosmic particle inducing the extended air
showers (EAS), i.e. a quadratic dependence of the radio pulse energy vs.
primary particle energy. In contrast to the fluorescence technique with a duty
cycle of about 12% (fluorescence detectors can work only during moonless
nights), the radio technique allows nearly full-time measurements and long
range observations due to the high transparency of the air to radio signals in
the investigated frequency range.

The radio detection technique will be complementary to the water Cherenkov
detectors and allows a more precise study of the electromagnetic part of air
showers in the atmosphere. In addition to a strong physics motivation, many
technical aspects relating to the efficiency, saturation effects and dynamic
range, the precision for timing, the stability of the hardware developed,
deployed and used, as well as the data collecting and system-health monitoring
processes will be studied and optimized.

One of the currently developed technique is an radio signals power estimation
based on the wavelet transforms. The main motivation of a development based on
much more sophisticated algorithms is that the efficiency of the radio self-
trigger is often very low. Mostly registered events contain only noise. The
significant improvement of the trigger efficiency is the crucial factor. A lot
of data analyzing off-line requires non-negligible amount of man-power. Much
wiser approach is much more efficient trigger. The presented wavelet trigger
is an alternative proposal to the currently operating algorithms.

##  II Timing and Limitations

The radio signal is spread in time interval of an order of couple hundred
nanoseconds, most of registered samples gave a time interval below 150 ns. The
frequency window for the maximal antenna efficiency is $\sim$30 - 80 MHz. This
range is additionally filtered by a band-pass filter. According to the Nyquist
theorem the sampling frequency should be twice higher than the maximal
frequency in a investigated spectrum. The anti-aliasing filter should have the
cut-off frequency of $\sim$85 MHz. Taking into account some width of the
transition range for the filter (from pass-band to stop-band) the final
sampling frequency should not be lower than 170 MHz (200 MHz in our
considerations). This frequency corresponds to 5 ns between rising edges of
the clock. The widths of the radio pulses $\sim$100 ns are consistent with the
theoretical estimations [3], [4] and the LOPES data [5], [6]. The width of
$\sim$100 ns by the 200 MSps sampling corresponds to 20 time bins. A minimal
length of the FFT routine covering the pulse width equals 32.

Only a single FFT32 routine for on-line calculation of Fourier coefficients
for data is needed. Fourier coefficients for various wavelets can be
calculated earlier and be available for final the power estimation as
constants. Nevertheless, we also consider 16-point FFT routine to minimize a
FPGA resource occupancy and to increase a safety margin for a registered
performance. The Quartus® II environment for the Altera® FPGA programming
provides parameterized FFT routines with various architectures: streaming,
variable streaming, burst and buffered burst. For the variable streaming
provides also fixed or floating-point algorithms with natural or bit-reversed
order. However, all routines deliver the FFT coefficients in a serial form. No
any Altera® routine allows calculating all FFT coefficients simultaneously.

##  III Wavelets

The wavelet transform can be used to analyze time series that contain non-
stationary power at many different frequencies [7]. Let us investigate a time
series X, with values of $x_{n}$, at time index n. Each value is separated in
time by a constant time interval $\Delta$t. The wavelet transform $W_{n}(s)$
is just the inner product (or convolution) of the wavelet function $\psi$ with
our original time series:

$W_{n}(s)=\sum\limits_{m=0}^{N-1}x_{m}\psi^{*}\left[\frac{(m-n)\Delta
t}{s}\right]$ (1)

where s denotes the frequency scale and the asterisk (*) complex conjugate,
respectively. A variation of the wavelet scale s and a transition along the
localized time index n allow a construction of a dependence showing both the
amplitude of any features vs. the scale and how this amplitude varies with
time. Although it is possible to calculate the wavelet transform according to
(1), it is considerably faster to do the calculations in the Fourier space.

To approximate the continuous wavelet transform, the convolution (1) should be
done N times for each scale, where N is the number of points in the time
series. However, the choice of doing all N convolutions is arbitrary, we can
decrease N to a smaller number to reduce computation time if the precision is
satisfactory. By choosing N points, the convolution theorem allows performing
all N convolutions simultaneously in the Fourier space using the discrete
Fourier transform (DFT). The DFT of $x_{n}$ is

$X_{k}=\frac{1}{N}\sum\limits_{n=0}^{N-1}x_{n}e^{-2\pi kn/N}$ (2)

where k = 0,…,N - 1 is the frequency index. In the continuous limit, the
Fourier transform of a function $\psi(t/s)$ is given by $\Psi(s\omega)$. By
the convolution theorem, the wavelet transform is the inverse Fourier
transform of the product:

$W_{n}(s)=\sum\limits_{k=0}^{N-1}\bar{X}_{k}\bar{\Psi}^{*}(s\omega_{k})e^{i\omega_{k}n\Delta
t}$ (3)

where the angular frequency is defined as

$\displaystyle\omega_{k}=\left\\{\begin{array}[]{cccccc}2\pi k/(N\Delta
t)&,&k\leq N/2&&&\\\ -2\pi k/(N\Delta t)&,&k>N/2&&&\end{array}\right.$ (6)

Using (3) and a standard Fourier transform routine, one can calculate the
continuous wavelet transform (for a given s) at all n simultaneously. Wavelets
coefficients allow an estimation of signal power. The global wavelet power
spectrum is defined as $|W_{n}(s)|^{2}$ [8] and the total signal power
$\bar{W}^{2}$ can be expressed as follows:

$\sum\limits_{j=0}^{M-1}|W_{j}|^{2}=\frac{1}{N}\sum\limits_{j=0}^{M-1}\sum\limits_{k=0}^{N-1}|W_{k,j}|^{2}=\frac{1}{N}\sum\limits_{j=0}^{M-1}\sum\limits_{k=0}^{N-1}|\bar{X}_{k}\times\bar{\Psi}_{k,j}|^{2}$
(7)

A sum of products of Fourier coefficients calculated in a 32-point Fast
Fourier Transform (FFT32) routine for ADC data data ($x_{n}$) in each clock
cycle with pre-calculated Fourier coefficients of the reference wavelet gives
an estimation of the power for selected types of the wavelet. The on-line
power analysis requires simultaneous calculations of the power for many (i.e.
m) wavelets with the various frequency scale s.

We are considering two-dimensional analysis of the spectrogram containing a
distribution of partial power contributions $|W_{k}(s)|^{2}$ for the Fourier
index k vs. m scaled wavelets. The on-line analysis is simplified and strongly
depends on available FPGA resources. Such an analysis of the radio signals in
real time for high frequency sampling is a challenge and it requires very
efficient algorithm implementation into the FPGA. Fortunately, variable
precision DSP blocks in the newest Altera Cyclone V family offer high-
performance, power-optimized, and fully registered multiplication operations
for 9-bit, 18-bit, and 27-bit word lengths.

##  IV 32-point FFT Algorithm

Let us consider the N-point DFT $\bar{X}$

$\bar{X}_{k=0,\ldots,N-1}=\sum\limits_{n=0}^{N-1}x_{n}e^{-2i\pi kn/N}$ (8)

where the $x_{n}$, as samples from an ADC chip, are real. The formula (8) can
be split into two or more parts by rearranging the sum and indices. The
standard approach for formula simplification is through a Radix-2 Decimation-
in-Time (DiT) or Decimation-in-Frequency algorithm (DiF).

For Radix-2 DiT, we get :

$\bar{X}_{k}=\sum\limits_{n=0}^{\frac{N}{2}-1}x_{2n}e^{-i\frac{2\pi
kn}{N/2}}+e^{-i\frac{2\pi
k}{N}}\sum\limits_{n=0}^{\frac{N}{2}-1}x_{2n+1}e^{-i\frac{2\pi kn}{N/2}}=$ (9)
$=DFT_{\frac{N}{2}}[x_{0},x_{2},...,x_{N-2}]+W^{k}_{N}\times
DFT_{\frac{N}{2}}[x_{1},x_{3},...,x_{N-1}]$

The N-point DFT can be easily split into two N/2-point transforms. Outputs
from DFT procedures are complex. So, a calculation of final DFT coefficients
by using a DiT algorithm requires complex multiplication for final merging of
data from parallel DFT procedures with lower order, i.e., multiplication of
twiddle factors $W_{N}^{k}=e^{-i\frac{2\pi k}{N}}$.

Altera® provides a library routine of a complex multiplication in the FPGA.
However, a 16x16 bit operation requires 6 DSP embedded 9x9 multipliers, even
in the most economical (canonical) mode. Generally, a complex multiplication
in the FPGA is rather resource-intensive and if possible it should be replaced
by a multiplication by real variables.

For Radix-2 DiF, we get :

$\bar{X}_{2k}=\sum\limits_{n=0}^{N-1}x_{n}W^{2kn}_{N}=DFT_{\frac{N}{2}}\left[x_{n}+x_{n+\frac{N}{2}}\right]$
(10)
$\bar{X}_{2k+1}=\sum\limits_{n=0}^{N-1}x_{n}W^{(2k+1)n}_{N}=DFT_{\frac{N}{2}}\left[\left(x_{n}-x_{n+\frac{N}{2}}\right)W^{n}_{N}\right]$

The standard Radix-2 DIF algorithm rearranges the DFT equation (8) into two
parts: computation of the even-numbered discrete-frequency indices
$\bar{X(k)}$ for k=[0,2,4,$\ldots$], and computation of the odd-numbered
indices k=[1,3,5,$\ldots$]. This corresponds to splitting an N-point DFT into
two k = N/2-point routines. The first corresponding twiddle factor is
$e^{-i\frac{2\pi}{N}\frac{N}{2}}=-1$. The first operations are simple sums and
subtractions of real variables. Each operation related to the consecutive
twiddle factor will be performed in a single clock cycle.

The algorithm of Decimation in Frequency used for the 32-point DFT allows
splitting eq. 8 as follows:

$\bar{X}_{k=2p}=\sum\limits_{n=0}^{15}A_{n}e^{-i\pi kn/8}\hskip
14.22636pt\Rightarrow\hskip 14.22636ptFFT16_{even}$ (11)
$\bar{X}_{k=2p+1}=\sum\limits_{n=0}^{15}A_{n+16}e^{-i\pi(2p+1)n/16}$ (12)

where for n = 0,1,…,15

$A_{n}=x_{n}+x_{n+16}\hskip 14.22636ptA_{n+16}=x_{n}-x_{n+16}$ (13)

The next twiddle factors are:

$\displaystyle W_{B}=e^{-i\pi/2}=-i\hskip
19.91692ptW_{C}=e^{-i\pi/4}=\gamma(1-i)$ (14) $\displaystyle
W_{D}=e^{-i\pi/8}=\alpha-i\beta\hskip 8.53581ptW_{E}=e^{-i\pi/16}=\xi-i\eta$
(15) $\displaystyle W_{F}=e^{-3i\pi/16}=\sigma-i\rho$ (16)

where

$\displaystyle\gamma$ $\displaystyle=$ $\displaystyle cos(\pi/4)\hskip
8.53581pt\alpha=cos(\pi/8)\hskip 8.53581pt\beta=sin(\pi/8)$ (17)
$\displaystyle\xi$ $\displaystyle=$ $\displaystyle cos(\pi/16)\hskip
17.07164pt\eta=sin(\pi/16)$ (18) $\displaystyle\sigma$ $\displaystyle=$
$\displaystyle cos(3\pi/16)\hskip 11.38109pt\rho=sin(3\pi/16)$ (19)

The DiF algorithm and further optimizations take into account only FFT
coefficients with indices k = 0,…,15. Due to real input data ($x_{0,...31}$),
the higher FFT coefficients have well known symmetry :
$Re\bar{X}_{32-n}=Re\bar{X}_{n}$ and $Im\bar{X}_{32-n}=-Im\bar{X}_{n}$
($n>0$). Calculation of the $\bar{X}_{0,...15}$ according to the pure Radix-2
DiF algorithm requires 8 pipeline stages. For $\bar{X}_{0,4,8,12,16}$, 2
pipeline stages are necessary only for synchronization.

According to eq. (11) all $\bar{X}_{0,2,4,...,14}$ with even indices could be
calculated by the algorithm presented in [9]. Variables $x_{n}$ in Fig. 2 in
[9] would be replaced by $A_{n}$ according to eq. (13). An application of a
modified algorithm reduces the number of $9\times 9$ multipliers from 12 to 10
only, and shorten the pipeline chain on stages (the last 2 stages are simple
registers for synchronization).

For the odd indices, stages $B$ and $C$ for k=16,…,19 and k = 24,…27 are pure
delay lines, while for neighboring indices k=20,…,23 and k = 28,…31,
mathematical operations are performed in a cascade. Let us multiply the
$A_{16,...19}$ and $A_{24,...27}$ by the factor $\lambda={\gamma}^{-1}$. Then
to adjust variables in the $C$ stage for odd FFT coefficients (for k =
20,21,22,33 and k = 28,29,30,31), we set

$C_{k}=\lambda\times\gamma=B_{k}$ (20)

So, by such a redefinition, the $C$ stage for odd FFT indices becomes a pure
pipeline stage. It can be removed with one pipeline stage for the even FFT
indices. In order to come back to the correct values, the coefficients in the
$F$ stage can be simple redefined as:

$\displaystyle\alpha^{\prime}$ $\displaystyle=$
$\displaystyle\gamma\times\alpha\hskip
14.22636pt\beta^{\prime}=\gamma\times\beta\hskip
14.22636pt\xi^{\prime}=\gamma\times\xi$ (21) $\displaystyle\sigma^{\prime}$
$\displaystyle=$ $\displaystyle\gamma\times\sigma\hskip
17.07164pt\rho^{\prime}=\gamma\times\rho\hskip
17.07164pt\eta^{\prime}=\gamma\times\eta$ (22)

but for indices k = 16,20,24,28 we have to use 4 additional multipliers.
Nevertheless, at this cost we save one pipeline stage, and depending on the
width of buses in the final FFT coefficients, we save at least 1000 logic
elements.

Figure 1: An optimized structure with a reduced of a single pipeline stage at
the cost of only 4 additional multipliers (8 DSP $9\times 9$ blocks).

##  V Wavelet Power Calculation

Two families of reference Morlet wavelets were selected for the scaling factor
$s^{-1}=\alpha$ = 0.04, 0.004, respectively:

$f_{\alpha,freq}(k)=cos\left(2\pi
k\frac{freq_{wavelet}}{sampling}\right)exp(-\alpha\cdot k^{2})$ (23)

where -16$\leq$k$\leq$15, $sampling$ = 200 (MHz) [10].

Figure 2: An energy distribution as a function of time and frequency given for
scaling factors $\alpha$ = 0.04 (1st graph) and $\alpha$ = 0.004 (2nd and 3rd)
for typical events (1st and 2nd) as well as for a suspicious event (3rd
graph).

Wavelets for $\alpha$ = 0.04 are squeezed to non-negligible values in a range
of -10$\leq$k$\leq$10\. For $\alpha$ = 0.004, wavelets are stretched to a
larger range beyond -16$\leq$k$\leq$15\. $|\bar{X}_{k}|$ distributions are
relatively wide for $\alpha$ = 0.04; a separation between frequency ranges is
limited. Much better separation is for $\alpha$ = 0.004. This value reduces
more a leakage between frequency bins. It was selected for preliminary
estimation of the signal power, although it corresponds to longer stretch of
time than the typical length of 150 ns and wavelets in the interval of 150 ns
do not vanish completely as for $\alpha$ = 0.01.

A sum of products of Fourier coefficients ($\bar{X}_{k}$), calculated in the
32-point Fast Fourier Transform (FFT32) routine for ADC data ($x_{n}$) (in
each clock cycle), with pre-calculated Fourier coefficients of the reference
wavelet ($\bar{\Psi}_{k}$), gives an estimation of the power for selected
types of the wavelet (Eq. 7). The on-line power analysis requires simultaneous
calculations of the power for many wavelets with the given scaling factor
$s=\alpha^{-1}$. The data acquisition system is equipped in the band-pass
filter with lower and higher cut-off frequencies at 30 and at 80 MHz,
respectively. Fig. 2a and Fig. 2b show an energy distribution for a typical
radio event. The energy is cumulated mostly in a frequency range of
40$\leq$freq$\leq$70 MHz. Fig. 2a is calculated for the scaling factor
$\alpha$ = 0.04 and presents a rough structure of the energy distribution.
This is due to an overlap of the wavelet Fourier distributions and spectral
leakage between neighboring frequency sectors. For the scaling factor $\alpha$
= 0.004 the energy distribution is reconstructed with a very high precision
(see Fig. 2b).

Figure 3: A histogram of frequency indices when maximum of energy appears
given for 77594 AERA events. Energy distribution for each time bin of an event
is scaled to get maximal value of 100%. The histogram is built from time bins
of events corresponding to a maximal signals.

The analysis of AERA data allowed an identification of several ”suspicious”
events with significantly different spectral characteristics (Fig. 2c) from
”typical” ones (Fig. 2b). Fig. 3 shows a distribution of energy vs. frequency.
Energy is concentrated mostly in an expected frequency range 30 - 80 MHz (in
agreement with a characteristic of the band-pass filter). As an estimator for
a selection of untypical events we defined a spectral energy contributions
(for each reference wavelet) averaged for 32 or 16 time bins around the
maximal signal. Next, the spectral distortion estimator (SDE) is calculated as
follows:

$SDE=\sum\limits_{k=k_{5MHz}}^{k_{low}-1}|W_{k}|^{2}+\sum\limits_{k=k_{high}}^{k_{100MHz}}|W_{k}|^{2}-\sum\limits_{k=k_{low}}^{k_{high}}|W_{k}|^{2}$
(24)

SDE $\geq$ 0 selects events with a significant energy contribution in
peripheral energy ranges (traces marked as ”suspicious”). Table I shows that
the SDE rate for many widths of peripheral bands (calculated for 77594 AERA
events). Conservatively calculated variants A and B show negligible amount of
events. However, for more realistic conditions taking into account borders of
analog band-pass filter, the rate reaches few percents.

The energy contributions for the reference wavelets shown in Table I are
calculated as follows (denoted with SEL=1 or 0, respectively):

  * •

$|W_{j}|^{2}=\sum\limits_{k=0}^{N-1}|\bar{X}_{k}\times\bar{\Psi}_{k,j}|^{2}$
SEL=1

  * •

$|W_{j}|^{2}=|\bar{X}_{k}\times\bar{\Psi}_{k,k}|^{2}$ SEL=0

with averaging over 32 or 16 samples (AVG=32 or 16) for two scaling factors
$\alpha$ = 0.004 and $\alpha$ = 0.0001, respectively.

For all configurations there is no any event corresponding to A conditions
(sum of energy in peripheral bands : 6.25-12.5 MHz + 97.75-100 MHz). However,
for bands (6.25-31.25 MHz + 81.25-100 MHz) which should be cut-off by the
analog band-pass filter (variant D), few percents of ”untypical” events were
found.

TABLE I: A rate of ”suspicious” for SDE $\geq$ 0 low | SEL | 6.25-12.5 | 6.25-18.75 | 6.25-25 | 6.25-31.25
---|---|---|---|---|---
high | /AVG | 93.75-100 | 87.5-100 | 81.25-100 | 81.25-100
variant |  | A | B | C | D
rate | 1-32 | 0.000% | 0.005% | 0.14% | 1.74%
| 1-16 | 0.000% | 0.006% | 0.15% | 1.78%
$\alpha$ = | 0-32 | 0.000% | 0.015% | 0.22% | 3.05%
0.004 | 0-16 | 0.000% | 0.019% | 0.23% | 3.11%
rate | 1-32 | 0.000% | 0.005% | 0.14% | 1.78%
| 1-16 | 0.000% | 0.009% | 0.15% | 1.80%
$\alpha$= | 0-32 | 0.000% | 0.017% | 0.23% | 3.10%
0.0001 | 0-16 | 0.000% | 0.021% | 0.24% | 3.15%

Table I shows that discrepancies a) by averaging on 32 or 16 samples as well
as b) scaling factors $\alpha$ = 0.004 or 0.0001, are negligible. Averaging on
16 samples simplifies the FPGA code. By the scaling factor $\alpha$ = 0.0001
the spectral leakage between neighboring wavelet coefficients is lower than
for $\alpha$ = 0.004 and the energy contributions for the reference wavelets
can be calculated by a simple product $|\bar{X}_{k}|\times C_{k,k}$ (SEL=0,
instead of the sum (SEL=1). An elimination of sums significantly simplifies
the FPGA code.

Number of events with maximal signal in frequency ranges [6.25-25 MHz] and
[81.25-100 MHz] is 1057 (1.36 %). This gives the same level as for the variant
D. We estimate that $\sim$1-3% of events have ”untypical” spectral
characteristics. Analyzed data are recorded directly from the ADC (are
unfiltered). Due to the analog band-pass filter events like in Fig. 2c,
actually, should not appear in a data stream. Their origin up to now is not
clear. If they correspond to same rare signal structures an on-line spectral
analysis and spectral trigger may support a recognition of some new phenomena.
Modern FPGAs allows an implementation of more and more sophisticated
algorithms.

Figure 4: The Fourier spectrum for 16 reference wavelets scaled for
$\alpha=0.0001$

##  VI FPGA implementation

The graphs in Fig. 2 are generated for 255 channels (255 reference wavelets).
This amount is certainly too large for an implementation in the FPGA. The
elementary component from Eq. (7) is a product of two complex numbers: Fourier
coefficients ($\bar{X}_{k}$) of real data ($x_{n}$) and Fourier coefficients
of the reference wavelet ($\bar{\Psi}_{k}$). The simplest way to calculate
this product on-line in the FPGA is usage the Altera® IP routine
ALTMULT_COMPLEX. However, the wavelet spectral coefficients are predefined as
constants for arbitrarily selected reference wavelets. A module of a complex
product $|\bar{X}_{k}\cdot\bar{\Psi}_{k,j}|^{2}$ can be calculated as a real
product of
$|\bar{X}_{k}|^{2}\cdot|\bar{\Psi}_{k,j}|^{2}=|\bar{X}_{k}|^{2}\cdot C_{k,j}$
(where $C_{k,j}$ are constants - previously pre-calculated wavelet
coefficients). We need calculate on-line only $|\bar{X}|^{2}$ and next
multiply by constant factors.

Fig. 4 shows the Fourier spectrum for only 16 reference wavelets scaled for a
possible big resolution ($\alpha=0.0001$). The power contribution to the sum
in Eq. 7 comes in fact from a single index - k. The contribution from
neighboring k-1 and k+1 indices are less than 2-3 %. Thus, the wavelet engine
can use much less DSP blocks focusing on only a single index which gives a
fundamental contribution to the total power. Practically, it means that
$|W_{k}|^{2}=|X_{k}|^{2}$ for all indices except the maximal one, which should
be scaled by the factor of 23,74611/11,73 $\sim$ 2\. A multiplication by a
factor of 2 in the FPGA is accomplished by a connection of the bus to the next
pipeline stage $S^{stage+1}[BUS..1]=S^{stage}[BUS-1..0]$.

AERA uses 14-bit ADCs. In the algorithm from Fig.1 each next pipeline stage
extends a width of data bus on a single bit. Outputs of $\bar{X}_{Re}$ and
$\bar{X}_{Im}$ have 20-bit buses. This provides a negligible approximation
errors. A maximal error of 8 ADC-counts by 20-bit resolution corresponds to
the error of 0.008%. However, a square of 20-bit variable would require 4 DSP
blocks. A reduction of a bus width to 16-bit only decreases the DSP blocks
utilization to 2 DSP per module, however at the cost of an accuracy.
Differences between exact and FPGA calculated modules reaches 1.5-2% but only
for less than 1% of events.

##  VII Trigger

Spectral energy contributions $|W_{k,j}|^{2}$ (Eq. 7) are integrated by boxcar
[11] averaging 16 samples to get $|W_{k}|^{2}$ (Eq. 24). Results from Table I
justify a shorter boxcar chain for FPGA code simplification and resources
economy.

If the MSB (sign bit) of the SDE equals to 0 it means the analyzing events has
an untypical spectral characteristic. Trigger is generated. Data is
continuously written into the left port of the dual-port RAM. In order to
catch the profile of the event, trigger is delayed in RAM-based shift register
on 512 clock cycles. Thus data is frozen in 1024-word DPRAM. The profile is
next read by the NIOS processor and sent via UART to the external host (PC).

##  VIII IIR-notch filter

The main purpose of the IIR (Infinite Impulse Response) notch filter [12] in
the radio detector is to increase the signal-to-noise ratio before triggering.
The previously implemented filter based on the FFT approach [13] was power
consuming and has been replaced by the IIR-notch one. The IIR-filters operate
on the time-domain and the output of the filter ($y_{i}$) is a linear
combination of the input samples ($x_{j}$) and the delayed feedback output
samples ($y_{j}$):

$\displaystyle y_{i}=x_{i}$ $\displaystyle-$
$\displaystyle(2\cos\omega_{N}\cdot x_{i-1})+x_{i-2}+$ (25) $\displaystyle+$
$\displaystyle(2r\cos\omega_{N}\cdot y_{i-1})-(r^{2}\cdot y_{i-2})$

The value of $r$ is to be between 0 and 1, where a value close to 1 indicates
a very narrow transmitter. A typical value chosen is $r=0.99$. $\omega_{N}$ is
the normalized filter frequency, which is determined by the notch frequency
$f_{N}$ and the sampling frequency $f_{S}$ as $\omega_{N}=(2\pi f_{N})/f_{S}$.

Figure 5: Excitations appearing in two different IIR-notch filters for test
frequencies 55.2 MHz and 25.39 MHz, respectively. The upper graph shows a zoom
of excitation for 55.20 MHz filter. The bottom graph shows a relatively often
excitations for 25.39 MHz of the base frequency. Let us notice that
excitations have huge amplitude, 4 timer larger than input signals.

Generally, the IIR filter may contain unstable regions, due to internal
feedbacks. The IIR-filter characteristics are optimized to cut-off narrow
bands corresponding to radio transmitters operating in region of radio
detectors. Band reject regions should be narrow enough in order not to affect
real cosmic ray signals. On the other hand very narrow filters are liable to
potential instabilities. Radio stations contain 4-stage filters implemented in
a cascade way suppressing 4 narrow bands. We have simulated responses of the
filter for the following band-reject frequencies: 27.12, 40.9, 55.2 and 70.7
MHz, respectively, with r = 0.99. Unfortunately, we found in Quartus
simulations instabilities for some configurations. Fig. 5a shows an excitation
on the output of the 3rd filter (55.2 MHz cut-off) for signal packages driving
the filter cascade. For 55.2 MHz filter excitations appear relatively rarely,
however, if occurs it may generate a spurious trigger. For others frequencies
we observed much higher rate (Fig. 5b).

Figure 6: Spectrograms of ADC test input (top) and IIR output (bottom).  TABLE II:  Excitations evidence vs. compiler parameters Effort | compiler | version | system | $f_{max}$ | exc. | exc.
---|---|---|---|---|---|---
level |  |  |  |  | sim | mea
Normal | 9.1sp2 | Web free | XP 32-Bit | 232.94 | YES | NO
Extra | 9.1sp2 | Web free | XP 32-Bit | 236.69 | NO | NO
Normal | 9.1 | Full paid | XP 32-Bit | 232.94 | YES | NO
Extra | 9.1 | Full paid | XP 32-Bit | 236.69 | NO | NO
Normal | 9.1 | Full paid | W7 64-Bit | 236.69 | NO | NO
Extra | 9.1 | Full paid | W7 64-Bit | 237.02 | NO | NO

A wavelet analysis of the IIR data output shows a dramatic change of spectral
characteristic for signals with excitations. A power concentrated originally
close to 25 MHz (Fig. 6a) spreads over much higher frequency range (Fig. 6b).
Spectral characteristics of excited events (Fig. 6b) are similar to
”suspicious” real events (Fig. 2c). At first we suspected that ”suspicious”
events in AERA database are registered by spurious triggers generated by IIR-
notch filter for disadvantageous correlations between the filter cut-off and a
basic signal frequency. But a careful analysis showed that analyzed events
came only from unfiltered ADC traces. The ”suspicious” event could not be
generated by a filter. So, the origin of the strange spectral characteristics
remained still mysterious.

Next steps showed that excitations depend on switches in the compiler and its
version! Originally, excitations were found in simulations for Quartus 9.1 sp2
free version on the XP 32-Bit OS. Simulation for other versions or systems did
not confirm an evidence of any excitations. Finally, we found that excitations
appear for ”Normal” position in Effort level (Setting option) for 32-Bit
compilers working on XP OS. 64-Bit version running on Windows 7 (64-Bit) does
not generate spurious filter responses (Table II). Nevertheless, the
registered performance for ”strange” configurations is high enough (a safety
margin is larger than 10%) to provide a stable internal data propagations
without any potential hazards or races.

Fortunately, an evidence of excitations has not been confirmed in the
laboratory tests for on the Cyclone III FPGA with sof generated by the Quartus
9.1 sp2 and 9.1 versions, independently free or paid. Problems suggested by
Quartus simulations did not occur in reality (Table II).

##  IX Laboratory tests

###  IX-A Excitations

As sources of artificial events we used:

  1. 1.

the IIR-notch filter implemented in FPGAs of two Altera development kits:

    1. (a)

with Cyclone III EP3C120F780C7 supported by ICB-HSMC daughter card with RS232
serial port allowing a connection between the NIOS and the PC, Cyclone III was
programmed by the sof file generated by Quartus 9.1 sp2 on 32-bit XP OS,

    2. (b)

with Cyclone V E 5CEFA7F31I7 supported by Altera HSMC-ADC-BRIDGE and Texas
Instr. ADS4249EVM (Evaluation Module - with double channel 14-bit ADC with
max. 250 MHz sampling), Cyclone V was programmed by the sof file generated by
the licensed version of Quartus 13.1 on Windows7 (64-bit OS)

  2. 2.

arbitrary pattern generator Tektronix AFG3252C driving the chain b)

When the wavelet engine recognized a signal with untypical spectral
characteristic a generated trigger frozen 1024 samples of averaged
$\bar{X}_{k=1,...15}$ in Dual-Port RAM embedded memory of 1024 words which
next were transmitted to the PC by a NIOS processor via RS232 serial port.
Amplitudes of input test vectors were lower than 2048 ADC-units. Cyclone III
was driven by the test vector stored in the embedded ROM, Cyclone V was driven
from the arbitrary pattern generator Tektronix AFG3252C (pattern of 128 kwords
of 14-bit data or burst signals) via ADS4249EVM and HSMC-ADC-BRIDGE. Cyclone V
supports On Chip Termination (OCT), Cyclone III does not. The Cyclone III
development kit was not equipped in termination resistors. Trigger was set:

  * •

just after the notch filter recognizing possible ”sparks” with an amplitude
much higher than maximal input signal - excitations from Fig. 5b),

  * •

after the wavelet engine when SDE output indicated a negative values.

Fortunately, we did not find excitations in Cyclone III test setup with sof
from Quartus 9.1 (also sp2). Spurious events were not found both by amplitude
comparator just after the notch filter as well as by the wavelet trigger
analyzing an energy contribution in frequency domain. According expectations,
no any spurious events were found in the Cyclone V test platform. Selected
patterns of real AERA events (with untypical spectral characteristics) driving
the FPGA from the arbitrary pattern generator AFG3252C generated expected
rates of wavelet trigger. Trigger rate for patterns from AFG3252C wired OR
with noise from 33250A remains on the same level in a relatively wide range of
additional noise.

Figure 7: Spectral characteristics measured a test pulse on Cyclone V
development kit driven from AFG3252C with an additional noise from 33250A.

###  IX-B Spectral characteristics

We tested also an influence of the noise on spectral characteristics short
pulses typical for radio events (50.62 MHz sine wave enveloped by a Gauss
bell). Even for a noise level comparable with a level of signal (Fig. 7a) the
signal spectral contribution is clearly visible and easy to detect. However,
for 3 times higher noise (Fig. 7b) we observe a dramatic change of spectra
practically making impossible a signal recognition. Let us see that for higher
signals (with the same signal to noise ratio - Fig. 7c) contributions for very
low and very high frequencies are lower in comparison to relatively low
signals, when quantization of signals additionally affect the spectra.

Practically, the input amplifiers seem to be configured with higher
amplification factors agreeing on relatively high (but white) noise than to
use low level quantized ADC signal significantly affected by digitization
processes. It means, the amplitude of the signal should not be too low, a
contributed noise reduces a errors coming from quantization processes.

The AHDL code implemented into Cyclone V 5CEFA7F31I7 FPGA has high enough
register performance to provide 15% safety margin (with 200 MHz sampling).

##  X Conclusions

An evidence of untypical spectral characteristic in real AERA events was a
motivation to develop a FPGA data processing block allowing on-line a
recognition of non-standard events. The origin of these events maybe ”natural”
(real new phenomena) or ”apparatus” (i.e. oscillations in filters or
differential nonlinearities of ADCs). Such a tool for sure helps to reject
spurious triggers and, as more important, may help to detect new category of
events or phenomena.

Laboratory tests confirmed in real time data processing

  * •

an efficiency of the wavelet trigger recognizing event with untypical spectral
characteristics,

  * •

no instabilities in IIR-notch filters for sof files generated by all tested
versions of Quartus compilers.

The presented above FPGA block can support standard triggers allowing a
rejection of suspicious or too noisy events and a suppression of a database
size.

## Acknowledgment

Acknowledgment: This work is being developed for the next generation of cosmic
ray detectors supported by the ASPERA-2 consortium and was funded by the
Polish National Center of Researches and Development under NCBiR Grant No.
ERA/NET/ASPERA/02/11 and by the National Science Centre (Poland) under NCN
Grant No. 2013/08/M/ST9/00322

The author would like to thank Charles Timmermans from Radboud University
Nijmegen, the Netherlands, for files with AERA radio events.

## References

  * [1] H. R. Allan, “Radio emission from extensive air showers”, _Prog. in Elem. Part. and Cos. Ray Phys._ , vol. 10, pp. 171, 1971.
  * [2] H. Falcke, P.W. Gorham, “Detecting radio emission from cosmic ray air showers and neutrinos with a digital radio telescope”,_Astropart. Phys._ vol. 19, pp. 477-494, July 2003.
  * [3] T. Huege, H. Falcke, Radio emission from cosmic ray air showers: Simulation results and parametrization, _Astropart. Phys._ vol. 24, Issues 1-2, pp. 116–136, Sept. 2005.
  * [4] T. Huege, R. Urlich, R. Engel, Monte Carlo simulations of geo-synchrotron radio emission from CORSIKA-simulated air showers, _Astropart. Phys._ vol. 27, Issue 5, pp. 392-405, June 2007.
  * [5] J. Petrovic et. al, Radio emission of highly inclined cosmic ray air showers measured with LOPES, _Astronomy & Astrophys._, 462, pp. 389-395, 2007.
  * [6] The LOPES Collaboration, Progress in air shower radio measurements: Detection of distant events, _Astropart. Phys._ vol. 26, Issues 4-5, pp. 332-340, Dec. 2006.
  * [7] I. Daubechies, The wavelet transform time-frequency localization and signal analysis, _IEEE Trans. Inform. Theory_ , vol. 36, pp. 961–1004, 1990.
  * [8] C. Torrence, G.P. Compo, A Practical Guide to Wavelet Analysis, _Bulletin of the American Meteorological Society_ , 79, pp. 61-78, 1998.
  * [9] Z. Szadkowski, 16-point discrete Fourier transform based on the Radix-2 FFT algorithm implemented into cyclone FPGA as the UHECR trigger for horizontal air showers in the Pierre Auger Observatory, _Nucl. Instr. Meth. A_ , vol. 560, Issue 2, pp. 309-316, May 2006,
  * [10] Z. Szadkowski, FPGA Based Wavelet Trigger in Radio Detection of Cosmic Rays, _Brazilian Journal of Physics_ , 2014 (in publication),
  * [11] H. Gemmeke, A. Grindler, H. Keim, M. Kleifges, N. Kunka, Z. Szadkowski, D. Tcherniakhovski, Design of the Trigger System for the Auger Fluorescence Detector, _IEEE Trans. on Nucl. Science_ , vol. 47, Issue: 2, pp. 371-375 Part: 1, Apr. 2000.
  * [12] J. L. Kelley, Data acquisition, triggering, and filtering at the Auger Engineering Radio Array, _Nucl. Instr. Meth. A_ , vol. 725, pp. 133-136, Oct. 2013,
  * [13] A. Schmidt, H. Gemmeke, A. Haungs, K-H. Kampert, C. Rühle, Z. Szadkowski, FPGA Based Signal-Processing for Radio Detection of Cosmic Rays _IEEE Trans. on Nucl. Science_ , vol. 58, pp. 1621-1627, Aug. 2011