11institutetext:  Université de Lyon, Lyon 2
ERIC EA3083
11email<EMAIL_ADDRESS>
# New Datasets and a Benchmark of Document Network Embedding Methods for
Scientific Expert Finding

Robin Brochier 11 Antoine Gourru 11 Adrien Guille 11 Julien Velcin 11

###### Abstract

The scientific literature is growing faster than ever. Finding an expert in a
particular scientific domain has never been as hard as today because of the
increasing amount of publications and because of the ever growing diversity of
expertise fields. To tackle this challenge, automatic expert finding
algorithms rely on the vast scientific heterogeneous network to match textual
queries with potential expert candidates. In this direction, document network
embedding methods seem to be an ideal choice for building representations of
the scientific literature. Citation and authorship links contain major
complementary information to the textual content of the publications. In this
paper, we propose a benchmark for expert finding in document networks by
leveraging data extracted from a scientific citation network and three
scientific question & answer websites. We compare the performances of several
algorithms on these different sources of data and further study the
applicability of embedding methods on an expert finding task.

††footnotetext: Copyright © 2020 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0). BIR
2020, 14 April 2020, Lisbon, Portugal.

##  1 Introduction

Many tools offer to search and filter the vast data sources available on the
Web. In particular, there is a multitude of platforms directed to the
scientific community. From the simple search engine for publications to the
social network for researchers, all consume and produce valuable data for
searching scientific content of interest. Expert finding is one the the most
challenging problem that finds application in both academia and the industry.
To tackle this challenge, recent advances in document network embedding (DNE)
has the potential to inspire new unsupervised models that can deal with the
heterogeneous network of documents of the scientific literature. However, the
design of such efficient algorithms heavily depends on the development of
strong evaluation frameworks.

In this paper, we propose a methodology and provide 4 datasets that extend the
limited scope of expertise retrieval evaluation frameworks. Furthermore, we
provide experiment results computed with unsupervised methods and we extend
document network embedding algorithms to this specific task.

Our contributions are the following:

  * •

we provide 4 datasets for expert finding extracted from a scientific
publication network and three question & answer (Q&A) websites and make them
publicly available 111https://github.com/brochier/expert_finding;

  * •

we describe an evaluation methodology based on the ranking of expert
candidates given a set of labeled document queries;

  * •

we report experiment results that give some insights on this expert finding
task;

  * •

we explore and analyze the use of state-of-the-art document network embedding
algorithms for expert finding and we show that further research is needed to
bridge the gap between DNE methods and expert finding.

The rest of the paper is organized as follows. In Section 2, we survey related
works. We detail in Section 3 our evaluation methodology, the datasets we
extracted, the evaluation measures and the algorithms we use. In Section 4 we
show and analyze the results of our experiments. Finally, in Section 5, we
discuss our findings and provide future directions.

##  2 Related Works

In this section, we first present a formal definition for expert finding. Then
we present algorithms of the literature that address expert finding. Finally,
we describe recent methods for document network embedding that have the
potential to deal with this particular task.

###  2.1 Formal definition of expert finding

The concept of expert finding can cover a large range of tasks. The main
principle behind expertise retrieval is the search for candidates given a
query. To match these two, an algorithm will be provided with some data to
link the output space, a ranking of candidates, with the input space, which is
often a textual content. However, many different types of data can be
considered to address this challenge. To fairly compare algorithms, we choose
a fixed structure for the data which reflects common use cases. Furthermore,
if supervised methods benefit from labeled fields of expertise associated with
the candidates, they are beyond the scope of this paper which focuses on
unsupervised methods only. Our goal is to compare methods that do not require
sometimes costly annotations.

Early works in expert search [8] usually consider a small set of topical
queries. The direct namings of these topics are used to retrieve a list of
candidates by leveraging a collection of documents they published (e.g.,
emails, scientific papers). This type of evaluation is used across several
public datasets [26, 15, 16].

More recently, the concept of expert finding has been merged into the wider
concept of entity retrieval [1]. As more and more complex data are produced on
the Web, expert finding becomes a particular application of entity search. At
the same time, Q&A websites such as Stack Overflow
222https://stackoverflow.com/ generate and make publicly accessible a big
amount of questions with expert answers, collaboratively curated by their
users. Several works address the search for experts in such websites [20, 28].
Often, the task consists in either finding the exact list of users who
answered a specific question or ranking the answers according to the user
votes. In the first case, the task involves considering the evolution of the
users across time and, in the second case, the task involves understanding the
intrinsic quality of a written answer. Nevertheless, [25] reviews several
models for expert finding in Q&A websites. Their experiments show that matrix
factorization-based methods perform better than tree based and ranking based
methods.

In this paper, we adopt the document-query methodology recently proposed in
[3]. The expert search is performed given a set of queries that are particular
textual instances of some expert topics (or fields of expertise). Given a
query, an algorithm should rank first the candidates that are associated to
the same fields of expertise. We provide 4 datasets for which we annotated
experts and document queries. Each dataset consists in candidates and
documents linked by authorship relations (candidate-document e.g. authorship)
and by response relations (document-document e.g citation or answer). A query
is therefore one of the documents (e.g. a scientific paper or a question) for
which we aim to retrieve some experts of the topics depicted in it. This
configuration reflects many real case scenarios such as (1) the automatic
search for scientific reviewers, (2) the recommendation of expert users in Q&A
websites or even (3) the retrieval of interesting profiles for job offers.

###  2.2 Algorithms for expert finding

Numerous works have addressed automatic expertise retrieval. We describe here
the main approaches and some interesting recent methods. P@noptic Expert [7]
creates meta-documents for a candidate by concatenating the contents of all
documents she produced. In this manner, ranking the candidates given a query
becomes a similarity search between the query representation and the meta-
documents representations. A voting model [14] computes the similarities
between the query and the documents. The algorithm then aggregates these
scores at the candidate level by using a fusion technique such as the
reciprocal rank [27]. A propagation model [21] takes advantage of the links
between candidates and documents to propagate the similarities between the
query and the documents. Using random walks with restart [17], the iterative
propagation of the scores converges in a few steps to a stationary
distribution over the candidates. WISER [6] models each candidate as a small,
weighted, sub-graph of the Wikipedia Knowledge Graph. Information derived from
these graphs and traditional document retrieval techniques are combined to
identify experts w.r.t a query. Note that methods leveraging external data are
out of the scope of our benchmark. LT Expertfinder is an evaluation framework
for expert finding [11] based on an interactive tool. It integrates various
existing algorithms (such as [1]) in a user-friendly way. The underlying
corpus used by this tool is the ACL Anthology Network. However, it does not
include a well-established ground truth to assess who are the experts. Indeed,
the evaluation is purely done in an online manner since the user has to
evaluate the degree of expertise based on several features, such as author’s
citations, h-index, keywords, etc. Recent works [23, 9] propose ad hoc
embedding techniques, whereas, in this work, we’re interested in measuring the
performance of conventional network embedding techniques.

###  2.3 Document network embedding

Network embedding [19, 12] provides an efficient approach to represent nodes
in a low dimensional vector space, suitable for solving various machine
learning tasks. Recent techniques extend NE for document networks. Text-
Associated DeepWalk (TADW) [24] extends DeepWalk to deal with textual
attributes. Yang et al. prove, following the work in [13], that Skip-Gram with
hierarchical softmax can be equivalently formulated as a matrix factorization
problem. TADW then consists in constraining the factorization problem with a
pre-computed representation of the documents by using Latent Semantic Analysis
(LSA) [10]. Graph2Gauss (G2G) [2] is an approach that embeds each node as a
Gaussian distribution instead of a vector. The algorithm is trained by passing
node attributes through a non-linear transformation via a deep neural network
(encoder). GVNR-t [4] is a matrix factorization approach for document network
embedding, inspired by GloVe [18], that simultaneously learns word, node and
document representations by optimizing a least-square objective over a co-
occurrence matrix of the nodes constructed by truncated random walks. IDNE [5]
introduces a Topic-Word Attention mechanism, trained from the connections of a
document network, to represent documents as mixtures of topics.

DNE algorithms do not directly apply to expert finding data since they are not
designed to handle multiple types of nodes, in particular candidate nodes. In
this paper, we show (1) two methods to extend their applicability to the task
of expert finding and (2) the impact of their representations when they are
used as document representations for traditional expert finding algorithms.

##  3 Evaluation Methodology

We present in this section the evaluation methodology that we follow to access
the performances of several algorithms for expert finding. We first describe
the task we seek to solve, then we describe the datasets that we extracted and
explain how we annotated them in order to access the quality of the
algorithms’ outputs. Finally, we detail the models used in our experiments.

###  3.1 Ranking expert candidates from document queries

Expert finding is a complex task that can be formalized in multiple ways.
Early works define this task as a ranking problem given several topic-queries
where the naming of these topics are directly used as queries to retrieve the
expert candidates. However, in many real world applications, a user is asked
to provide a specific and detailed query. In a Q&A website for instance, a
user usually exposes the problem she faces in full detail and does not
necessarily know the exact naming of the fields of expertise needed to solve
her problem. Furthermore, querying an algorithm with a small set of topic-
queries can lead to poor evaluation measures due to the usually small number
of fields of expertise associated with the dataset. For this reasons, we
follow the document-query evaluation methodology proposed in [3] by processing
4 datasets for which a set of document-queries is manually annotated.

The expert finding task in this paper is a ranking problem. Given a document
labeled with a ground truth set of fields of expertise, an algorithm is
queried to rank a set of candidates, among which a subset of experts are
associated with the same set of labels. The data provided to the algorithms
consists in a corpus of $n_{d}$ documents $D$, $n_{c}$ candidates $C$, a
network of authorship with adjacency matrix $A_{dc}\in\mathbb{N}^{n_{d}\times
n_{c}}$ and a network of documents with adjacency matrix
$A_{dd}\in\mathbb{N}^{n_{d}\times n_{d}}$. Figure 1 shows an hypothetical
dataset used in this paper. The ranking is performed in an unsupervised
setting, that is, no ground truth labels of expertise are given to the
algorithms. The set of labeled documents (the queries) can be smaller than
$n_{d}$ and the set of labeled candidates (experts) can be smaller than
$n_{c}$ (i.e not all documents and candidates are labeled).

$D_{1}$$D_{2}$$D_{3}$$D_{4}$$D_{5}$$D_{6}$$C_{1}$$C_{2}$$C_{3}$$C_{4}$$C_{5}$$A_{dc}\text{~{}~{}}\Big{\\{}$$A_{dd}\text{~{}~{}}\Big{\\{}$$D\text{~{}~{}~{}~{}~{}~{}}\Big{\\{}$$C\text{~{}~{}~{}~{}~{}}\Big{\\{}$Expertise
labels Figure 1: Hypothetical example of an expert finding dataset we use in
this paper. 5 candidates are authors of 6 documents. The 6 documents are
connected to each other by citation in a scientific corpus, or by answer in a
same post in a Q&A website. Among the candidates, 3 are known to be experts in
stars and/or in circles. 4 documents are associated to these 2 fields of
expertise as well. In our evaluation methodology, we query an algorithm with
these 4 documents and expect a ranking of candidates that will match each
document’s fields of expertise. As an example, a perfect algorithm might
generate the rankings $D_{1}\mapsto C_{3}C_{4}C_{5}C_{1}C_{2}$ and
$D_{6}\mapsto C_{4}C_{5}C_{3}C_{2}C_{1}$.

To evaluate the candidate scores provided by the algorithms, we compare the
resulting rankings with the ground truth fields of expertise. If a document is
associated with three different labels, we expect the algorithm to rank first
all experts associated to at least one of these labels. We report the area
under the ROC curve (AUC), the precision at 10 (P@10) and the average
precision (AP) and we compute their standard deviation along the queries. That
is, we evaluate the robustness of the algorithms against the variety of
document-queries.

###  3.2 Datasets

We consider 4 datasets. The first one is an extract of DBLP [22] in which a
list 199 experts in 7 fields are annotated [26] by human judgments
333https://lfs.aminer.cn/lab-datasets/expertfinding/##expert-list. Our dataset
only considers the annotated experts and the other candidates that are close
in the co-authorship network which explains the relatively small size of our
network compared to the original one. In addition to the expert annotations,
our evaluation framework requires document annotations since we adopt the
document-query methodology for expertise retrieval. We asked two PhD students
in computer science to associate independently 20 randomly drawn documents per
field of expertise (140 in total). Then, only the labels on which the two
annotators agreed were kept, leaving 114 annotated papers. The mean Cohen’s
kappa coefficient across the labels is $0.718$. An advantage of our
methodology is that we can evaluate the algorithms on more queries (114
documents) than the traditional method (7 labels). This allows us to assess
the robustness of the algorithms by computing the standard deviations of the
ranking metrics along all queries. However, one might suggest that these 7
labels do not reflect a representative set of expertise as there are too
broad. For this reason, we seek for a wider granularity of expertise by the
use of well-know question & answer website.

If scientific publication networks are easy to find on the Web, scientific
expertise annotations are rarely available for both authors and publications.
We use data downloaded in June 2019 from Stack Exchange
444https://archive.org/details/stackexchange to create datasets for expert
finding collected from three communities closely related to research. Academia
555https://academia.stackexchange.com/ is dedicated to academics and higher
education. Mathoverflow 666https://mathoverflow.net/ gathers professional
mathematicians and is widely used by researchers. Stats
777https://stats.stackexchange.com/ (also known as Cross Validated) addresses
statistics, machine learning and data mining issues. For each dataset, we
first keep questions with at least 10 user votes that have at least one answer
with 10 user votes or more. We build the networks by linking questions with
their answers and by linking answers with the users who published them. The
field of expertise are the tags associated with the questions. Only the tags
that occur at least 50 times are kept. We annotate an expert with the tags of
a question if her answer to that question received at least 10 votes. Note
that the tags are first provided by the users who ask the questions but they
are thereafter verified by experimented users.

The general properties of our 4 datasets are presented in Table 1. The
annotations and the preprocessed datasets are made publicly available.

Table 1: General properties of the datasets. | # candidates | # documents | # labels | # experts | # queries | label example
---|---|---|---|---|---|---
DBLP | 707 | 1641 | 7 | 199 | 114 | ’information_extraction’
Stats | 5765 | 14834 | 59 | 5765 | 3966 | ’maximum-likelihood’
Academia | 6030 | 20799 | 55 | 6030 | 4214 | ’recommendation-letter’
Mathoverflow | 7382 | 38532 | 98 | 7382 | 10614 | ’galois-representations’

###  3.3 Algorithms

We run the experiments with 4 baseline algorithms and 4 document network
embedding algorithms. The laters are adapted with two aggregation schemes in
order to deal with the candidates since they are primarily designed for
document network only. These aggregations are arbitrary and are voluntarily
the most straightforward way to run DNE algorithms on bipartite networks of
authors-documents. We further discuss these choices in section 4.

####  3.3.1 Baselines

We run the experiments with the same models as in [3], using the tf-idf
representations and the cosine similarity measure. Also, we add a random model
to have reference metrics:

  * •

Random model: we randomly draw scores between 0 and 1 for each candidate;

  * •

P@noptic model [7]: we concatenate the textual content of each document
associated to the candidates, use their tf-idf representations and compute the
cosine similarity to produce the scores;

  * •

Voting model [14]: we use the reciprocal rank to aggregate the scores at the
candidate level;

  * •

Propagation model [21]: we concatenate the two adjacency matrices $A_{dc}$ and
$A_{dd}$ to construct a transition matrix between candidates and documents
such that $A=\left(\begin{array}[]{cc}A_{dd}&A_{dc}\\\
A_{dc}^{\intercal}&0\end{array}\right)$. The initial scores are the cosine
similarities between the tf-idf representations of the query and the
documents. The scores are propagated iteratively until convergence with a
restart probability of $0.5$.

We also run the voting and propagation models using document representations
produced by IDNE in place of the tf-idf vectors. The document network provided
to IDNE has adjacency matrix $A_{d}=A_{dc}A_{dc}^{\intercal}+A_{dd}$.

####  3.3.2 Extending DNE algorithms for expert finding

DNE methods usually operate in networks of documents, with no candidate nodes.
To apply them in the context of expert finding, we propose two straightforward
approaches:

  * •

pre-aggregation: as in the P@noptic model, meta-documents are generated by
aggregating the documents produced by each candidates. Furthermore, an
adjacency matrix of a meta-network between candidates and documents is
constructed. We compute a candidate network as
$A_{c}=A_{dc}^{\intercal}A_{dc}$ and a document network as
$A_{d}=A_{dc}A_{dc}^{\intercal}+A_{dd}$. The meta-network is then
$A=\left(\begin{array}[]{cc}A_{d}&A_{dc}\\\
A_{dc}^{\intercal}&A_{c}\end{array}\right)$. The candidate and document
representations are then generated by treating this meta-network and the
concatenation of the documents and meta-documents as an ordinary instance of
document network. From this meta-network, we generate representations with the
DNE algorithms. The scores of the candidates are generated by cosine
similarity between the representation of the document-query and the
representations of the candidates;

  * •

post-aggregation: in this setting, we first train the DNE algorithm on the
network of documents defined by $A_{d}=A_{dc}A_{dc}^{\intercal}+A_{dd}$. Once
the representations are generated for all documents, a representation for a
candidate is computed by averaging the vectors of all documents associated to
her. The scores are then computed by cosine similarity.

We run the experiments with 4 document network embedding algorithms, using the
authors’ implementations. For all methods, the dimension of the
representations is set to 256:

  * •

TADW [24]: we follow the original paper by using 20 iterations and a penalty
term $\lambda=0.2$;

  * •

GVNR-t [4]: we use $\gamma=10$ random walks of length $t=40$, a sliding window
of size $l=5$ and a threshold $x_{\text{min}}=2$ with 4 iterations;

  * •

Graph2gauss (G2G) [2]: we make sure the loss function converges before the
maximum number of iterations;

  * •

IDNE [5]: we run all experiments with $n_{t}=32$ topic vectors with 5000
balanced mini-batches of 16 positive samples and 16 negative samples.

##  4 Experiment Results

Tables 2 to 5 show the experiments results. In the following, we analyze the
performances of the aggregation scheme against the baseline algorithms, we
highlight the interesting results obtained when using the baselines with pre-
computed document representations with a DNE algorithm and finally we make
some observations on the differences between the datasets. Note that the
implementation of TADW, provided by the authors, could not scale to
Mathoverflow.

###  4.1 Baselines versus DNE aggregation schemes

For all datasets, the propagation model performs generally better than the
other algorithms, particularly in terms of precision. Both aggregation schemes
yield to poor results and none of these two methods appear to be better than
the other. GVNR-t is the best algorithm among the document network embedding
models. We believe that, if DNE algorithms are well suited for document
network representation learning, the gap between simple tasks such as node
classification and link prediction and the task of expert finding is too big
for our naive aggregation schemes to perform well. Especially, the network
structure changes significantly between an homogeneous network and an
heterogeneous network. Moreover, expert finding algorithms often benefit from
information about the centrality of the candidates and documents. DNE
algorithms do not particularly preserve this information neither do our
aggregation schemes.

###  4.2 Using DNE as document representations for the baselines

Since the baseline algorithms perform well, we study the possibility to apply
them using a DNE algorithm for the representations of the documents. We only
report the results with the representations computed with IDNE but we observe
the same behaviors with other DNE models. First, these representations
constantly improve the voting model, which achieves best results in terms of
AUC on Stats and Mathoverflow. Then, the most surprising effect is the
significant decrease of performance of the propagation model. If the precision
for the first ranked candidates is not affected, the AUC score significantly
drops for the three Q&A datasets. We believe that document network embeddings
captures too long-range dependencies between the documents in the network,
which are then subsequently exaggerated by the propagation procedure. Figure 2
shows the effect of the representations used with the propagation model on the
ROC curve.

(a) Propagation model with tf-idf representations: the curve has a nice shape
which means the ranking of candidates are good even for the last ranked
experts.

(b) Propagation model with IDNE representations: the first ranked candidates
are good but the algorithm tends to wrongly rank last many true experts.

Figure 2: Effect of IDNE representations on the propagation model. Using
document network embeddings significantly damages the rankings.

###  4.3 Differences between the datasets

The results achieved by the algorithms on all three Stack Exchange datasets
are consistent. However, they do not behave the same with DBLP. First, DNE
methods get closer scores to the baselines on DBLP. In the Q&A datasets, the
interactions are more isolated i.e. there are more users having fewer
interactions. This difference of network properties might disadvantage DNE
methods who are usually trained on scale-free networks whose degree
distribution follows a power law. Moreover, the propagation method does not
suffer with DBLP from the decrease of performance induced by the IDNE
representations. We hypothesize that the low number of expertise fields
associated with this dataset largely reduces the effect described in the
previous section.

##  5 Discussion and Future Work

In this paper, we provide experiment materials for expert finding with the
help of four annotated datasets and further report results based on several
baseline algorithms. Moreover, we study the ability of document network
embedding methods to tackle the expert finding challenge. We show that DNE
algorithms can not be trivially adapted to achieve state-of-the-art scores.
However, we reveal that document network embeddings can improve the voting
model but diminish the propagation model.

In future work, we would like to find an efficient way to bridge the gap
between DNE algorithms and expert finding. To do so, taking the heterogeneity
into account should help better capturing the real similarity between a
document and a candidate. Furthermore, a deeper analysis of the interplay
between the candidates and the text content of the documents appears to be a
necessary way to better understand the task of expert finding.

Table 2: Mean scores with their standard deviations on DBLP | AUC | P@10 | AP
---|---|---|---
random | 49.47 (09.80) | 05.00 (06.66) | 07.09 (03.81)
panoptic (tf-idf) | 74.06(12.94) | 22.37 (16.35) | 23.24 (12.55)
voting (tf-idf) | 78.60 (11.97) | 26.05 (15.76) | 28.24 (13.92)
propagation (tf-idf) | 79.26 (13.09) | 33.07 (19.61) | 34.66 (18.21)
pre-agg TADW | 65.84 (12.94) | 15.61 (11.63) | 17.26 (08.78)
pre-agg GVNR-t | 76.90 (11.46) | 19.04 (11.70) | 21.39 (09.61)
pre-agg G2G | 72.87 (12.75) | 15.70 (11.62) | 18.53 (09.37)
pre-agg IDNE | 78.08 (11.27) | 20.18 (11.85) | 22.00 (09.87)
post-agg TADW | 68.01 (13.37) | 16.32 (11.57) | 18.01 (08.97
post-agg GVNR-t | 73.91 (13.93) | 18.86 (12.19) | 20.57 (10.33)
post-agg G2G | 68.94 (15.23) | 16.23 (12.02) | 18.21 (09.76)
post-agg IDNE | 76.87 (13.36) | 19.04 (14.57) | 21.57 (10.96)
voting (IDNE) | 82.23 (11.08) | 34.82 (18.46) | 37.27 (16.16)
propagation (IDNE) |  82.44 (16.14) |  44.47 (22.91) |  47.01 (22.06)
Table 3: Mean scores with standard deviations on Stats | AUC | P@10 | AP
---|---|---|---
random | 50.01 (02.24) | 04.52 (07.02) | 04.96 (02.81)
panoptic (tf-idf) | 79.47 (06.22) | 13.45 (13.39) | 15.22 (05.62)
voting (tf-idf) | 84.96 (05.22) | 52.53 (16.13) | 31.01 (06.58)
propagation (tf-idf) | 86.33 (05.64) |  91.53 (13.44) |  44.09 (07.70
pre-agg TADW | 63.07 (07.70) | 11.42 (12.34) | 08.45 (03.87)
pre-agg GVNR-t | 70.67 (09.49) | 21.12 (20.99) | 12.43 (07.30
pre-agg G2G | 63.63 (07.62) | 12.93 (12.06) | 07.81 (04.15
pre-agg IDNE | 65.07 (09.05) | 13.37 (13.48) | 09.40 (05.19)
post-agg TADW | 68.74 (07.02) | 13.67 (12.59) | 09.99 (04.37)
post-agg GVNR-t | 66.56 (08.61) | 22.47 (15.92) | 10.75 (05.42
post-agg G2G | 62.53 (07.44) | 11.95 (11.86) | 07.48 (04.13)
post-agg IDNE | 65.63 (08.57) | 13.34 (13.13) | 09.38 (04.94
voting (IDNE) |  86.94 (04.91) | 53.91 (18.06) | 32.18 (08.33)
propagation (IDNE) | 67.62 (10.11) | 90.43 (15.20) | 33.07 (08.93)

Table 4: Mean scores with standard deviations on Academia | AUC | P@10 | AP
---|---|---|---
random | 50.02 (01.78) | 05.93 (08.07) | 06.09 (02.72)
panoptic (tf-idf) | 81.54 (04.36) | 18.35 (18.76) | 22.93 (07.14)
voting (tf-idf) | 85.88 (03.47) | 57.99 (15.87) | 37.66 (05.83
propagation (tf-idf) |  88.02 (03.32) |  99.01 (03.57) |  54.04 (05.44)
pre-agg TADW | 61.47 (06.16) | 11.09 (12.04) | 09.29 (03.53)
pre-agg GVNR-t | 64.22 (09.69) | 25.67 (23.27) | 13.07 (07.54)
pre-agg G2G | 61.54 (05.38) | 14.30 (12.91) | 08.74 (03.69)
pre-agg IDNE | 58.74 (07.49) | 10.21 (11.58) | 08.41 (03.99
post-agg TADW | 71.94 (04.63) | 14.44 (12.87) | 12.68 (04.37
post-agg GVNR-t | 61.22 (06.24) | 20.70 (14.59) | 10.19 (04.21)
post-agg G2G | 58.87 (05.79) | 12.80 (12.06) | 08.12 (03.67)
post-agg IDNE | 59.97 (07.40) | 10.61 (11.19) | 08.76 (04.17))
voting (IDNE) | 86.79 (03.90) | 55.81 (17.35) | 37.13 (07.58)
propagation (IDNE) | 61.35 (08.56) | 95.02 (10.15) | 31.27 (08.21)
Table 5: Mean scores with standard deviations on Mathoverflow | AUC | P@10 | AP
---|---|---|---
random | 49.98 (01.62) | 06.44 (08.28) | 06.53 (03.06)
panoptic (tf-idf) | 81.87 (04.46) | 21.95 (19.15) | 22.95 (07.54)
voting (tf-idf) | 86.80 (03.23) | 61.11 (18.68) | 40.10 (08.27)
propagation (tf-idf) | 88.08 (03.38) |  93.68 (12.16) |  49.58 (08.90)
pre-agg TADW | NA | NA | NA
pre-agg GVNR-t | 65.34 (09.22) | 44.02 (28.31) | 16.88 (08.55)
pre-agg G2G | 66.84 (08.99) | 22.95 (17.81) | 12.49 (05.70)
pre-agg IDNE | 67.01 (09.26) | 22.96 (17.84) | 13.40 (06.02)
post-agg TADW | NA | NA | NA
post-agg GVNR-t | 63.84 (07.59) | 41.81 (22.68) | 14.96 (06.25)
post-agg G2G | 65.06 (09.09) | 22.43 (16.94) | 11.78 (05.51)
post-agg IDNE | 66.74 (09.10) | 21.92 (17.21) | 13.11 (05.87)
voting (IDNE) |  88.71 (03.76) | 68.46 (18.53) | 43.53 (09.90)
propagation (IDNE) | 69.38 (09.65) | 92.35 (13.88) | 39.62 (09.89)

## References

  * [1] Balog, K., Serdyukov, P., De Vries, A.P.: Overview of the trec 2010 entity track. Tech. rep., NORWEGIAN UNIV OF SCIENCE AND TECHNOLOGY TRONDHEIM (2010)
  * [2] Bojchevski, A., Günnemann, S.: Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. In: International Conference on Learning Representations. pp. 1–13 (2018)
  * [3] Brochier, R., Guille, A., Rothan, B., Velcin, J.: Impact of the query set on the evaluation of expert finding systems. Proceedings of the 3rd Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2018) co-located with the 41st International ACM SIGIR Conference (2018)
  * [4] Brochier, R., Guille, A., Velcin, J.: Global vectors for node representations. In: The World Wide Web Conference. pp. 2587–2593. ACM (2019)
  * [5] Brochier, R., Guille, A., Velcin, J.: Inductive document network embedding with topic-word attention. In: Proceedings of the 42nd European Conference on Information Retrieval Research. Springer (2020)
  * [6] Cifariello, P., Ferragina, P., Ponza, M.: Wiser: A semantic approach for expert finding in academia based on entity linking. Inf. Syst. 82, 1–16 (2019)
  * [7] Craswell, N., Hawking, D., Vercoustre, A.M., Wilkins, P.: P@noptic expert: Searching for experts not just for documents. In: Ausweb Poster Proceedings, Queensland, Australia. vol. 15, p. 17 (2001)
  * [8] Craswell, N., de Vries, A.P., Soboroff, I.: Overview of the trec 2005 enterprise track. In: Trec. vol. 5, pp. 1–7 (2005)
  * [9] Dargahi Nobari, A., Sotudeh Gharebagh, S., Neshati, M.: Skill translation models in expert finding. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. pp. 1057–1060. ACM (2017)
  * [10] Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990)
  * [11] Fischer, T., Remus, S., Biemann, C.: Lt expertfinder: An evaluation framework for expert finding methods. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). pp. 98–104 (2019)
  * [12] Grover, A., Leskovec, J.: node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 855–864. ACM (2016)
  * [13] Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization. In: Advances in neural information processing systems. pp. 2177–2185 (2014)
  * [14] Macdonald, C., Ounis, I.: Voting for candidates: adapting data fusion techniques for an expert search task. In: Proceedings of the 15th ACM international conference on Information and knowledge management. pp. 387–396. ACM (2006)
  * [15] Macdonald, C., Ounis, I., Soboroff, I.: Overview of the trec 2007 blog track. In: TREC. vol. 7, pp. 31–43 (2007)
  * [16] Mislevy, R.J., Riconscente, M.M.: Evidence-centered assessment design. In: Handbook of test development, pp. 75–104. Routledge (2011)
  * [17] Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: Bringing order to the web. Tech. rep., Stanford InfoLab (1999)
  * [18] Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532–1543 (2014)
  * [19] Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: Online learning of social representations. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 701–710. ACM (2014)
  * [20] Riahi, F., Zolaktaf, Z., Shafiei, M., Milios, E.: Finding expert users in community question answering. In: Proceedings of the 21st International Conference on World Wide Web. pp. 791–798. ACM (2012)
  * [21] Serdyukov, P., Rode, H., Hiemstra, D.: Modeling multi-step relevance propagation for expert finding. In: Proceedings of the 17th ACM conference on Information and knowledge management. pp. 1133–1142. ACM (2008)
  * [22] Tang, J., Zhang, J., Yao, L., Li, J., Zhang, L., Su, Z.: Arnetminer: extraction and mining of academic social networks. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 990–998. ACM (2008)
  * [23] Van Gysel, C., de Rijke, M., Worring, M.: Unsupervised, efficient and semantic expertise retrieval. In: WWW. vol. 2016, pp. 1069–1079. The International World Wide Web Conferences Steering Committee (2016)
  * [24] Yang, C., Liu, Z., Zhao, D., Sun, M., Chang, E.: Network representation learning with rich text information. In: Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)
  * [25] Yuan, S., Zhang, Y., Tang, J., Hall, W., Cabotà, J.B.: Expert finding in community question answering: a review. Artificial Intelligence Review 53(2), 843–874 (2020)
  * [26] Zhang, J., Tang, J., Li, J.: Expert finding in a social network. In: International Conference on Database Systems for Advanced Applications. pp. 1066–1069. Springer (2007)
  * [27] Zhang, M., Song, R., Lin, C., Ma, S., Jiang, Z., Jin, Y., Liu, Y., Zhao, L., Ma, S.: Expansion-based technologies in finding relevant and new information: Thu trec 2002: Novelty track experiments. NIST SPECIAL PUBLICATION SP 251, 586–590 (2003)
  * [28] Zhao, Z., Zhang, L., He, X., Ng, W.: Expert finding for question answering via graph regularized matrix completion. IEEE Transactions on Knowledge and Data Engineering 27(4), 993–1004 (2014)


# Understanding Depth Map Progressively: Adaptive Distance Interval Separation
for Monocular 3d Object Detection

1st Xianhui Cheng  School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>2nd Shoumeng Qiu  School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>3rd Zhikang Zou  Baidu Inc.
Shenzhen, China
<EMAIL_ADDRESS>4th Jian Pu∗ Institute of Science and Technology for
Brain-Inspired Intelligence
Fudan University
Shanghai, China
<EMAIL_ADDRESS>5th Xiangyang Xue  School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>
###### Abstract

Monocular 3D object detection aims to locate objects in different scenes with
just a single image. Due to the absence of depth information, several
monocular 3D detection techniques have emerged that rely on auxiliary depth
maps from the depth estimation task. There are multiple approaches to
understanding the representation of depth maps, including treating them as
pseudo-LiDAR point clouds, leveraging implicit end-to-end learning of depth
information, or considering them as an image input. However, these methods
have certain drawbacks, such as their reliance on the accuracy of estimated
depth maps and suboptimal utilization of depth maps due to their image-based
nature. While LiDAR-based methods and convolutional neural networks (CNNs) can
be utilized for pseudo point clouds and depth maps, respectively, it is always
an alternative. In this paper, we propose a framework named the Adaptive
Distance Interval Separation Network (ADISN) that adopts a novel perspective
on understanding depth maps, as a form that lies between LiDAR and images. We
utilize an adaptive separation approach that partitions the depth map into
various subgraphs based on distance and treats each of these subgraphs as an
individual image for feature extraction. After adaptive separations, each
subgraph solely contains pixels within a learned interval range. If there is a
truncated object within this range, an evident curved edge will appear, which
we can leverage for texture extraction using CNNs to obtain rich depth
information in pixels. Meanwhile, to mitigate the inaccuracy of depth
estimation, we designed an uncertainty module. To take advantage of both
images and depth maps, we use different branches to learn localization
detection tasks and appearance tasks separately. Our approach significantly
enhances the baseline and outperforms depth-assisted techniques, as shown by
our extensive experiments on the KITTI monocular 3D object detection
benchmark.

##  I Introduction

The urgent need for inexpensive sensors to perceive the 3D world in real-world
applications, such as autonomous driving, virtual reality, and robotics, is
the motivation for this study. LiDAR devices have shown promising results [42,
43]; however, their high cost makes them challenging to deploy and maintain
widely. To address this issue, there is a pressing need for cheaper object
detectors based on vision. Monocular 3D object detection, which uses only one
camera to perceive object localization, physical dimensions, and orientation,
has received increasing attention from both industry and academia. Despite the
potential benefits, accurate 3D information perception remains a significant
challenge due to the lack of depth information available from monocular
cameras. Therefore, there is a need to solve several problems in monocular 3D
object detection.

Figure 1: Comparison between general depth maps and our proposed separated
depth maps is shown in the figure. The top image shows the original input, the
middle image displays the estimated depth map, and the bottom image
illustrates our separated depth map. Notably, in the yellow box, a parked car
is barely visible in the general depth map but is clearly distinguishable in
our separated depth map.

Recent methods of monocular 3D object detection can be categorized into four
main groups based on their ideas for improvement: image-based, imitation-
based, pseudo LiDAR-based, and depth-assisted approaches.

Image-based methods rely solely on an image as input. Given the absence of
depth information, these methods attempt to establish connections between 2D
and 3D object information using geometric methods or feature enhancement
techniques. Unlike LiDAR, which can accurately measure distance, monocular
methods face challenges in accurately locating objects in space based on the
image plane.

Pseudo-LiDAR-based methods aim to bridge the gap between pixels and point
clouds. Depth maps from the depth estimation task can be transformed to
generate pseudo-LiDAR point clouds. Well-developed LiDAR-based detectors can
then be directly applied, while some methods use CNNs to process pseudo-LiDAR.
However, the accuracy of depth estimation methods may significantly affect the
data transformation of pseudo-LiDAR, making it unsuitable for point cloud
detectors trained on accurate point cloud data. Furthermore, overfitting
frequently occurs, particularly in the KITTI dataset [35].

To achieve excellent performance similar to LiDAR-based or multicamera image-
based methods, knowledge distillation is used to improve monocular 3D object
detectors. Successful teacher models impressively guide monocular detectors.
However, the strict requirements for sensor space-time synchronization and
calibration make it challenging to deploy the model on different vehicles.

Depth-assisted methods complement the rich semantic information of images with
the distance information of depth maps. Previous works mainly focus on
effectively extracting and fusing depth information with image features, often
using CNNs. CNNs are suitable for extracting semantic and texture features,
but depth maps lack these features. Therefore, the decision to let depth maps
directly pass through CNNs needs reconsideration. Additionally, the accuracy
of the depth map estimation significantly affects the model’s performance. It
is essential to distinguish which parts of the depth map are trustworthy and
should be given special consideration. Furthermore, images and depth maps have
different areas of expertise in the 3D object detection task. Images are
better suited for extracting semantic information, while depth maps are ideal
for determining location information. Therefore, they should play to their
strengths and complement each other.

In this paper, we summarize three problems and provide corresponding
solutions. The problems are as follows: (i) Extracting depth map information
effectively. Depth maps have physical meaning, while CNNs are better suited
for texture extraction. To extract depth map information more effectively, we
need to make a connection between texture and physical distance. One way to do
this is by dividing the depth map into sub-depth maps (SDs) based on the
specified distance. We can then preserve pixels falling into a specific range
as another sub-depth map. By doing this, we can combine the advantages of
depth maps and CNNs, better recognize edges, and have an intuitive
explanation. (ii) Dealing with inaccurate depth maps. To address the problem
of inaccurate depth map estimation, we propose using an uncertainty map
aligned with the size of the depth map. The uncertainty map gives confidence
for every pixel on the depth map and allows the model to judge whether the
location of depth maps is credible. If the depth value tends to be inaccurate,
the corresponding uncertainty map suppresses the error caused by the
inaccurate region, and the credible values play a greater role. The module of
the uncertainty map generally makes depth maps more reliable. (iii) Utilizing
the advantages of images and depth maps. Images and depth maps have their
respective advantages in appearance and localization features. To make the
most of these advantages, we designed a feature decoupling module that
separates output heads according to the input modality. By using different
branches to learn appearance and localization detection task heads, the model
can better recognize categories and understand spatial information.

Our main contributions are as follows:

  * •

We actively create edges to connect the physical meaning of depth maps with
the ability of CNNs.

  * •

We propose a simple and effective uncertainty map to reduce the noise caused
by inaccurate depth estimation and make the depth map more reliable.

  * •

We separate output heads according to the input modality, which improves the
overall performance of the model without affecting the calculation cost.

  * •

Extensive experiments show the effectiveness of our proposed approach among
monocular 3D object detection methods on the KITTI benchmark.

Figure 2:  Visualization of our proposed framework is shown below, where two
branches for appearance and localization are denoted in different colors.
Dashed boxes represent corresponding output tasks. In the appearance branch,
the appearance information $I_{A}$ is generated from both the image feature
$F_{I}$ and the depth feature $F_{D}$. The uncertainty map and separated depth
maps with adaptive distance intervals are generated by encoding $F_{I}$ and
$F_{D}$ as $F_{U}$ and $F_{SDs}$, respectively. These features are then fused
into localization information $I_{L}$ and fed to the corresponding heads.

##  II Related work

###  II-A Image-based approaches

It is challenging to infer 3D spatial information from a single image due to
the inherent lack of depth information. Pioneering works [1, 2, 4, 5] use
relatively stable 2D detection results to predict 3D boxes. Some approaches
match 2D boxes with predefined 3D shapes for different cars [3]. Other works
focus on disentangling complex interactions between parameters, achieving
remarkable progress [6, 7]. M3D-RPN [8] proposes a framework that shares 2D-3D
features with sets of anchors [9], while RTM3D [10] estimates several
projected key points to construct 3D bounding boxes with geometric
relationships. Heatmap-based approaches consider objects as points [11, 12,
13, 14], significantly reducing inference time. In some cases, auxiliary
information support can significantly increase the performance of monocular 3D
detectors. Kinematic3d [15] proposes establishing a kinematic model using
continuous frames of images and correcting the object’s location in space
according to the kinematic model recommendations and regression branches’
results. CaDDN [16] learns the categorical depth distribution from sampled
point clouds in the bird’s-eye-view (BEV) at the training stage, utilizing
point clouds to guide depth features.

###  II-B Pseudo LiDAR-based approaches

Compared to the success of LiDAR-based methods, monocular 3D object detectors
have significant room for improvement. Wang et al. [17] converted depth map
pixels obtained from the depth estimation task into a pseudo-LiDAR point cloud
representation, which was then processed using a LiDAR-based detection network
[18]. PatchNet [19] proposed a novel perspective on the pseudo-LiDAR
representation. In addition to using LiDAR-based detectors, simple CNNs can
also produce considerable results. Wang et al. [20] analyzed the impact of 3D
object parameters on the results and adopted different strategies based on the
degree of influence. NeighborVote [21] proposes a joint decision-making model
based on the pseudo-LiDAR generated from the estimated depth map. The feature
points surrounding the proposed object form their own predictions, and these
predictions are then combined to form the final output.

###  II-C Imitation-based approaches

In the task of 3D object detection, LiDAR and stereo modalities have achieved
remarkable performance. Some algorithms aim to modify monocular 3D detectors
by adopting the paradigms of the aforementioned modalities to enable them to
perform well under different modality settings [22, 23, 24]. DD3D [25]
explores the role of depth maps in the detection task and proposes a transfer
model between depth estimation and 3D detection. The well-trained knowledge
from depth estimation is then applied to the image, and the results
demonstrate that the implicit use of depth information can significantly
enhance the 3D detection results. CMKD [26] transfers knowledge from LiDAR to
the image via a knowledge distillation semi-supervised training framework,
resulting in significant performance improvement. MonoPS [27] contends that
the gap between stereo and image is much smaller than that between LiDAR and
image. It replaces stereo images from the stereo-based detection architecture
with defined pseudo-stereo image features, resulting in promising results. To
imitate the representations of 3D point clouds, Feng et al. [28] align 2D
proposals with 3D point clouds.

###  II-D Depth-assisted approaches

D4LCN [29] uses depth maps as guidance to distinguish foreground and
background in scenes, while DDMP-3D [30] utilizes graphic message passing
mechanisms and convolution kernels on depth maps to dynamically extract
features. Moreover, instance-level center-aware depth guidance is proposed to
reduce depth estimation errors. DRF-Net [31] discusses the appearance and
location information of objects and uses a feature reflecting module to make
the most of different features. Cai et al. [32] treat monocular 3D object
detection as several sub-tasks, roughly estimating object depth with an
auxiliary prior 2D structured polygon and finally refining it with the depth
map.

##  III Methodology

In this section, we present the framework of our proposed ADISN. Subsequently,
we provide a detailed description of three major structures proposed in this
study, which include adaptive distance interval separation, uncertainty map,
and feature decoupling module.

###  III-A Pipeline overview

We employed M3D-RPN [8] as our baseline detector due to its simplicity and
influence on several depth-assisted methods. We followed the baseline’s anchor
definition and learning parameters. Our network accepts both image and depth
map inputs. We used DORN [33] as a monocular depth estimator to generate depth
maps. As illustrated in Fig 2, we encoded the images and depth maps as $F_{I}$
and $F_{D}$, respectively. We fused $F_{I}$ and $F_{D}$ to create an
uncertainty map $U$ and adaptive distance lower and upper bounds. We extracted
feature $F_{SDs}$ from Sub-Depth maps (SDs) and combined it with $F_{D}$ and
$F_{U}$ from the uncertainty map to obtain the overall localization
information feature $I_{L}$. Simultaneously, we fused $F_{I}$ and $F_{D}$ to
produce the overall appearance information $I_{A}$. Finally, $I_{L}$ and
$I_{A}$ were fed into output task heads with feature decoupling. We elaborate
on all the modules mentioned above in the following subsections.

Figure 3: Example of separated sub-depth maps: The top-left panel shows a
sketch map of the side view with objects truncated using our adaptive
intervals. The top-right panel displays the original input image, while the
bottom panels show the separated sub-depth maps.

###  III-B Adaptive distance interval separation

In this section, we propose an adaptive distance interval separation module
(ADIS) to handle different distance intervals in the depth map. Since the
intensity of the pixel value of the depth map represents the physical meaning
of distance, we divide the depth map into several intervals based on distance.
For each sub-depth (SD), only pixels within the selected distance interval are
retained while the remaining positions are assigned to zero. This generates a
sharp edge near the reserved area on the subgraph, which is favorable for the
strength of CNNs. If the selected distance interval truncates the object,
there will be significant changes in the subgraph, making it conducive to
positioning the object in space.

We need to generate upper and lower bounds adaptively from each unique input
feature to obtain an adaptive distance interval. We define the interval
sequence as $\mathbf{d_{i}}=(d_{i-1},d_{i}),i=1,2,...,n_{d}$, where $n_{d}$
denotes the number of distance intervals we want, and $d_{i}$ is the distance
learned from the network. Every $\mathbf{d_{i}}$ has its upper and lower
bounds, which describe a range of distances. There is no intersection between
intervals ($\mathbf{d_{i}}\cap\mathbf{d_{j}}=\phi,i\neq j$).

Given feature $F_{ID}\in\mathbb{R}^{C\times H\times W}$, which denotes the
fusion of $F_{I}$ and $F_{D}$, we use a $1\times 1$ convolution to reduce the
channel and flatten the feature map to a vector of size $1\times(H\times W)$.
After fully connected layers, we have a distance feature of size $1\times
n_{d}$, and the $\mathbf{d}=(d_{0},d_{1},...d_{n_{d}}),d_{0}=0$ is generated
using the softmax operation. Therefore, we can divide the input depth map
$Dep$ into its subgraphs $SD_{i},i=1,2,...,n_{d}$ with $d_{i}$ using Equation
1.

$\footnotesize
SD_{i}=\left\\{\begin{matrix}SD^{(w,h)}=Dep^{(w,h)},&\sum_{j=0}^{i-1}d_{j}\leq
SD^{(w,h)}\leq\sum_{j=0}^{i}d_{j}\\\
SD^{(w,h)}=0,&Otherwise.\end{matrix}\right.$ (1)

Figure 4: Visualization of uncertainty maps. The top shows input images, the
middle displays estimated depth maps, and the bottom exhibits our proposed
uncertainty maps. As demonstrated by the confidence levels, nearby areas of
the depth maps are more reliable, while the inaccuracy of the distant areas
and the sky is reduced.

Here, $(w,h)$ represents the pixel coordinates in the image of size $H\times
W$. The estimated depth value of the pixels is preserved in the SD, and the
pixel coordinates correspond to the image, so it benefits CNNs to extract
depth information. The ADIS module is more effective than directly applying
CNNs to the depth map since it generates SDs with obvious curved edges
precisely caused by the truncation of objects. When $n_{d}$ is large enough,
in fact, the SDs we divide would be the dense LiDAR point cloud. Fig 3 shows
the generated SDs, these SDs contain obvious curved edges, which are precisely
caused by the truncation of objects.

###  III-C Uncertainty map

Our proposed model heavily relies on the accuracy of the depth map during the
feature extraction stage. However, the depth map we utilize is obtained from a
pre-existing depth estimation method, which implies that it may not be
entirely precise. The regions that are not trained thoroughly by the LiDAR in
the depth estimation task, such as faraway objects or the sky, tend to perform
poorly, and this inaccuracy remains on the depth map. This can be
disadvantageous when estimating the depth of a dense image, as it can
negatively impact the model’s performance. To address this, we draw
inspiration from methods based on uncertainty that utilize uncertainty
measurements to mitigate errors when there is noise in the initial condition.
Thus, we incorporate an uncertainty map into our network to guide the model
during the process of extracting depth information. As we aim to mitigate the
inaccuracy of the depth map, our depth extraction module learns the
uncertainty point by point, suppressing inaccurate pixels in the estimated
depth map and reducing the negative impact on the model. To enable the
uncertainty map to locate the regions to be suppressed in the image and depth
map, we jointly generate the uncertainty feature $U$ by combining the image
and depth map features. This uncertainty feature is also used when extracting
$\mathbf{d}$, ensuring that the two modules work together to further improve
the accuracy of the depth map. The details of our approach are provided below:

$U^{H\times W}=\mathbf{1}^{H\times W}-Sigmoid(Upsampling(F_{ID}))^{H\times
W},$ (2)

shows the process of generating the uncertainty map, where
$\mathbf{1}^{H\times W}$ represents the $H\times W$ matrix with elements of 1,
and $F_{ID}$ is the fusion of $F_{I}$ and $F_{D}$. The sigmoid function used
ensures that each pixel in the corresponding depth map is assigned an
uncertainty value between 0 and 1. To avoid gradient vanishing in the early
stage of training, we obtain $1-U$ instead of directly predicting $U$. The
trained uncertainty maps are presented in 4. The final output feature is the
result of combining the SDs, which is obtained through the following equation:

$F_{SD}=Conv(\mathbf{SD}\times\mathbf{U}),$ (3)

where $\mathbf{SD}$ and $\mathbf{U}$ denote the stacked $SD_{i}$ and $U$,
respectively. Both are duplicated to the same channel according to the
$n_{d}$.

TABLE I:  Comparison of Different Methods for the “Car” Category on the KITTI
Test Set with AP 3D and AP BEV. We Highlight the Best Results in bold and the
Second Place in underlined

Experiments | Implementation mode | AP 3D | AP BEV
---|---|---|---
Easy | Mod. | Hard | Easy | Mod. | Hard
M3D-RPN(base line) [8] | image-based | 14.76 | 9.71 | 7.42 | 21.02 | 13.67 | 10.23
MonoCInIS [36] | image-based | 15.82 | 7.94 | 6.68 | 22.28 | 11.64 | 9.95
ImVoxelNet[37] | image-based | 17.15 | 10.97 | 9.15 | 25.19 | 16.37 | 13.58
M3DSSD [38] | image-based | 17.51 | 11.46 | 8.98 | 24.15 | 15.93 | 12.11
AM3D [39] | pseudo LiDAR-based | 16.50 | 10.74 | 9.52 | 25.03 | 17.32 | 14.91
PatchNet[19] | pseudo LiDAR-based | 15.68 | 11.12 | 10.17 | 22.97 | 16.86 | 14.97
Zhao et al.[40] | pseudo LiDAR-based | 15.86 | 11.41 | 10.08 | 28.79 | 18.61 | 15.27
Cai et al.[32] | depth-assisted | 11.68 | 7.28 | 5.69 | 24.62 | 14.66 | 11.46
Ji et al.[41] | depth-assisted | 12.46 | 7.86 | 6.30 | 19.85 | 13.07 | 10.29
D4LCN [29] | depth-assisted | 16.65 | 11.72 | 9.51 | 22.51 | 16.02 | 12.55
ADISN(Ours) | depth-assisted | 17.87 | 12.14 | 10.42 | 27.18 | 17.91 | 15.33

###  III-D Feature decoupling

Inspired by DRF-Net [31], DDMP-3D [30], and other monocular 3D object
detection methods, it has been noted that the position of the object in space,
i.e., $(x,y,z)$, is the most critical factor in the performance of monocular
3D detection. However, images suffer from the weakness of understanding depth,
and the depth estimation task exists to address this issue. Experiments have
shown that even if only a single path of depth maps is used as input, the
performance of 3D detection is better than when only RGB images are used.
Nevertheless, relying solely on depth maps makes it difficult to understand
the appearance and category of the object. Moreover, over-reliance on depth
maps is affected by inaccurate depth estimation, especially at long distances.
Therefore, RGB images are still necessary. They perform better in semantics
than depth maps and can complement each other, which is also the advantage of
depth-assisted methods. From this perspective, appearance and location
features should be different and related. Compared with DRF-Net [31] from the
perspective of general features, we consider the modality of input and use
ADIS for further learning of depth maps.

Specifically, we divide the detection head into two streams: the appearance
head and the localization head. The feature before the final output $I_{A}$
represents the appearance information, while $I_{L}$ represents the location
information, each generated in the previous stage, $I_{A}=F_{I}+F_{D}$, and
$I_{L}=F_{D}+F_{SD}+F_{U}$. In the stream of appearance, we still maintain the
depth features. By robustly combining the extracted RGB and depth features,
our algorithm does not rely too much on the depth map that may be biased, thus
improving the overall performance of our model.

##  IV Experiments

###  IV-A Dataset

The KITTI dataset is a widely recognized benchmark for evaluating 3D object
detection algorithms, comprising 7481 training samples and 7518 testing
samples, with both 3D point cloud and color images provided. In our approach,
we rely solely on color images. The evaluation is conducted on three
categories of objects, namely Car, Pedestrian, and Cyclist, using the average
precision (AP) metric, which is a commonly used evaluation metric for 3D and
Bird’s-Eye-View(BEV) object detection, calculated using 40 recall positions.
As per the official KITTI evaluation protocol, objects are classified into
three difficulty levels: easy, moderate, and hard based on the object size,
occlusion state, and truncation level. The algorithms’ final performance is
ranked based on the moderately difficult results. In this paper, we use the
Average Precision (AP) as the evaluation metric, with an Intersection over
Union (IoU) threshold of 0.7. Under this standard, the definition of AP is as
follows: for each class, we calculate a set of confidence scores and
corresponding ground-truth labels, and sort them in descending order of
confidence scores. Then, for each confidence threshold, we compute the
precision and recall, and the corresponding AP value. Finally, we take the
average AP over all classes to obtain the final AP value. This metric can be
used to evaluate the accuracy and robustness of object detection models.

###  IV-B Implementation details

We implemented our proposed method using PyTorch 1.2.0. Ablation study
experiments were conducted on a GTX1080Ti GPU with 11 GB memory, with a batch
size of 2 and a learning rate of 0.004 for 80000 iterations. For the test set,
we trained with a batch size of 4, a learning rate of 0.008 for 120000
iterations, and a Tesla V100 with 32 GB. We used DenseNet-121 as the backbone
for extracting both the image and depth map. Since the backbone is trained on
ImageNet, we normalized the input using means of (0.485, 0.456, 0.406) and
stds of (0.229, 0.224, 0.225), and the optimizer was SGD. We set the number of
intervals $n_{d}$ to 8. Since $n_{d}$ is generated using full connection
layers, the size of the feature map must be strictly controlled, hence we
padded every input image to size (1760, 512) at the beginning of both the
training and testing steps.

TABLE II:  Comparison of the Use of Different Depth Maps for Category “Car” on
the KITTI Validation Set with AP 3D

Experiments | Depth map | AP 3D
---|---|---
Easy | Mod. | Hard
M3D-RPN(base line) [8] | - | 14.53 | 11.07 | 8.65
D4LCN [29] | DORN[33] | 22.32 | 16.20 | 12.30
ADISN(Ours) | DORN[33] | 26.86 | 17.28 | 13.15
Improvement over [29] | - | +4.54 | +1.08 | +0.85
D4LCN [29] | PSMNet[34] | 25.24 | 19.80 | 16.45
ADISN(Ours) | PSMNet[34] | 36.02 | 23.25 | 18.19
Improvement over [29] | - | +10.78 | +3.45 | +1.74

###  IV-C Comparison

Table I displays the performance comparison of the KITTI test set in the
category of cars based on AP 3D and AP BEV, compared under the IoU=0.7
standard. Our proposed method achieves superior performance among monocular 3D
object detection methods, ranking 1st in the Easy, Moderate, and Hard
categories with AP 3D. We observe significant improvement in the 3D detection
task compared to other advanced methods, with an increase of 3.11 on Easy,
2.43 on Moderate, and 3.00 on Hard when compared to the baseline. Our method
is based on M3D-RPN [8], and despite the improved M3D-RPN [8] named “3D-Net”
being used as the baseline by D4LCN [29], our method retains its advantages.
This indicates that the depth map has a significant improvement in the
monocular 3D object detection task, and that older methods such as the
baseline are still competitive in recent methods. Under the AP BEV index, we
rank second in Easy and Moderate and first in Hard. Recently, monocular
methods no longer compare results on the validation set, as performance often
drops significantly on the test set. Therefore, comparing results on the test
set is the most objective method. Overall, our comprehensive experimental
results confirm that our method significantly improves the monocular 3D object
detector.

TABLE III: Comparison of Using Different $n_{d}$ for Category “Car” on the
KITTI Validation Set

Settings of $n_{d}$ | AP 3D
---|---
Easy | Mod. | Hard
0 | 22.93 | 14.81 | 11.58
4 | 23.64 | 16.21 | 11.47
8 | 26.86 | 17.28 | 13.15
16 | 27.92 | 17.14 | 13.15
32 | 26.65 | 16.68 | 12.64

TABLE IV:  Comparison of Various Modules in the Proposed Method for the
Category “Car” on the KITTI Validation Set

Experiments | Our proposed operations | AP BEV
---|---|---
Separated head | ADIS(8) | Un | Feature decoupling | Easy | Mod. | Hard
M3D-RPN(image only)[8] |  |  |  |  | 14.43 | 10.94 | 8.57
M3D-RPN(depth only)[8] |  |  |  |  | 19.75 | 13.08 | 9.46
M3D-RPN(image+depth)[8] |  |  |  |  | 20.90 | 14.33 | 9.93
ours | ✔ |  |  |  | 22.93 | 14.81 | 11.58
✔ | ✔ |  |  | 23.09 | 15.77 | 12.15
✔ | ✔ | ✔ |  | 23.84 | 16.12 | 12.64
✔ | ✔ |  | ✔ | 25.03 | 16.63 | 12.43
✔ | ✔ | ✔ | ✔ | 26.86 | 17.28 | 13.15

###  IV-D Ablation Study

We compared our proposed method with M3D-RPN [8] and D4LCN [29] on the KITTI
validation set using different depth estimation methods (see Table II). Our
method showed advantages with different depth maps, with greater advantages
when the depth map accuracy was higher. This suggests that our method’s
approach to understanding depth maps is more effective, as a more accurate
depth estimation is closer to point clouds and more intuitive.

We conducted a simple experiment to determine the number of intervals
($n_{d}$) according to the final model, as shown in Table III. Increasing
$n_{d}$ improves the effect, but also incurs greater computational costs. When
$n_{d}$ was set to 16, performance on Hard no longer increased, as hard
objects are difficult to estimate accurately in the depth estimation task.
Setting $n_{d}$ to greater than 32 may lead to overfitting and degrade
performance. After comprehensive consideration, we set $n_{d}$ to 8.

Table IV shows the ablation experiment of our proposed method. We disassembled
each module to verify their effectiveness. The performance of M3D-RPN [8] as a
baseline was lower than the original paper’s report due to device issues.
Notably, even using a depth map directly as the input of M3D-RPN [8] yielded
better performance than the original RGB image, indicating the crucial role of
depth maps in extracting position information for monocular 3D object
detection. A simple fusion of image and depth map features yielded performance
improvements, likely due to the complementarity of depth maps and images. We
separately set the detector heads of different tasks according to feature
characteristics to improve the results. The ADIS module significantly improved
performance by further extracting depth features. Adding an uncertainty map
also improved overall performance. Furthermore, our feature decoupling module
further improved performance.

Fig 5 shows visualizations of the effectiveness of our proposed method. Our
method outperformed M3D-RPN [8] for distant or occluded objects and improved
many missed detection situations.

Figure 5: The qualitative comparison of the ground truth (green), the baseline
(blue), and our proposed method (red) on the KITTI validation set. Our
proposed method predicts accurate 3D bounding boxes of occluded and distant
objects.

##  V Conclusion

In this paper, we present a novel approach to understanding depth map
representation by proposing ADISN. Our sub-depth map establishes a
relationship between 3D and 2D expressions, providing insights into the
physical meaning of depth maps and highlighting the advantages of CNNs over
converting depth maps to 3D or 2D representations. To address the issue of
inaccurate estimated depth maps, we introduce the use of an uncertainty map,
which improves the reliability of learning in unclear areas by relying more on
clear images. Additionally, we propose a method where appearance and
localization heads learn from different branches, allowing images and depth
maps to leverage their respective strengths. Extensive experimentation shows
that our proposed method outperforms the highly competitive KITTI monocular 3D
object detection task.

##  VI Future work

Monocular 3D object detection is always a hot topic in the field of autonomous
driving before reliable and cost-effective LiDAR sensors become available.
Recent research has mostly focused on improving the performance of monocular
3D object detectors using other modalities, partly because it is relatively
easy to adapt existing high-performance algorithms by pruning them into
monocular detectors, and partly because it may be inappropriate to rely solely
on a single image to reconstruct a 3D scene. Previous work has pre-trained
neural networks on depth estimation tasks and fine-tuned them on object
detection tasks with color images, implicitly learning depth features, and
achieving impressive performance. Modalities or paradigms that are independent
of RGB inputs may possess more knowledge suitable for inferring 3D space, and
extracting and refining such knowledge into single images may be a direction
for future work.

## References

  * [1] Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., and Urtasun, R, ”Monocular 3d object detection for autonomous driving,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2147-2156.
  * [2] Ku, J., Pon, A. D., and Waslander, S. L., ”Monocular 3d object detection leveraging accurate proposals and shape reconstruction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11867-11876.
  * [3] Chabot, F., Chaouch, M., Rabarisoa, J., Teuliere, C., and Chateau, T, ”Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2040-2049.
  * [4] Mousavian, A., Anguelov, D., Flynn, J., and Kosecka, J., ”3d bounding box estimation using deep learning and geometry,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 7074-7082.
  * [5] Liu, L., Lu, J., Xu, C., Tian, Q., and Zhou, J., ”Deep fitting degree scoring network for monocular 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1057-1066.
  * [6] Simonelli, A., Bulo, S. R., Porzi, L., López-Antequera, M., and Kontschieder, P., ”Disentangling monocular 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1991-1999.
  * [7] Qin, Z., Wang, J., and Lu, Y., ”Monogrnet: A general framework for monocular 3d object detection,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 9, pp.5170-5184, 2021.
  * [8] Brazil, G., and Liu, X., ”M3d-rpn: Monocular 3d region proposal network for object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 9287-9296.
  * [9] Liu, Y., Yixuan, Y., and Liu, M., ”Ground-aware monocular 3d object detection for autonomous driving.” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp.919-926, 2021.
  * [10] Li, P., Zhao, H., Liu, P., and Cao, F., ”Rtm3d: Real-time monocular 3d detection from object keypoints for autonomous driving,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 644-660.
  * [11] Zhou, X., Wang, D., and Krähenbühl, P., ”Objects as points,” _arXiv preprint arXiv_ :1904.07850, 2019.
  * [12] Liu, Z., Wu, Z., and Tóth, R., ”Smoke: Single-stage monocular 3d object detection via keypoint estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 996-997.
  * [13] Zhang, Y., Lu, J., and Zhou, J., ”Objects are different: Flexible monocular 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3289-3298.
  * [14] Lu, Y., Ma, X., Yang, L., Zhang, T., Liu, Y., Chu, Q., Yan, J. and Ouyang, W, ”Geometry uncertainty projection network for monocular 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021, pp. 3111-3121.
  * [15] Brazil, G., Pons-Moll, G., Liu, X., and Schiele, B., ”Kinematic 3d object detection in monocular video,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 135-152.
  * [16] Reading, C., Harakeh, A., Chae, J., and Waslander, S. L., ”Categorical depth distribution network for monocular 3d object detection,” _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8555-8564.
  * [17] Wang, Y., Chao, W.L., Garg, D., Hariharan, B., Campbell, M. and Weinberger, K.Q., ”Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8445-8453.
  * [18] Weng, X., and Kitani, K., ”Monocular 3d object detection with pseudo-lidar point cloud,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0-0.
  * [19] Ma, X., Liu, S., Xia, Z., Zhang, H., Zeng, X., and Ouyang, W., ”Rethinking pseudo-lidar representation,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 311-327.
  * [20] Wang, L., Zhang, L., Zhu, Y., Zhang, Z., He, T., Li, M. and Xue, X., ”Progressive coordinate transforms for monocular 3d object detection,” _Advances in Neural Information Processing Systems_ vol. 34, pp. 13364-13377, 2021.
  * [21] Chu, X., Deng, J., Li, Y., Yuan, Z., Zhang, Y., Ji, J. and Zhang, Y., ”Neighbor-vote: Improving monocular 3d object detection through neighbor distance voting,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , pp. 5239-5247, 2021.
  * [22] Peng, L., Liu, F., Yu, Z., Yan, S., Deng, D., Yang, Z., Liu, H. and Cai, D., ”Lidar point cloud guided monocular 3d object detection,” in _Proceedings of the European Conference on Computer Vision_ , 2022, pp. 123-139.
  * [23] Chong, Z., Ma, X., Zhang, H., Yue, Y., Li, H., Wang, Z. and Ouyang, W., ”Monodistill: Learning spatial features for monocular 3d object detection,” _arXiv preprint arXiv_ :2201.10830, 2022.
  * [24] Zhou, Z., Du, L., Ye, X., Zou, Z., Tan, X., Zhang, L., Xue, X. and Feng, J., ”SGM3D: stereo guided monocular 3d object detection.” _IEEE Robotics and Automation Letters_ , vol 7, no. 4, pp. 10478-10485, 2022.
  * [25] Park, D., Ambrus, R., Guizilini, V., Li, J., and Gaidon, A., ”Is pseudo-lidar needed for monocular 3d object detection?” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3142-3152.
  * [26] Hong, Y., Dai, H., and Ding, Y., ”Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection,” in _Proceedings of the European Conference on Computer Vision_ , 2022, pp. 87-104.
  * [27] Chen, Y. N., Dai, H., and Ding, Y., ”Pseudo-stereo for monocular 3d object detection in autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 887-897.
  * [28] Feng, D., Han, S., Xu, H., Liang, X., and Tan, X., in ”Point-Guided Contrastive Learning for Monocular 3-D Object Detection,” _IEEE Transactions on Cybernetics_ , 2021.
  * [29] Ding, M., Huo, Y., Yi, H., Wang, Z., Shi, J., Lu, Z., and Luo, P., ”Learning Depth-Guided Convolutions for Monocular 3D Object Detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11672-11681.
  * [30] Wang, L., Du, L., Ye, X., Fu, Y., Guo, G., Xue, X., Feng, J. and Zhang, L., ”Depth-conditioned dynamic message propagation for monocular 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 454-463.
  * [31] Zou, Z. et al., ”The devil is in the task: Exploiting reciprocal appearance-localization features for monocular 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2713-2722.
  * [32] Cai, Y., Li, B., Jiao, Z., Li, H., Zeng, X. and Wang, X., ”Monocular 3d object detection with decoupled structured polygon estimation and height-guided depth estimation,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 7, 2020.
  * [33] Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D., ”Deep ordinal regression network for monocular depth estimation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2002-2011.
  * [34] Chang, J. R., and Chen, Y. S., ”Pyramid stereo matching network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5410-5418.
  * [35] Geiger, A., Lenz, P., and Urtasun, R., ”Are we ready for autonomous driving? the kitti vision benchmark suite,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE_ , 2012, pp. 3354–3361.
  * [36] Heylen, J. et al., ”Monocinis: Camera independent monocular 3d object detection using instance segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 923-934.
  * [37] Rukhovich, D., Vorontsova, A. and Konushin, A., ”Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 2397-2406.
  * [38] Luo, S., Dai, H., Shao, L. and Ding, Y., ”M3dssd: Monocular 3d single stage object detector,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6145-6154.
  * [39] Ma, X., Wang, Z., Li, H., Zhang, P., Ouyang, W. and Fan, X., ”Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving,”in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6851-6860.
  * [40] Zhao, D., Ji, C. and Liu, G.,”Monocular 3D Object Detection Based on Pseudo Multimodal Information Extraction and Keypoint Estimation,” _Applied Sciences_ , vol. 13, no. 3, p.1731, 2023.
  * [41] Ji, C., Liu, G. and Zhao, D., ”Monocular 3D object detection via estimation of paired keypoints for autonomous driving,” _Multimedia Tools and Applications_ , vol. 81, no. 4, pp.5973-5988, 2022.
  * [42] Ju, B., Zou, Z., Ye, X., Jiang, M., Tan, X., Ding, E., and Wang, J., ”Paint and Distill: Boosting 3D Object Detection with Semantic Passing Network,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 5639-5648.
  * [43] Liu, Z. et al, ”Multi-Modal 3D Object Detection by Box Matching,” _arXiv preprint arXiv_ :2305.07713, 2023.