# Federated Bayesian Network Ensembles

Florian van Daalen, Lianne Ippel, Andre Dekker, and Inigo Bermejo F. van
Daalen, I. Bermejo, and A. Dekker are with the Department of Radiation
Oncology (MAASTRO) GROW School for Oncology and Reproduction Maastricht
University Medical Centre+ Maastricht the Netherlands
L. Ippel is with Statistics Netherlands Heerlen the Netherlands.

###### Abstract

Federated learning allows us to run machine learning algorithms on
decentralized data when data sharing is not permitted due to privacy concerns.
Ensemble-based learning works by training multiple (weak) classifiers whose
output is aggregated. Federated ensembles are ensembles applied to a federated
setting, where each classifier in the ensemble is trained on one data
location. In this article, we explore the use of federated ensembles of
Bayesian networks (FBNE) in a range of experiments and compare their
performance with locally trained models and models trained with VertiBayes, a
federated learning algorithm to train Bayesian networks from decentralized
data. Our results show that FBNE outperforms local models and provides a
significant increase in training speed compared with VertiBayes while
maintaining a similar performance in most settings, among other advantages. We
show that FBNE is a potentially useful tool within the federated learning
toolbox, especially when local populations are heavily biased, or there is a
strong imbalance in population size across parties. We discuss the advantages
and disadvantages of this approach in terms of time complexity, model
accuracy, privacy protection, and model interpretability.

###### Index Terms:

Federated Learning, Bayesian network, privacy preserving, Federated Ensembles,
Ensemble Learning

##  1 Introduction

†† _The views expressed in this paper are those of the authors and do not
necessarily reflect the policy of Statistics Netherlands._ †† _This research
received funding from the Netherlands Organization for Scientific Research
(NWO): Coronary ARtery disease: Risk estimations and Interventions for
prevention and EaRly detection (CARRIER): project nr. 628.011.212._

Federated learning allows machine learning algorithms to be applied to
decentralized data when data sharing is not an option due to privacy concerns
[1]. Traditionally federated learning approaches train a model iteratively on
local data[2]. The local results are then averaged back into a single global
model. Privacy is preserved using epsilon-differential privacy[3], homomorphic
encryption[4], and multiparty computation (MPC)[5] during this process. The
specific techniques used, depend on the way the data is split across the
various parties. If the data is split horizontally, i.e, each party has data
belonging to a different population but the attributes are the same, simpler
techniques can be used. While this approach yields good results in many cases,
it suffers from several limitations.

A major downside is that it does not explicitly consider heterogeneity across
different data sites. That is, it assumes that the data is independent and
identically distributed (IID) over the various parties. However, in practice,
federated environments will often be subject to local biases. For example, a
common scenario where federated learning is implemented is when multiple
hospitals combine their data to build a joint model[6]. These hospitals may
have very different population sizes, which may cause the final model to
overfit on the biggest hospital. Additionally, the hospitals might have biases
in their populations: e.g., an urban and a rural hospital will have different
patient populations. Furthermore, the hospitals may even be on opposite sides
of the globe, adding further biases into the distribution due to cultural,
socio-economic, and many other factors. Simply averaging over these diverse
populations may result in models that fit neither population, or it may result
in the model overfitting on one particular population, while ignoring others.

If the data is split vertically, , i.e. when different parties have different
variables about the same population, it may be possible that there are
dependencies between the parties. For example, if one party is a general
practitioner (GP) and the other party is the specialist clinician, the GP
might have referred the patient to the specialist and there will be a
dependency between the two datasets. The GP might, for example, have started
treatment based on the data he has, which will influence the data the
specialist receives.

These diverse types of bias may create problems when using the traditional
federated learning approach. An alternative is the use of federated ensembles,
ensembles of classifiers, each of which has been trained on the local data of
each party in a federated setting[7]. Ensemble based learning works by
combining multiple (weak) classifiers which work together to jointly produce
classifications using various voting schemes[8]. It relies on a diverse set of
classifiers, under the assumption that if one classifier makes a mistake the
other classifiers will correct it. This allows ensembles to achieve a high
performance, even when the individual classifiers are weak. A major advantage
of ensemble learning is that it can deal with non-IID data[9]. It can even
take advantage of the dependencies by using (dynamically) weighted voting
schemes. For example, an ensemble of experts can weigh the votes of
classifiers trained on a similar population, or trained on specific sub-tasks,
more strongly[10, 11].

Current research into federated ensembles is limited[7]. There is only a small
body of current work specifically looking into federated ensembles. There are
still several general open questions, mainly:

  1. 1.

How to share class labels in a privacy preserving manner in a vertically
distributed setting.

  2. 2.

How to detect and exploit the subtle biases and dependencies that exist across
parties.

  3. 3.

How to determine correlations between the various attributes split across
parties.

In addition to these general questions, there is also room for exploring
different types of ensembles, using different base-classifiers. In this
article, we will explore the use of federated ensembles consisting of Bayesian
networks (BN). We will compare these ensembles with VertiBayes[12], a
federated implementation of BNs, as well as with a BN that was centrally
trained. We will compare the various options in terms of technical complexity,
training time required, privacy protection, and model performance.

##  2 Methods

###  2.1 Bayesian networks

Bayesian networks (BN) are widely used probabilistic graphical models
consisting of a directed acyclic graph (the structure) where each node
represents a variables and arcs represent conditional dependencies and a set
of conditional probability distributions (the parameters), one per node[13].
Their ability to combine existing expert knowledge with data has given them
great utility and popularity. In addition, their graphical representation and
probabilistic reasoning makes them relatively intuitive to understand models
for non-technical personnel. This makes them especially useful in scenarios
where non-technical personnel need to be able to understand the models, for
example when models are used to inform clinical decisions.

###  2.2 VertiBayes

VertiBayes[12] is an implementation of BN learning algorithms in a federated
environment. It works both in vertical and horizontally distributed scenarios,
as well as in a hybrid scenario. In addition to this, it can deal with missing
data[14, 15]. Furthermore, it includes a federated implementation of the K2
algorithm[16], allowing it to learn a network structure on the fly. Lastly,
VertiBayes includes several validation methods, of various computational
complexity, that can be used to validate the model in a privacy preserving
manner. This makes it an appropriate tool in a federated setting where data
quality across parties may not be guaranteed.

It has a similar performance compared to a centrally trained model. In
addition, it provides the same privacy guarantees as the n-party scalar
product protocol[17] used. However, it is considerably more time consuming to
train a model using VertiBayes than it is to train a model centrally. The time
complexity mostly depends on the number of probabilities that need to be
calculated during parameter learning.

###  2.3 Federated Bayesian Network Ensembles

Federated Bayesian Network Ensembles (FBNE) are an ensemble learning approach
where the base classifier consists of Bayesian networks. Each data owner
within the federated setting makes their own Bayesian network based on locally
available data. This local data is only enriched with the class label (should
this not be available locally) in a privacy preserving manner using
VertiBayes. The local classifiers can then be used in an ensemble to classify
a new individual.

It is possible to use FBNE in a horizontally split, vertically split and
hybrid settings. In the case of a hybrid split, one may decide to build the
models purely based on local data, or to allow the hybrid variables to also
utilize the data available at other data parties. For example, if party 1
contains attribute A & B, and party 2 contains attributes B & C, we can choose
to either build a model using only the data available locally at party 1, or
to build a model which also includes the data party 2 has regarding attribute
B. In addition, it is possible with the use of predefined structures to mix
and match variables from various parties to create the optimal ensemble.

####  2.3.1 Privacy risks

The privacy risks posed by FBNE are the same as those posed by VertiBayes.
That is to say, there are no major risks during training. However, the BNs
themselves do still contain information. The structure and CPDs included in
the BNs will be revealed if they are published. The published networks can be
used to predict missing values in a dataset by a third party. This is inherent
to how BNs work and as such is unavoidable. An ensemble of BNs poses a similar
risk.

####  2.3.2 Runtime advantages

FBNE are significantly faster than VertiBayes as the majority of the
calculations can be done locally. This minimizes the use of complex MPC needed
to preserve privacy.

####  2.3.3 Performance advantages and disadvantages

FBNE may outperform a single model. The ensembles may catch local biases that
are lost when only a single model is built. Furthermore, by using weighted
voting it becomes possible to create a mixture of experts. This is especially
advantageous if it is known that the various data parties have biases in their
data. For example, in a scenario where hospitals work together to build an
ensemble it may be useful to weigh the votes if one hospital specializes in a
certain type of patient.

####  2.3.4 Interpretability

Ensembles are generally less interpretable than a single classifier. However,
Bayesian networks are highly interpretable. While it is not possible to
directly detect interactions across the various individual classifiers within
the ensemble it is possible to use the individual classifiers to guide
research into variable interactions in a smart manner. For example, by
comparing the sparsity of the local networks to determine if certain variables
are actually of interest to the outcome variable. In addition to this, it can
be possible to use expert knowledge to deduce possible interactions. For
example take the following toy examples shown in figure 1.

Figure 1: Two local networks based on locally available data

In addition to these two models, expert knowledge indicates that poverty
increases the likelihood of smoking. Based on this expert knowledge and the
local models that were created it can be deduced that the global structure
might be similar to the network shown in figure 2

Figure 2: Global network created by combining the two local models from figure
1 while utilizing expert knowledge.

Furthermore, it should be noted that it is possible to apply feature selection
across ensembles using wrapper-based approaches. By utilizing this type of
feature selection, it may be possible to detect mediating effects between
attributes.

##  3 Experiments

We ran a range of experiments to assess the performance of FBNE111An
implementation of FBNE can be found in the following git repository:
https://github.com/MaastrichtU-CDS/bayesianEnsemble. We used the following
datasets: Iris[18], Autism[19], Asia[20], Diabetes[21], Alarm[22] and
Mushroom[23].

The Iris dataset contains 5 attributes and contains 150 individuals. It
contains no missing values. The Autism dataset has 20 attributes and contains
704 individuals. The Asia dataset contains 10.000 individuals and 8
attributes. The Alarm dataset also contains 10.000 individuals and 37
attributes. The Diabetes dataset contains 768 individuals and 9 attributes.
The mushroom dataset has 23 attributes and contains 8124 individuals.

Each dataset was tested with varying levels of missing values (no missing
values, 5%, 10%, 30% missing at random ) We tested the following federated
split scenarios:

  * •

Vertical split with the attributes split randomly between parties. Each party
has at least 2 local attributes. This was done for a setting with 2 and a
setting with 3 parties.

  * •

Horizontal split with the samples split randomly between parties. Each party
has at least 50 records. Again this was done for a setting with 2 and a
setting with 3 parties.

    * –

Additionally the horizontal splits were tested at varying levels of bias. To
induce this bias we would make it more likely for individuals of a certain
label to be put in a specific data station, e.g. individuals with the class
label “true” are more likely to be put in party 1, individuals with class
label “false” are more likely to be put in party 2. In the case of non-binary
labels, such as for the Iris dataset, the bias would be created by piting
label 1 against the rest. The following levels of bias were introduced: no
bias, 75%, 85%, and 95% bias.

  * •

Hybrid split. The attributes are randomly split in two, the samples in one of
these splits is then randomly split in two again. This results in 3 parties in
total, each with at least 2 local attributes and 50 local records.

An illustration of the various data splits can be found in tables IV, 4 and
III.

Figure 3: Horizontally split data

Figure 4: Vertically split data

Figure 5: Hybrid split data

Each of these scenarios was run 10 times for each dataset. It should be noted
that not every dataset could be used in every scenario. For example, the Iris
dataset only contains 5 attributes and thus cannot be used in a 3-party
vertically split scenario.

In addition to these random splits, an experiment was also run where data was
manually split in a “realistic” manner in a vertical setting based on expert
knowledge. These experiments attempt to represent a realistic split where
different parties collected different types of data. For example, the Autism
data set contains attributes representing answers to a questionnaire, and some
general personal attributes such as age, sex, and country of residence. This
data was split in such a way that one party had the questionnaire answers, and
the other party had the remaining attributes. As mentioned above, horizontal
bias is addressed separately in the horizontal experiments.

We compared the performance of FBNE with VertiBayes as well as a centrally
trained Bayesian network. In addition, the performance of the local models on
their local data is used as a baseline, after all if the local model already
creates a sufficiently strong classifier there is no need for a federated
model. The performance was measured by calculating the AUC using a 10-fold
cross validation.

In all cases both structure and parameters were learnt. Continuous variables
were discretized into intervals that contained at least 10% of the total
population.

##  4 Results

A selection of the results can be seen in tables tables I, II, III and IV. The
selected results illustrate the general trends seen across all experiments.
The remaining experimental results can be found in the appendix.

In all tables the highest AUC is indicated in green, the second highest in
yellow.

TABLE I: Experimental results vertically split 2-party scenarios where attributes were randomly split across parties. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Central | VertiBayes
Alarm population size: 10000 | 0 | 0,888* | 0,793† | 0,675 | 0,789 | 0,789
Asia population size: 10000 | 0 | 0,996* | 0,918 | 0,886 | 0,995† | 0,991
0.05 | 0,742† | 0,694 | 0,696 | 0,735 | 0,776*
0.1 | 0,622† | 0,594 | 0,591 | 0,615 | 0,677*
0.3 | 0,418 | 0,401 | 0,415 | 0,42† | 0,581*
Autism population size: 704 | 0 | 0,909† | 0,803 | 0,824 | 0,824 | 0,93*
0.05 | 0,8† | 0,71 | 0,727 | 0,735 | 0,853*
0.1 | 0,737† | 0,67 | 0,672 | 0,675 | 0,834*
0.3 | 0,531† | 0,484 | 0,497 | 0,498 | 0,716*
Diabetes population size: 768 | 0 | 0,811* | 0,744 | 0,69 | 0,78 | 0,808†
0.05 | 0,757† | 0,658 | 0,684 | 0,723 | 0,772*
0.1 | 0,693† | 0,597 | 0,624 | 0,668 | 0,767*
0.3 | 0,452† | 0,418 | 0,406 | 0,44 | 0,667*
Iris population size: 150 | 0 | 0,94* | 0,882 | 0,908† | 0,885 | 0,75
0.05 | 0,892* | 0,833 | 0,789 | 0,877† | 0,787
0.1 | 0,782* | 0,702 | 0,705 | 0,706† | 0,701
0.3 | 0,658† | 0,6 | 0,606 | 0,673* | 0,607
Mushroom population size: 8124 | 0 | 0,999* | 0,987† | 0,898 | 0,999* | 0,986
TABLE II: Experimental results vertically split 2-party scenarios where attributes were manually split across parties to simulate a realistic biased split. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Central | VertiBayes
Autism population size: 704 | 0 | 0,889* | 0,832 | 0,720 | 0,851† | 0,834
0.05 | 0,789* | 0,726 | 0,669 | 0,738 | 0,780†
0.1 | 0,728† | 0,678 | 0,599 | 0,689 | 0,743*
0.3 | 0,497† | 0,491 | 0,379 | 0,497† | 0,625*
Iris population size: 150 | 0 | 0,940* | 0,845 | 0,888 | 0,912† | 0,748
0.05 | 0,887* | 0,635 | 0,872 | 0,878† | 0,736
0.1 | 0,774* | 0,640 | 0,696 | 0,699† | 0,671
0.3 | 0,654† | 0,477 | 0,664 | 0,667* | 0,622
Mushroom population size: 8124 | 0 | 0,992* | 0,880 | 0,987 | 0,987† | 0,987†
TABLE III: Experimental results hybrid split 3-party scenarios where only locally available data was used in the local model. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Party 3 | Central | VertiBayes
Asia population size: 10000 | 0 | 0,996* | 0,928 | 0,940 | 0,938 | 0,995† | 0,987
0.05 | 0,742 | 0,689 | 0,670 | 0,670 | 0,746† | 0,789*
0.1 | 0,627† | 0,611 | 0,596 | 0,595 | 0,624 | 0,668*
0.3 | 0,418† | 0,400 | 0,406 | 0,407 | 0,418† | 0,568*
Autism population size: 704 | 0 | 0,911* | 0,805 | 0,821 | 0,820 | 0,833† | 0,826
0.05 | 0,803† | 0,707 | 0,718 | 0,719 | 0,741 | 0,932*
0.1 | 0,739† | 0,671 | 0,667 | 0,665 | 0,698 | 0,891*
0.3 | 0,537† | 0,473 | 0,493 | 0,493 | 0,491 | 0,783*
Diabetes population size: 768 | 0 | 0,814* | 0,700 | 0,734 | 0,735 | 0,776† | 0,781
0.05 | 0,749† | 0,685 | 0,657 | 0,656 | 0,726 | 0,753*
0.1 | 0,694† | 0,633 | 0,596 | 0,607 | 0,676 | 0,740*
0.3 | 0,452† | 0,420 | 0,407 | 0,409 | 0,440 | 0,691*
Iris population size: 150 | 0 | 0,952* | 0,912† | 0,888 | 0,894 | 0,885 | 0,700
0.05 | 0,886* | 0,784 | 0,849 | 0,853 | 0,874† | 0,805
0.1 | 0,798* | 0,725 | 0,730 | 0,725 | 0,688 | 0,642
0.3 | 0,633† | 0,603 | 0,575 | 0,582 | 0,669* | 0,610
TABLE IV: Experimental results horizontally split 2-party scenarios where records are randomly split across parties. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Central | VertiBayes
Asia population size: 10000 | 0 | 0,996* | 0,995† | 0,995† | 0,995† | 0,988
0.05 | 0,740 | 0,741† | 0,741† | 0,740 | 0,765*
0.1 | 0,624† | 0,623 | 0,623 | 0,624† | 0,670*
0.3 | 0,416 | 0,418† | 0,418† | 0,415 | 0,568*
Autism population size: 704 | 0 | 0,868* | 0,777 | 0,774 | 0,847† | 0,834
0.05 | 0,787* | 0,464 | 0,464 | 0,737 | 0,781†
0.1 | 0,691† | 0,466 | 0,463 | 0,691† | 0,746*
0.3 | 0,534† | 0,392 | 0,413 | 0,492 | 0,628*
Diabetes population size: 768 | 0 | 0,781† | 0,500 | 0,500 | 0,781† | 0,782*
0.05 | 0,725 | 0,480 | 0,480 | 0,729† | 0,752*
0.1 | 0,671 | 0,447 | 0,447 | 0,680† | 0,736*
0.3 | 0,441† | 0,431 | 0,437 | 0,437 | 0,648*
Iris population size: 150 | 0 | 0,957* | 0,903 | 0,898 | 0,925† | 0,782
0.05 | 0,877† | 0,829 | 0,837 | 0,879* | 0,759
0.1 | 0,790* | 0,749 | 0,774† | 0,7049566 | 0,675
0.3 | 0,611† | 0,494 | 0,533 | 0,677* | 0,609
Mushroom population size: 8124 | 0 | 0,988* | 0,988* | 0,988* | 0,987† | 0,987†

###  4.1 Runtime

The training process for FBNE is consistently faster than VertiBayes in every
experimental setting and for every dataset used. However, it should be noted
that the differences vary widely and depend on the dataset. These differences
are mainly driven by the different network structures. FBNE has the advantage
that most calculations can be done locally. During our experiments FBNE was
faster than VertiBayes by a factor that ranged from twice as fast to fifty
times as fast. It should be noted that it is difficult to predict per scenario
how large the processing speed gains will be as it is difficult to predict the
resulting network structure.

###  4.2 Performance

FBNE largely shows the same performance as VertiBayes, scoring similar AUC’s.
However, it should be noted that for specific datasets, and for specific data-
splits, the ensembles can perform significantly better. With the largest
difference being nearly a 0.1 difference in AUC.

In addition to outperforming VertiBayes by a relevant margin in specific
scenarios, the following general trends were visible in our experiments.
First, FBNE performs very well in scenarios with no missing data. FBNE
achieved the highest AUC in $80\%$ of the scenarios with no missing data.
However, VertiBayes performs better in scenarios with missing values.
VertiBayes achieved the highest AUC in roughly $70\%$ of the scenarios.

Another interesting trend that is visible in our experiments is that in
horizontally split scenarios the local models can perform well if the local
data quality is high, occasionally performing similarly to the federated and
centralized models. However, it should be noted that this is rare.

It is important to note that neither approach was optimized, and better models
can potentially be created. For example, a network structure generated using
expert knowledge, as opposed to using an automatic approach like the K2
algorithm, may result in better Bayesian networks. Both VertiBayes and FBNE
could benefit from such expert knowledge. In addition, weighted voting, and
especially dynamically weighted voting, can improve the performance of FBNE.

##  5 Discussion

Our experiments show that FBNE can be a suitable solution in certain
scenarios. In this section we will take a deeper dive into the differences
between FBNE and VertiBayes.

###  5.1 Runtime

The reduced training time that was observed during the experiments is a strong
advantage in favor of the federated ensembles, especially in time critical
applications or in cases when MPC solutions such as VertiBayes are too time
consuming. FBNE has this advantage due to the fact that the vast majority of
calculations can be done locally and thus requires far fewer computationally
difficult operations than VertiBayes does.

In addition to the improved runtime already observed here, it should be noted
that our implementation of the ensembles has not been fully optimized. There
is room for further improvements, especially with respect to parallelization,
which will further increase the gap in terms of runtime. However, since the
goal of this study was to simply explore the potential of FBNE, not to provide
an optimized implementation, this will not a part of this study.

###  5.2 Performance

FBNE largely showed the same performance as VertiBayes, however, in specific
scenarios it significantly outperformed VertiBayes. With the largest
difference being nearly a 0.1 difference in AUC. This does indicate that FBNE
are potentially very useful in the right situation. However, it is very
difficult to determine when this is the case without simply training the FBNE.

Two trends were visible with respect to the performance differences. First,
FBNE performs very well in scenarios with no missing data. However, VertiBayes
performs better in scenarios with missing values. There are two plausible
reasons that explain why VertiBayes performs better in these scenarios. The
first explanation is that FBNE did not have a large, or diverse, enough
ensemble in the experimental scenarios to work properly. Ensemble learning
relies on a diverse set of classifiers which can correct each other’s
mistakes, with only 2 or 3 models in our scenarios the ensembles may not be
able to do this consistently. The second possible explanation is that the
synthetic data generation step within VertiBayes allows it to bootstrap itself
for an improved performance.

Another interesting trend that is visible in our experiments is that in
horizontally split scenarios the local models occasionally perform well when
local data is of a high quality, especially when the local population is not
biased in any way. This reminds us that it is always important to ask if a
federated model is truly necessary. Building a federated model is only
worthwhile if the data added from other parties adds extra information. But if
your local data is already sufficiently large, and representative of the true
population, then a federated model may not be needed. However, if the local
data is small, or biased in some way, then a federated approach is needed.

It is important to note that neither approach was optimized, and better models
can potentially be created. For example, both VertiBayes and FBNE can benefit
from improvements, such as using expert knowledge to build the optimal
structures. In addition to improvements that could be applied to both
approaches, weighted voting, and especially dynamically weighted voting, can
improve the performance of FBNE.

###  5.3 Privacy concerns & disclosure control

Our implementation of FBNE uses VertiBayes at its core. As such, the privacy
guarantees are largely the same. However, there are two aspects in which they
potentially differ from VertiBayes:

  1. 1.

The classification of individuals, and evaluation of the model.

  2. 2.

The consequences of having multiple networks.

With FBNE, the individual classifiers can create their classifications fully
locally at the party they belong to given that required attributes for each
local model should be available locally. These individual classifications can
be combined using homomorphic encryption, resulting in a final classification
which can be shared. For example, estimated probabilities can be weighted
according to the voting power of a particular classifier, then encrypted using
an additive homomorphic encryption scheme, and all encrypted weighted
probabilities are summed. The sum is then decrypted and divided by the total
weight to get the weighted average probabilities, which determine the final
classification. Combining the votes in this way prevents any local data from
being shared and allows FBNE to be evaluated without the need of the
homomorphic encryption and a privacy preserving n-scalar product protocol[17]
or other more complex evaluation methods that VertiBayes needs
[van_daalen_vertibayes_2022]. This is a strong advantage when new samples need
to be classified or predicted in a federated manner.

The other aspect in which FBNE differs from VertiBayes is that the end-result
consists of multiple networks instead of one. Ensembles might provide a minor
advantage with respect to privacy in this case. In both cases it is possible
to learn dependencies between attributes and certain statistics about the
training set, based on the CPDs and network structures. Similarly, based on
local, incomplete data, the network can be used to predict missing values.
These are unavoidable consequences of using Bayesian networks. However,
because FBNE has the information split up over multiple networks, it will be
difficult to do this for every attribute. It is very difficult to determine
any relation between two attributes when those two attributes are split over
two Bayesian networks, as discussed in section 2.3.4. This provides some
additional privacy protection compared to a single network.

###  5.4 When should FBNE be preferred over VertiBayes

Due to the significant advantage in terms of runtime it may be beneficial to
use FBNE as an exploratory first step before deciding if using VertiBayes is
worth it. This naïve approach will already provide reasonable results.

The general advantages of each approach can be found in table V.

TABLE V: Comparison of main features of FBNE and VertiBayes FBNE |  VertiBayes
---|---
Faster |  More complete view of dependencies between attributes
Slightly better privacy guarantees |  Easier to understand & interpret than an ensemble
Possible to capture biases in local population |  May outperform FBNE when the ensemble is too small or not diverse enough
Can easily classify new samples in a privacy preserving manner |

In addition to these advantages which hold in general, one of the two may
perform better depending on how the data happened to be split. However, there
is currently no good way to predict which approach will achieve the highest
accuracy.

##  6 Conclusion

In this article, we have proposed the use of Federated Bayesian Network
Ensembles (FBNE) and assessed their usefulness in a battery of experiments. We
have shown the approach performs well in a range of situations and datasets,
often achieving similar results when compared to VertiBayes, an alternative
federated method. FBNE are significantly faster than VertiBayes, provide
slightly better privacy guarantees, and are easier to use in a scenario where
future classifications will also be done in a federated setting. On the other
hand, VertiBayes results in more interpretable models and makes it easier to
determine the dependencies between variables split over multiple sites.

The notable advantage in terms of runtime does mean that it is easily possible
to use FBNE as an initial exploratory option. Since it is currently not
possible to preemptively determine which approach will result in the highest
accuracy, using this naive approach and simply training both models might be
the best course of action for now. Additionally, this means it can be very
useful when exploring new federated datasets.

###  6.1 Future work

We would like to explore ways to determine which approach is more effective as
it would be highly beneficial to be able to know beforehand if an ensemble-
based approach will outperform a single model. Additionally, we would like to
run experiments to discover if (dynamically) weighted voting could be used to
significantly improve the performance of the ensembles. Lastly, it would be
extremely valuable if these experiments could be run on real use cases. This
would allow the experiments to work with realistic biases and remove the need
to artificially create these biases in our experimental setup. Resulting in
much more realistic experimental scenarios.

## References

  * [1] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D’Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konecný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, H. Qi, D. Ramage, R. Raskar, M. Raykova, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao, “Advances and Open Problems in Federated Learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, Jun. 2021, publisher: Now Publishers, Inc. [Online]. Available: https://www.nowpublishers.com/article/Details/MAL-083
  * [2] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_. PMLR, Apr. 2017, pp. 1273–1282, iSSN: 2640-3498. [Online]. Available: https://proceedings.mlr.press/v54/mcmahan17a.html
  * [3] C. Dwork and A. Roth, “The Algorithmic Foundations of Differential Privacy,” _Foundations and Trends® in Theoretical Computer Science_ , vol. 9, no. 3-4, pp. 211–407, Aug. 2015. [Online]. Available: http://www.nowpublishers.com/articles/foundations-and-trends-in-theoretical-computer-science/TCS-042
  * [4] P. V. Parmar, S. B. Padhar, S. N. Patel, N. I. Bhatt, and R. H. Jhaveri, “Survey of Various Homomorphic Encryption algorithms and Schemes,” _International Journal of Computer Applications_ , vol. 91, no. 8, pp. 26–32, Apr. 2014. [Online]. Available: http://research.ijcaonline.org/volume91/number8/pxc3895081.pdf
  * [5] A. C. Yao, “Protocols for secure computations,” in _23rd Annual Symposium on Foundations of Computer Science (sfcs 1982)_ , Nov. 1982, pp. 160–164, iSSN: 0272-5428.
  * [6] T. M. Deist, F. J. W. M. Dankers, P. Ojha, M. Scott Marshall, T. Janssen, C. Faivre-Finn, C. Masciocchi, V. Valentini, J. Wang, J. Chen, Z. Zhang, E. Spezi, M. Button, J. Jan Nuyttens, R. Vernhout, J. van Soest, A. Jochems, R. Monshouwer, J. Bussink, G. Price, P. Lambin, and A. Dekker, “Distributed learning on 20 000+ lung cancer patients – The Personal Health Train,” _Radiotherapy and Oncology_ , vol. 144, pp. 189–200, Mar. 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167814019334899
  * [7] F. van Daalen, L. Ippel, A. Dekker, and I. Bermejo, “Federated Ensembles: a literature review,” Dec. 2022\. [Online]. Available: https://www.researchsquare.com
  * [8] D. Opitz and R. Maclin, “Popular Ensemble Methods: An Empirical Study,” _Journal of Artificial Intelligence Research_ , vol. 11, pp. 169–198, Aug. 1999. [Online]. Available: https://jair.org/index.php/jair/article/view/10239
  * [9] F. van Daalen, E. Smirnov, N. Davarzani, R. Peeters, J. Karel, and H. B.-L. Rocca, “An Ensemble Approach to Time Dependent Classification,” in _2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA)_ , Dec. 2018, pp. 1007–1011.
  * [10] M. Reisser, C. Louizos, E. Gavves, and M. Welling, “Federated Mixture of Experts,” p. arXiv:2107.06724, Jul. 2021. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2021arXiv210706724R
  * [11] M. N. H. Nguyen, S. R. Pandey, K. Thar, N. H. Tran, M. Chen, W. S. Bradley, and C. S. Hong, “Distributed and Democratized Learning: Philosophy and Research Challenges,” _IEEE Computational Intelligence Magazine_ , vol. 16, no. 1, pp. 49–62, 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9321418/
  * [12] F. van Daalen, L. Ippel, A. Dekker, and I. Bermejo, “VertiBayes: Learning Bayesian network parameters from vertically partitioned data with missing values,” Oct. 2022, arXiv:2210.17228 [cs]. [Online]. Available: http://arxiv.org/abs/2210.17228
  * [13] J. Pearl, _Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference_. Elsevier, Jun. 2014, google-Books-ID: mn2jBQAAQBAJ.
  * [14] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” _Journal of the Royal Statistical Society. Series B (Methodological)_ , vol. 39, no. 1, pp. 1–38, 1977, publisher: [Royal Statistical Society, Wiley]. [Online]. Available: https://www.jstor.org/stable/2984875
  * [15] S. L. Lauritzen, “The EM algorithm for graphical association models with missing data,” _Computational Statistics & Data Analysis_, vol. 19, no. 2, pp. 191–201, Feb. 1995. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0167947393E0056A
  * [16] G. F. Cooper and E. Herskovits, “A Bayesian method for the induction of probabilistic networks from data,” _Machine Learning_ , vol. 9, no. 4, pp. 309–347, Oct. 1992. [Online]. Available: https://doi.org/10.1007/BF00994110
  * [17] F. van Daalen, L. Ippel, A. Dekker, and I. Bermejo, “Privacy Preserving n-Party Scalar Product Protocol,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 34, no. 4, pp. 1060–1066, Apr. 2023, conference Name: IEEE Transactions on Parallel and Distributed Systems.
  * [18] M. De Marsico, M. Nappi, D. Riccio, and H. Wechsler, “Mobile Iris Challenge Evaluation (MICHE)-I, biometric iris dataset and protocols,” _Pattern Recognition Letters_ , vol. 57, pp. 17–23, May 2015. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167865515000574
  * [19] F. Thabtah, “Autism Spectrum Disorder Screening: Machine Learning Adaptation and DSM-5 Fulfillment,” in _Proceedings of the 1st International Conference on Medical and Health Informatics 2017_ , ser. ICMHI ’17. New York, NY, USA: Association for Computing Machinery, 2017, pp. 1–6. [Online]. Available: https://doi.org/10.1145/3107514.3107515
  * [20] S. L. Lauritzen and D. J. Spiegelhalter, “Local Computations with Probabilities on Graphical Structures and Their Application to Expert Systems,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 50, no. 2, pp. 157–194, 1988, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1988.tb01721.x. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.2517-6161.1988.tb01721.x
  * [21] J. W. Smith, J. Everhart, W. Dickson, W. Knowler, and R. Johannes, “Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus,” _Proceedings of the Annual Symposium on Computer Application in Medical Care_ , pp. 261–265, Nov. 1988. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2245318/
  * [22] I. A. Beinlich, H. J. Suermondt, R. M. Chavez, and G. F. Cooper, “The ALARM Monitoring System: A Case Study with two Probabilistic Inference Techniques for Belief Networks,” _AIME 89_ , pp. 247–256, 1989, publisher: Springer, Berlin, Heidelberg. [Online]. Available: https://link.springer.com/chapter/10.1007/978-3-642-93437-7_28
  * [23] J. C. Schlimmer, “Concept acquisition through representational adjustment,” Jul. 1987. [Online]. Available: https://escholarship.org/uc/item/48r6d4z0

|  Florian van Daalen Florian van Daalen received his BSc degree in Knowledge
Engineering from University Maastricht in 2012 and his MSc degree in
Artificial Intelligence in 2014. He is currently working toward the PhD degree
in Clinical Data Science within the Clinical Data Science group, University
Maastricht, Netherlands. His research interests include privacy preserving
techniques, federated learning, and ensemble based learning.
---|---
|  Lianne Ippel Lianne Ippel received her PhD in Statistics from Tilburg
University on analyzing data streams with dependent observations, for which
she won the dissertation award from General Online Research conference (2018).
After a Postdoc at Maastricht University, she now works at Statistics
Netherlands where she works at the methodology department on international
collaborations and innovative methods for primary data collection.
---|---
|  Andre Dekker Prof. Andre Dekker, PhD (1974) is a medical physicist and
professor of Clinical Data Science at Maastricht University Medical Center and
Maastro Clinic in The Netherlands. His Clinical Data Science research group
(50 staff) focuses on 1) federated FAIR data infrastructures, 2) AI for health
outcome prediction models and 3) applying AI to improve health. Prof. Dekker
has authored over 200 publications, mentored more than 30 PhD students and
holds multiple awards and patents on the topic of federated data and AI. He
has held visiting scientist appointments at universities and companies in the
UK, Australia, Italy, USA and Canada.
---|---
|  Inigo Bermejo Inigo Bermejo received the BSc degree on Computer Engineering
from the University of the Basque Country, Spain, in 2006 and the PhD in
Intelligent Systems from UNED, Spain, in 2015. He is currently a postdoctoral
researcher at the Clinical Data Science group, Maastricht University. His
research interests include privacy preserving techniques, prediction modeling
and causal inference.
---|---

The remaining experimental results are contained in the following pages of
this appendix.

TABLE VI: Experimental results vertically split 3-party scenarios where attributes were randomly split across parties. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Party 3 | Central | VertiBayes
Alarm population size: 10000 | 0 | 0,884* | 0,790 | 0,669 | 0,561 | 0,790† | 0,790†
Asia population size: 10000 | 0 | 0,997* | 0,824 | 0,873 | 0,902 | 0,996† | 0,986
0.05 | 0,739† | 0,657 | 0,651 | 0,664 | 0,736 | 0,750*
0.1 | 0,622† | 0,607 | 0,519 | 0,583 | 0,621 | 0,704*
0.3 | 0,418† | 0,380 | 0,394 | 0,407 | 0,417 | 0,569*
Autism population size: 704 | 0 | 0,934† | 0,784 | 0,787 | 0,740 | 0,832 | 0,977*
0.05 | 0,797* | 0,742 | 0,664 | 0,434 | 0,732 | 0,780†
0.1 | 0,734* | 0,686 | 0,605 | 0,447 | 0,694 | 0,730†
0.3 | 0,497† | 0,492 | 0,388 | 0,369 | 0,489 | 0,627*
Diabetes population size: 768 | 0 | 0,801* | 0,659 | 0,647 | 0,675 | 0,789† | 0,783
0.05 | 0,753* | 0,654 | 0,630 | 0,598 | 0,728 | 0,740†
0.1 | 0,695† | 0,613 | 0,614 | 0,560 | 0,678 | 0,804*
0.3 | 0,445† | 0,407 | 0,383 | 0,382 | 0,438 | 0,552*
Mushroom population size: 8124 | 0 | 0,989* | 0,818 | 0,987 | 0,589 | 0,988† | 0,987
TABLE VII: Experimental results vertically split 3-party scenarios where attributes were manually split across parties. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Party 3 | Central | VertiBayes
Autism population size: 704 | 0 | 0,920* | 0,843 | 0,730 | 0,811 | 0,830† | 0,829
0.05 | 0,797* | 0,742 | 0,664 | 0,434 | 0,732 | 0,780†
0.1 | 0,734* | 0,686 | 0,605 | 0,447 | 0,694 | 0,730†
0.3 | 0,497† | 0,492 | 0,388 | 0,369 | 0,489 | 0,627*
Mushroom population size: 8124 | 0 | 0,991* | 0,881 | 0,986 | 0,680 | 0,988† | 0,986
TABLE VIII: Experimental results hybrid split 3-party scenarios where hybrid split attributes can fully incorperated into the local models. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Party 3 | Central | VertiBayes
Asia population size: 10000 | 0 | 0,996† | 0,885 | 0,929 | 0,929 | 0,995† | 0,999*
0.05 | 0,743 | 0,709 | 0,717 | 0,722 | 0,745† | 0,766*
0.1 | 0,623† | 0,612 | 0,554 | 0,555 | 0,619 | 0,669*
0.3 | 0,419† | 0,401 | 0,410 | 0,412 | 0,419† | 0,567*
Autism population size: 704 | 0 | 0,903* | 0,807 | 0,817 | 0,822 | 0,849† | 0,847
0.05 | 0,793* | 0,724 | 0,710 | 0,715 | 0,753 | 0,777†
0.1 | 0,740† | 0,671 | 0,667 | 0,671 | 0,682 | 0,749*
0.3 | 0,527† | 0,481 | 0,493 | 0,493 | 0,503 | 0,747*
Diabetes population size: 768 | 0 | 0,811* | 0,731 | 0,695 | 0,700 | 0,779† | 0,776
0.05 | 0,692† | 0,604 | 0,608 | 0,607 | 0,673 | 0,734*
0.1 | 0,755* | 0,670 | 0,670 | 0,673 | 0,726 | 0,752†
0.3 | 0,456† | 0,416 | 0,403 | 0,404 | 0,439 | 0,697*
Iris population size: 150 | 0 | 0,939* | 0,889† | 0,879 | 0,886 | 0,883 | 0,771
0.05 | 0,892* | 0,783 | 0,835 | 0,830 | 0,876† | 0,736
0.1 | 0,788* | 0,702 | 0,719 | 0,724† | 0,713 | 0,704
0.3 | 0,653† | 0,588 | 0,595 | 0,599 | 0,662* | 0,611
TABLE IX: Experimental results horizontally split 3-party scenarios where records are randomly split across parties. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Missing Data Level | FBNE | Party 1 | Party 2 | Party 3 | Central | VertiBayes
Asia population size: 10000 | 0 | 0,995* | 0,995* | 0,995* | 0,995* | 0,995* | 0,987
0.05 | 0,741† | 0,741† | 0,741† | 0,741† | 0,730 | 0,763*
0.1 | 0,623† | 0,623† | 0,623† | 0,623† | 0,618 | 0,674*
0.3 | 0,418† | 0,418† | 0,418† | 0,418† | 0,417 | 0,568*
Autism population size: 704 | 0 | 0,889* | 0,836 | 0,836 | 0,836 | 0,838† | 0,829
0.05 | 0,794* | 0,736 | 0,736 | 0,736 | 0,754 | 0,780†
0.1 | 0,724† | 0,687 | 0,687 | 0,687 | 0,688 | 0,749*
0.3 | 0,544† | 0,493 | 0,494 | 0,493 | 0,494 | 0,625*
Diabetes population size: 768 | 0 | 0,775 | 0,780† | 0,780† | 0,780† | 0,786* | 0,778
0.05 | 0,730† | 0,727 | 0,727 | 0,727 | 0,720 | 0,753*
0.1 | 0,674† | 0,673 | 0,673 | 0,673 | 0,667 | 0,733*
0.3 | 0,448† | 0,439 | 0,437 | 0,438 | 0,439 | 0,648*
Iris population size: 150 | 0 | 0,960* | 0,896 | 0,897† | 0,890 | 0,890 | 0,761
0.05 | 0,875 | 0,876 | 0,877† | 0,875 | 0,879* | 0,768
0.1 | 0,826* | 0,703 | 0,703 | 0,703 | 0,709 | 0,676
0.3 | 0,632 | 0,666* | 0,666* | 0,666* | 0,664† | 0,626
TABLE X: Experimental results horizontally split 2-party scenarios where records are randomly split across parties. Varies levels of bias were introduced in this experiment where the level of bias corresponds to the probability of an individual with first class label to be assigned to party 1. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Bias Level | Missing Data Level | Ensemble | Party 1 | Part 2 | Central | VertiBayes
Asia population size: 10000 |  0.75 | 0 | 0,995† | 0,995† | 0,995† | 0,996* | 0,987
0.05 | 0,743† | 0,741 | 0,741 | 0,742 | 0,765*
0.1 | 0,624† | 0,623 | 0,623 | 0,624† | 0,670*
0.3 | 0,419† | 0,419† | 0,418 | 0,417 | 0,568*
0.85 | 0 | 0,996* | 0,995† | 0,995† | 0,996* | 0,986
0.05 | 0,741† | 0,741† | 0,741† | 0,737 | 0,763*
0.1 | 0,622 | 0,623 | 0,623 | 0,628† | 0,671*
0.3 | 0,419 | 0,419 | 0,419 | 0,420† | 0,568*
0.95 | 0 | 0,995† | 0,995† | 0,995† | 0,996* | 0,986
0.05 | 0,741 | 0,741 | 0,741 | 0,742† | 0,764*
0.1 | 0,624 | 0,623 | 0,623 | 0,627† | 0,669*
0.3 | 0,420† | 0,419 | 0,419 | 0,418 | 0,567*
Autism population size: 704 |  0.75 | 0 | 0,876* | 0,500 | 0,780 | 0,845† | 0,835
0.05 | 0,786* | 0,488 | 0,487 | 0,748 | 0,780†
0.1 | 0,708† | 0,501 | 0,487 | 0,698 | 0,749*
0.3 | 0,535† | 0,380 | 0,456 | 0,500 | 0,627*
0.85 | 0 | 0,880* | 0,500 | 0,770 | 0,848† | 0,835
0.05 | 0,779* | 0,464 | 0,464 | 0,731† | 0,779*
0.1 | 0,722† | 0,445 | 0,450 | 0,688 | 0,746*
0.3 | 0,527† | 0,427 | 0,454 | 0,493 | 0,630*
0.95 | 0 | 0,898* | 0,534 | 0,500 | 0,830 | 0,834†
0.05 | 0,779* | 0,464 | 0,512 | 0,736† | 0,779*
0.1 | 0,724† | 0,443 | 0,528 | 0,675 | 0,747*
0.3 | 0,512† | 0,430 | 0,450 | 0,499 | 0,629*
Diabetes population size: 768 |  0.75 | 0 | 0,778 | 0,500 | 0,500 | 0,787* | 0,780†
0.05 | 0,737† | 0,480 | 0,480 | 0,731 | 0,753*
0.1 | 0,676† | 0,447 | 0,447 | 0,674 | 0,736*
0.3 | 0,430 | 0,405 | 0,359 | 0,445† | 0,645*
0.85 | 0 | 0,781* | 0,500 | 0,500 | 0,776† | 0,781*
0.05 | 0,730† | 0,480 | 0,480 | 0,730† | 0,754*
0.1 | 0,675† | 0,447 | 0,447 | 0,678 | 0,736*
0.3 | 0,412 | 0,349 | 0,406 | 0,444† | 0,645*
0.95 | 0 | 0,772 | 0,500 | 0,500 | 0,786* | 0,780†
0.05 | 0,581 | 0,480 | 0,499 | 0,732† | 0,754*
0.1 | 0,530 | 0,447 | 0,459 | 0,672† | 0,737*
0.3 | 0,354 | 0,370 | 0,346 | 0,442† | 0,646*
Iris population size: 150 |  0.75 | 0 | 0,942* | 0,876 | 0,735 | 0,890† | 0,767
0.05 | 0,870† | 0,727 | 0,782 | 0,875* | 0,760
0.1 | 0,802* | 0,588 | 0,703 | 0,718† | 0,669
0.3 | 0,608 | 0,544 | 0,611 | 0,676* | 0,607
0.85 | 0 | 0,950* | 0,782 | 0,692 | 0,896† | 0,779
0.05 | 0,870† | 0,625 | 0,793 | 0,879* | 0,766
0.1 | 0,746* | 0,604 | 0,712 | 0,710 | 0,678
0.3 | 0,636† | 0,478 | 0,571 | 0,669* | 0,600
0.95 | 0 | 0,915* | 0,560 | 0,653 | 0,895† | 0,780
0.05 | 0,800† | 0,497 | 0,688 | 0,873* | 0,771
0.1 | 0,752* | 0,504 | 0,641 | 0,710† | 0,667
0.3 | 0,633† | 0,469 | 0,480 | 0,673* | 0,608
TABLE XI: Experimental results horizontally split 3-party scenarios where records are randomly split across parties. Varies levels of bias were introduced in this experiment where the level of biass corresponds to the probability of an individual with first class label to be assigned to party 1. ’*’ Indicates the best performing model, ’†’ indicates the second best performing model. | AUC
---|---
Name | Bias Level | Missing Data Level | Ensemble | Party 1 | Party 2 | Party 3 | Central | VertiBayes
Asia population size: 10000 |  0.75 | 0 | 0,996* | 0,995† | 0,995† | 0,995† | 0,995† | 0,988
0.05 | 0,740 | 0,741 | 0,741 | 0,741 | 0,745† | 0,766*
0.1 | 0,622 | 0,623 | 0,623 | 0,623 | 0,624† | 0,668*
0.3 | 0,419† | 0,418 | 0,418 | 0,418 | 0,418 | 0,569*
0.85 | 0 | 0,996* | 0,995† | 0,995† | 0,995† | 0,996* | 0,986
0.05 | 0,741† | 0,741† | 0,741† | 0,741† | 0,740 | 0,769*
0.1 | 0,625 | 0,623 | 0,623 | 0,623 | 0,628† | 0,671*
0.3 | 0,419† | 0,418 | 0,418 | 0,418 | 0,417 | 0,568*
0.95 | 0 | 0,995† | 0,995† | 0,995† | 0,995† | 0,996* | 0,987
0.05 | 0,743 | 0,741 | 0,741 | 0,741 | 0,745† | 0,762*
0.1 | 0,622 | 0,623 | 0,623 | 0,623 | 0,623† | 0,671*
0.3 | 0,419† | 0,418 | 0,418 | 0,418 | 0,416 | 0,569*
Autism population size: 704 |  0.75 | 0 | 0,911* | 0,836 | 0,836 | 0,836 | 0,843† | 0,833
0.05 | 0,797* | 0,736 | 0,736 | 0,736 | 0,732 | 0,776†
0.1 | 0,728† | 0,687 | 0,687 | 0,687 | 0,700 | 0,746*
0.3 | 0,541† | 0,493 | 0,494 | 0,494 | 0,491 | 0,627*
0.85 | 0 | 0,905* | 0,836 | 0,836 | 0,836 | 0,832 | 0,842†
0.05 | 0,810* | 0,736 | 0,736 | 0,736 | 0,732 | 0,785†
0.1 | 0,739† | 0,687 | 0,688 | 0,687 | 0,687 | 0,744*
0.3 | 0,541† | 0,493 | 0,494 | 0,494 | 0,492 | 0,623*
0.95 | 0 | 0,899* | 0,836 | 0,836 | 0,836 | 0,844† | 0,835
0.05 | 0,785* | 0,736 | 0,736 | 0,736 | 0,745 | 0,780†
0.1 | 0,720† | 0,687 | 0,687 | 0,687 | 0,677 | 0,745*
0.3 | 0,490 | 0,494 | 0,494 | 0,494 | 0,495† | 0,630*
Diabetes population size: 768 |  0.75 | 0 | 0,788* | 0,780 | 0,780 | 0,780 | 0,779 | 0,786†
0.05 | 0,760† | 0,727 | 0,727 | 0,727 | 0,727 | 0,761*
0.1 | 0,690† | 0,673 | 0,672 | 0,672 | 0,680 | 0,743*
0.3 | 0,430 | 0,437 | 0,438† | 0,438† | 0,434 | 0,653*
0.85 | 0 | 0,785* | 0,780† | 0,780† | 0,780† | 0,776 | 0,777
0.05 | 0,745† | 0,727 | 0,727 | 0,727 | 0,726 | 0,747*
0.1 | 0,674 | 0,673 | 0,673 | 0,673 | 0,676† | 0,737*
0.3 | 0,394 | 0,437 | 0,438 | 0,439† | 0,438 | 0,648*
0.95 | 0 | 0,752 | 0,780† | 0,780† | 0,780† | 0,782* | 0,780†
0.05 | 0,631 | 0,727 | 0,727 | 0,727 | 0,723† | 0,755*
0.1 | 0,564 | 0,672 | 0,673 | 0,673† | 0,668 | 0,740*
0.3 | 0,353 | 0,438 | 0,438 | 0,438 | 0,443† | 0,649*


# Focusing on what to decode and what to train: Efficient Training with HOI
Split Decoders and Specific Target Guided DeNoising

Junwen Chen Yingcheng Wang Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo,
Japan
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent one-stage transformer-based methods achieve notable gains in the Human-
object Interaction Detection (HOI) task by leveraging the detection of DETR.
However, the current methods redirect the detection target of the object
decoder, and the box target is not explicitly separated from the query
embeddings, which leads to long and hard training. Furthermore, matching the
predicted HOI instances with the ground-truth is more challenging than object
detection, simply adapting training strategies from the object detection makes
the training more difficult. To clear the ambiguity between human and object
detection and share the prediction burden, we propose a novel one-stage
framework (SOV), which consists of a subject decoder, an object decoder, and a
verb decoder. Moreover, we propose a novel Specific Target Guided (STG)
DeNoising training strategy, which leverages learnable object and verb label
embeddings to guide the training and accelerate the training convergence. In
addition, for the inference part, the label-specific information is directly
fed into the decoders by initializing the query embeddings from the learnable
label embeddings. Without additional features or prior language knowledge, our
method (SOV-STG) achieves higher accuracy than the state-of-the-art method in
one-third of training epochs.

##  1 Introduction

Recent Human-Object Interaction (HOI) detection studies are mainly built on
the object detection framework. The most widely used datasets, HICO-DET [2]
and V-COCO [10], share the same object categories as the MS-COCO dataset [22].
Following the definition of the HOI instance $\\{B_{s},(B_{o},O),V\\}$, which
is a tuple of the subject (human) box $B_{s}$, the object box $B_{o}$ with
class $O$, and the verb class $V$, detecting methods are split into one-stage
and two-stage methods. In the beginning, a multi-stream architecture built on
top of a CNN-based object detector is commonly adopted in the two-stage
methods [2, 9, 28, 8]. Multi-stream methods resolve the HOI detection problem
in split parts and have a great potential to improve. By introducing the human
pose information [16, 19, 41], the language priors [7, 41], or graph structure
[7, 32, 37], CNN-based two-stage methods achieve considerable accuracy. On the
other hand, CNN-based one-stage methods [20, 42, 33] leverage interaction
points to detect possible interaction between the subject and object and
achieve promising performance.

Figure 1:  End-to-end training pipeline of our SOV-STG. Our SOV framework
splits the decoding process into three parts for each element of the HOI
instance. Our STG training strategy efficiently transfers the ground-truth
information to label embeddings through additional denoising queries.

Figure 2:  Comparison of the training convergence curves of the state-of-the-
art methods on the HICO-DET dataset. The mAP is evaluated under the default
setting of HICO-DET.

The attention mechanism of the transformer is more flexible than the CNN
architecture in handling the relationships of features at different locations
in the feature map and extracting global context information [6]. At first,
the transformer-based methods [31, 45, 4, 14] show the advantage of the
attention mechanism by adopting DETR [1] in the HOI detection task. QPIC [31]
and HOITrans [45] follow the same training pipeline as the DETR by viewing the
HOI detection problem as a set prediction problem. Without the matching
process in one-stage and two-stage CNN-based methods, QPIC and HOITrans adopt
a compact encoder-decoder architecture to predict the HOI instances directly.
However, the compact architecture with a single decoder binds the feature of
the subject and object localization and verb recognition together. As a
result, even leveraging the DETR model pretrained on the COCO dataset, the
finetuning for QPIC and HOITrans still needs 150 and 250 epochs. Following
one-stage methods [36, 21, 35, 12, 43, 34] improve the single decoder design
by disentangling the object localization and the verb recognition in a cascade
manner. Specifically, GEN-VLKT [21] improves the cascade decoder design of CDN
[36] by introducing two isolated queries of humans and objects in an instance
decoder and fusing the human and object features in an interaction decoder.
However, the subject and object detection are still tangled in the instance
decoder, and the spatial information is implicitly represented by the query
embeddings. Consequently, the training of GEN-VLKT is still hard and slow,
which needs 90 epochs. On the other hand, the two-stage transformer-based
methods [38, 24, 39] stack additional interaction pair detection modules on
top of the object decoder without modifying the subject and object detection
part. Thus, compared with one-stage methods, two-stage methods can focus on
filtering the interaction pairs and achieve higher accuracy than the one-stage
transformer-based methods with fewer training epochs.

To relieve the training burden caused by the ambiguity of the query and
decoding process in recent one-stage methods [31, 36, 21], our motivation can
be summarized in two aspects: 1) how to focus on decoding specific targets and
2) how to guide the training with specific priors. For the first aspect, we
revisit the decoding pipeline of the transformer-based method. Recent one-
stage methods [31, 36, 21] redirect the decoding target of the decoder
pretrained from the object detection task, which leads to slow training
convergence. To this end, as shown in Figure 1, according to the definition of
the HOI instance, we propose a new framework (SOV), which fully splits the
decoding process into three parts: Subject detection, Object detection, and
Verb recognition. Specifically, the object decoder, subject decoder, and verb
decoder are assigned to decode the object, subject, and verb class,
respectively. Furthermore, the spatial information (anchor boxes) and label
information (label queries) are explicitly disentangled and fed into the
decoders to guide the feature extraction. By doing so, each decoder can focus
on specific targets and share the training burden. In Figure 2, we compare the
training convergence with recent SOTA methods. From the results, SOV takes
advantage of the balanced decoding pipeline and the training converges faster
than the current SOTA methods. Moreover, the object detection part of SOV is
the same as the pretrained detection model, which makes the training more
stable and achieves a notable high accuracy at the early stage of the
training.

For the second aspect, we focus on how to obtain specific label priors to
initialize the label queries for HOI detection with an effective training
strategy. As shown in Figure 1, we introduce a novel Spcific Target Guided
(STG) denoising training strategy for HOI detection, which constructs a
connection between the ground-truth label information and predefined label
priors (embeddings) to guide the training. With the specific priors, the
queries of the inference part are able to be represented as the weighted sum
of the label embeddings by the learnable coefficient matrices. Moreover, we
leverage the verb label embeddings to guide the verb recognition in the verb
recognition part to improve the verb representation learning capabilities. In
Figure 2, we illustrate the training convergence of SOV and QPIC with STG, and
the results show that our STG strategy effectively accelerates the training
convergence before the learning rate drops and finally improves the
performance.

In summary, our contributions are mainly in two aspects: (1) we propose a
novel one-stage framework (SOV) to enable the model to concentrate on what to
detect and what to recognize; (2) we propose a novel training strategy (STG)
to allow the model to learn label-specific information from the ground-truth.
With the SOV framework design and the STG training strategy, we achieve a new
state-of-the-art performance on the HOI detection benchmark with 3× fewer
training epochs (30 epochs on HICO-DET) than the current state-of-the-art
method.

Figure 3:  The overall framework of our SOV-STG. SOV is composed of the
feature extractor and SOV decoders. The label embeddings $\bm{t}_{o}$ and
$\bm{t}_{v}$ learned by our STG training strategy are used to initialize the
label queries $\bm{Q}_{ov}$ with learned coefficient matrices $\bm{A}_{o}$ and
$\bm{A}_{v}$. The subject and object decoder leverage the learnable anchor
boxes $B_{s}$ and $B_{v}$ to predict the subject and object boxes, and the
verb boxes $B_{v}$ are generated by the adaptive shifted MBR according to the
subject and object boxes.

##  2 Related Work

Predicting interactions with specific priors. For one-stage transformer-baesd
methods, how to extract the interaction information under a predefined
representation of the interaction region is a key issue. Recent studies [3,
27, 15] attempt to leverage the deformable attention mechanism [44] to guide
the decoding by reference points. QAHOI [3] and FGAHOI [27] view the
deformable transformer decoder’s reference point as the HOI instance’s anchor
and use the anchor to guide the subject and object detection. However, QAHOI
and FGAHOI still use the HOI query embeddings to predict all the elements of
the HOI instance. MSTR [15] proposes to use the subject, object, and context
reference points to represent the HOI instance and predict the subject,
object, and verb based on the reference points. The context reference point is
defined as the center of the subject and object reference point, which follows
the idea of the interaction point [20, 33, 42]. Nevertheless, the query
embedding in MSTR is used to predict the final boxes and labels of the HOI
instance and still suffers from ambiguous representations. Besides, QAHOI,
FGAHOI, and MSTR use x-y coordinates as the spatial priors to guide the
decoding, while the box size priors are not considered. In contrast, our SOV
explicitly defines the subject and object anchor boxes as the spatial priors
and refines the anchor boxes layer by layer. Moreover, for the verb
recognition part of our SOV, we introduce a verb box, which is directly
generated from the subject and object boxes, to guide the verb feature
extraction.

Effective learning with ground-truth guided. For the object detection methods
of the DETR family [1, 44, 23], DN-DETR [18] shows that using the ground-truth
information to guide the training can accelerate the training convergence and
improve the performance. In the HOI detection task, HQM [40] encodes the
shifted ground-truth boxes as hard-positive queries to guide the training.
However, the ground-truth label information is not considered in HQM. DOQ [29]
introduces the oracle queries to implicitly encode the ground-truth boxes of
human-object pairs and the object labels, and guide the decoder to learn to
reconstruct the ground-truth HOI instances. However, DOQ implicitly encodes
the same number of oracle queries as the ground-truth with learnable weights
and only uses the learned weights during training. Without a complete and
clear usage of ground-truth information, both HQM and DOQ still need 80 epochs
to converge. Different from DOQ and HQM, we introduce denoising queries to
encode the ground-truth information and guide the training. Moreover, our STG
is used to learn the label priors for our model, and we intuitively use a
”select” and a ”weighted sum” approach to transfer the ground-truth label
information to the denoising queries and inference queries, respectively.

##  3 HOI Efficient Decoding and Training

Figure 3 shows the overall architecture of our framework. First, a feature
extractor is used to extract the multi-scale global features. Then, the global
features are fed into the object and subject decoder with learnable anchor
boxes and label queries to predict pairs of subjects and objects. The label
queries are initialized by the label embeddings and learnable coefficient
matrices. The STG training strategy (In Sec 3.2) is used to learn the label
embeddings with the ground-truth information. Finally, pairs of subject and
object embeddings and boxes are fed into the verb recognition part to predict
the verb classes. The Subject-Object (S-O) attention module and Adaptive
Shifted Minimum Bounding Rectangle (ASMBR) used to fuse the subject and object
embeddings and generate the verb box for the verb decoder are introduced in
Sec 3.1.

Figure 4:  Illustration of S-O attention module. The S-O attention consists of
three parts: a sum operation to fuse the subject and object embeddings, a
cross-attention layer to integrate the verb priors, and a bottom-up path to
amplify the intermidiate embeddings.

###  3.1 HOI split decoders

To clarify the decoding target, the design of the split decoders is crucial
for our framework. Different from recent one-stage transformer-based methods
[31, 36, 21], which use a single decoder to detect objects and subjects, We
split the detection part into two decoders, the subject decoder, and the
object decoder, and share the prediction burden of the verb decoder. Moreover,
we explore the design of the multi-branch feature fusion module and a new way
to represent the interaction region for the verb decoder.

Subject Decoder and Object Decoder. The same as recent deformable transformer
based methods [3, 15, 27], we leverage a hierarchical backbone and deformable
transformer encoder [44] as the feature extractor to extract the multi-scale
global features $\bm{f}_{g}\in\mathbb{R}^{N_{g}\times D}$, where $N_{g}$ is
the number of the total pixels of the multi-scale feature maps and $D$ is the
hidden dimension of the embeddings in the whole transformer architecture. For
the decoders, we adopt an improved deformable transformer decoder proposed in
the recent object detection method [23], which is able to process the label
queries with the constraint of anchor boxes. As shown in Figure 3, the global
features are fed into the subject and object decoder with the learnable anchor
boxes. To maintain the detection capability of the object detector, the object
decoder with the feed-forward heads is the same as the one trained in the
detection task. Furthermore, we clone the object decoder to initialize the
subject decoder and alleviate the learning burden of the subject decoder. The
subject and object decoder both use the label queries
$\bm{Q}_{ov}\in\mathbb{R}^{N_{q}\times D}$ as the input queries, where $N_{q}$
is the number of queries and $D$ is the hidden dimension of the embeddings.
With the same query input, the pair of subject embeddings
$\bm{E}_{s}\in\mathbb{R}^{N_{q}\times D}$ and object embeddings
$\bm{E}_{o}\in\mathbb{R}^{N_{q}\times D}$ can share the same prior label
information. Besides, the subject and object embeddings with corresponding
learnable anchor boxes $\bm{B}_{s}$ and $\bm{B}_{o}$ are updated layer by
layer during decoding. Then, the object embeddings from the object decoder are
used to predict the object classes, and the subject and object boxes are used
to generate the verb boxes $\bm{B}_{v}$. Next, the object and subject
embeddings are fed into the S-O attention module to fuse the verb embeddings.
Finally, the verb boxes generated from the subject and object boxes with the
verb embeddings are fed into the verb recognition part to predict the verb
classes.

Verb Decoder with S-O attention module. As shown in Figure 3, the verb decoder
receives the input queries from the outputs of the subject and object decoder.
For the label queries used in the verb decoder, we introduce S-O attention
module to fuse the subject and object embeddings in a multi-layer manner. In
Figure 4, we illustrate the fusion process of our S-O attention module. Given
the subject embedding $\bm{e}_{s_{i}}\in\mathbb{R}^{N_{q}\times D}$ and object
embedding $\bm{e}_{o_{i}}\in\mathbb{R}^{N_{q}\times D}$ from the $i$-th layer
($i>1$), first, we sum the subject and object embeddings to fuse an
intermidiate embedding $\bm{e}_{so_{i}}=(\bm{e}_{o_{i}}+\bm{e}_{s_{i}})/2$.
Then, to guide the verb recognition with our predefined label priors, the
intermidiate embeddings $\bm{e}_{so_{i}}$ are used to absorb the prior
knowledge from the learnable verb label embeddings
$\bm{t}_{v}\in\mathbb{R}^{N_{q}\times D}$ (in Sec 3.2). Specifically, we use
$\bm{e}_{so_{i}}$ as the query and $\bm{t}_{v}$ as the key and value to
perform a cross-attention operation. Furthermore, we introduce a bottom-up
path to amplify the information from the bottom to the top layer. Finally, the
verb embedding $\bm{e}_{v_{i}}$ after the bottom-up path can be defined as:

$\begin{split}\bm{e}_{v_{i}}=&((\text{CrossAttn}(\bm{e}_{so_{i-1}},\bm{t}_{v})+\bm{e}_{so_{i-1}})\\\
&+(\text{CrossAttn}(\bm{e}_{so_{i}},\bm{t}_{v})+\bm{e}_{so_{i}}))/2\end{split}$
(1)

Then, the verb embeddings from the last layer are fed into the verb decoder to
further extract the global semantic information based on the global feature
$\bm{f}_{g}$ and the verb boxes.

Figure 5:  Illustration of ASMBR. The Shifted MBR is generated by shifting the
center of the MBR. The Adaptive Shifted MBR is generated by shrinking the
width and height of the Shifted MBR.

Verb box represented by ASMBR To constrain the verb feature extraction with
positional information in the verb decoder, as shown in Figure 5, we introduce
a novel representation, adaptive shifted minimum bounding rectangle (ASMBR)
for initializing the verb box. Unlike the previous CNN-based method, UnionDet
[13], which learns a union box to guide the verb recognition, the verb box of
SOV is directly initialized from the last layer subject and object boxes from
the subject and object decoder. To balance the attention between the subject
and object, we shift the center of the MBR to the center of the subject and
object boxes. Considering the boxes will overlap with each other, we shrink
the width and height of the MBR according to the spatial relationship between
the two boxes. With the shift and adapt operation, the verb box can constrain
the interaction region for sampling points of the deformable attention and
extract interaction information from specific subject and object pairs.
Finally, given the last layer subject box
$\bm{B}_{s}=(x_{s},y_{s},w_{s},h_{s})$ and object box
$\bm{B}_{o}=(x_{o},y_{o},w_{o},h_{o})$, where $(x,y)$ indicates the box
center, the verb box is defined as:

$\bm{B}_{v}=\bigg{(}\frac{x_{s}+x_{o}}{2},\frac{y_{s}+y_{o}}{2},w_{v},h_{v}\bigg{)}$
(2)
$w_{v}=\frac{w_{s}+w_{o}}{2}+|x_{s}-x_{o}|,h_{v}=\frac{h_{s}+h_{o}}{2}+|y_{s}-y_{o}|$
(3)

###  3.2 Specific Target Guided DeNoising Training

The object and verb labels are the targets of HOI detection, the two label
embeddings, which are used to initialize the label queries can be viewed as
the specific target priors. Since the denoising queries are generated from the
specific target priors and learned during the denoising training, we call our
training strategy as Specific Target Guided (STG) denoising. In this
subsection, we first introduce the definition and usage of the label priors,
then we introduce the STG denoising training strategy.

Label-specific Priors To explicitly equip the prior label knowledge into the
decoders and disentangle the training and decoding target, as shown in Figure
1, two kinds of learnable label embeddings are used to initialize the query
embeddings for SOV decoders. Specifically, in Figure 3, we define the object
label embeddings $\bm{t}_{o}\in\mathbb{R}^{C_{o}\times D}$ as the object label
priors, which consist of $C_{o}$ vectors with $D$ dimensions, where $C_{o}$ is
the number of object classes. Similarly, the verb label embeddings
$\bm{t}_{v}\in\mathbb{R}^{C_{v}\times D}$ are defined as the verb label
priors. With the object label and verb label priors, we first initialize the
query embeddings of object label $\bm{q}_{o}\in\mathbb{R}^{N_{q}\times D}$ and
verb label $\bm{q}_{v}\in\mathbb{R}^{N_{q}\times D}$ by linear combining the
object label and verb label embeddings with two learnable coefficient matrices
$\bm{A}_{o}\in\mathbb{R}^{N_{q}\times C_{o}}$ and
$\bm{A}_{v}\in\mathbb{R}^{N_{q}\times C_{v}}$, respectively. Then, we add the
object and verb label embeddings to obtain the inference query embeddings
$\bm{q}_{ov}\in\mathbb{R}^{N_{q}\times D}$. The initialization of
$\bm{q}_{o}$, $\bm{q}_{v}$, and $\bm{q}_{ov}$ is defined as follows:

$\bm{q}_{o}=\bm{A}_{o}\bm{t}_{o},\quad\bm{q}_{v}=\bm{A}_{v}\bm{t}_{v}$ (4)
$\bm{q}_{ov}=\bm{q}_{o}+\bm{q}_{v}$ (5)

Different from DN-DETR [18] and DOQ [29], which learn an encoding weight to
generate queries only used in training, we use the label embeddings both in
the denoising and inference parts and enable the inference part to obtain the
input query with label-specific information from the beginning.

Figure 6:  Illustration of adding noise to a ground-truth HOI instance. The
initialization consists of two parts, the object label and the verb label DN
queries initialization. The final DN query embeddings $\bm{q}^{dn}_{k}$ are
concatenated with the object label DN queries $\bm{q}^{\tilde{o}}_{k}$ and the
verb label DN queries $\bm{q}^{\tilde{v}}_{k}$.

Learning Priors with DeNoising Training In Figure 6, we show the
initialization of the DN (DeNoising) query embeddings and visualize the
process of adding noise to a ground-truth HOI instance. Given the ground-truth
object label set $\bm{O}_{gt}=\\{\bm{o}_{i}\\}_{i=1}^{K}$ and verb label set
$\bm{V}_{gt}=\\{\bm{v}_{i}\\}_{i=1}^{K}$ of an image, where $\bm{o}_{i}$ and
$\bm{v}_{i}$ are the labels of the object and verb classes, $K$ is the number
of ground-truth HOI instances, we generate $N_{p}$ groups of noised labels for
each of the ground-truth HOI instances. For the $k$-th ground-truth HOI
instance, the noised object labels are obtained by randomly flipping the
ground-truth index of the object label $\bm{o}_{k}$ to another object class
index. Because the verb label $\bm{v}_{k}$ consists of co-occurrence ground-
truth classes, to keep the co-occurrence ground-truth indices appearing in the
noised verb labels, we randomly flip the other indices of the ground-truth
verb label to generate the noised verb labels. Two flipping rate hyper-
parameters $\eta_{o}\in(0,1)$ and $\eta_{v}\in(0,1)$ are used to control the
percentage of the noised object labels and verb labels, respectively. Besides,
a verb class flipping rate hyper-parameter $\lambda_{v}\in(0,1)$ is used to
control the class-wise flipping rate in the verb labels. Next, we introduce a
”select” approach to ”encode” the noised labels to DN query embeddings.
Specifically, we directly compose the object DN query embeddings
$\bm{q}^{\tilde{o}}_{k}\in\mathbb{R}^{N_{p}\times D}$ by selecting class-
specific vectors $\\{\tilde{\bm{o}}^{j}_{k}\\}_{j=1}^{N_{p}}$ from the object
label embeddings $\bm{t}_{o}$ according to the indices of the noised object
labels. For encoding of the noised verb labels, we select and sum the class-
specific vectors to construct multi-class vectors
$\\{\tilde{\bm{v}}^{j}_{k}\\}_{j=1}^{N_{p}}$, and compose the verb DN query
embeddings $\bm{q}^{\tilde{v}}_{k}\in\mathbb{R}^{N_{p}\times D}$. Finally, we
concatenate the object DN query embeddings $\bm{q}^{\tilde{o}}_{k}$ and verb
DN query embeddings $\bm{q}^{\tilde{v}}_{k}$ to form the DN query embeddings
$\bm{q}^{dn}_{k}\in\mathbb{R}^{2N_{p}\times D}$ for the denoising training.
Since the specific target priors learned by the denoising training are also
used to guide the inference during end-to-end training, our STG can accelerate
the training convergence and improve the inference performance at the same
time. In addition, motivated by the box denoising strategy of DN-DETR [18], we
scale and shift pairs of ground-truth subject and object boxes to generate
$2N_{p}$ groups of noised anchor boxes for corresponding DN query embeddings.

###  3.3 Training and Inference

As shown in Figure 1, our proposed method SOV-STG is trained in an end-to-end
manner. For the inference queries $\bm{Q}_{ov}$, the Hungarian algorithm [17]
is used to match the ground-truth HOI instances with the predicted HOI
instances, and the matching cost and the training loss of predicted HOI
instances follow the previous deformable transformer based method [3]. For the
DN queries $\bm{Q}_{dn}$, the ground-truth indices used in query
initialization are used to match the predicted HOI instances, and the loss
function is the same as the inference queries. With the basic concept that the
same ground-truth label flipping rate is difficult for the model to denoise at
the beginning of the training but becomes acceptable during the training, we
further improve the denoising strategy by introducing a dynamic DN scale
factor $\gamma\in(0,1)$ to control the object label flipping rate $\eta_{o}$
and the verb label denoising rate $\eta_{v}$ according to the training epochs.
With the dynamic DN scale strategy, the label flipping rate $\eta$ will be set
to $\gamma\cdot\eta$ at the beginning of the training and linearly increase to
$\eta$ during the training. As the label embeddings used in the denoising
training part are also the specific target priors of the inference part, SOV-
STG uses all of the parameters in training and inference.

|  |  | Default | Known Object
---|---|---|---|---
Method | Epoch | Backbone | Full | Rare | Non-Rare | Full | Rare | Non-Rare
Two-stage |  |  |  |  |  |  |  |
CATN [5] | 12 | ResNet-50 | 31.86 | 25.15 | 33.84 | 34.44 | 27.69 | 36.45
STIP [39] | 30 | ResNet-50 | 32.22 | 28.15 | 33.43 | 35.29 | 31.43 | 36.45
UPT [38] | 20 | ResNet-101-DC5 | 32.62 | 28.62 | 33.81 | 36.08 | 31.41 | 37.47
Liu et al.  [24] | 129 | ResNet-50 | 33.51 | 30.30 | 34.46 | 36.28 | 33.16 | 37.21
One-stage |  |  |  |  |  |  |  |
QAHOI [3] | 150 | ResNet-50 | 26.18 | 18.06 | 28.61 | - | - | -
QPIC [31] | 150 | ResNet-50 | 29.07 | 21.85 | 31.23 | 31.68 | 24.14 | 33.93
MSTR [15] | 50 | ResNet-50 | 31.17 | 25.31 | 32.92 | 34.02 | 28.83 | 35.57
SSRT [12] | 150 | ResNet-101 | 31.34 | 24.31 | 33.32 | - | - | -
CDN-S [36] | 100 | ResNet-50 | 31.44 | 27.39 | 32.64 | 34.09 | 29.63 | 35.42
Zhou et al. [43] | 80 | ResNet-50 | 31.75 | 27.45 | 33.03 | 34.50 | 30.13 | 35.81
CDN-L [36] | 100 | ResNet-101 | 32.07 | 27.19 | 33.53 | 34.79 | 29.48 | 36.38
HQM (CDN-S) [40] | 80 | ResNet-50 | 32.47 | 28.15 | 33.76 | 35.17 | 30.73 | 36.50
RLIP-ParSe [34] | 90 | ResNet-50 | 32.84 | 34.63 | 26.85 | - | - | -
DOQ (CDN-S) [29] | 80 | ResNet-50 | 33.28 | 29.19 | 34.50 | - | - | -
GEN-VLKT-S [21] | 90 | ResNet-50 | 33.75 | 29.25 | 35.10 | 36.78 | 32.75 | 37.99
GEN-VLKT-M [21] | 90 | ResNet-101 | 34.78 | 31.50 | 35.77 | 38.07 | 34.94 | 39.01
GEN-VLKT-L [21] | 90 | ResNet-101 | 34.95 | 31.18 | 36.08 | 38.22 | 34.36 | 39.37
QAHOI-Swin-L [3] | 150 | Swin-Large-22K | 35.78 | 29.80 | 37.56 | 37.59 | 31.36 | 39.36
FGAHOI-Swin-L [27] | 190 | Swin-Large-22K | 37.18 | 30.71 | 39.11 | 38.93 | 31.93 | 41.02
SOV-STG-S | 30 | ResNet-50 | 33.80 | 29.28 | 35.15 | 36.22 | 30.99 | 37.78
SOV-STG-M | 30 | ResNet-101 | 34.87 | 30.41 | 36.20 | 37.35 | 32.46 | 38.81
SOV-STG-L | 30 | ResNet-101 | 35.01 | 30.63 | 36.32 | 37.60 | 32.77 | 39.05
SOV-STG-Swin-L | 30 | Swin-Large-22K | 43.35 | 42.25 | 43.69 | 45.53 | 43.62 | 46.11

Table 1:  Comparison to state-of-the-arts on the HICO-DET.

# | S-O Attention Designs | Default
---|---|---
last layer | multi-layer | Attention | Full | Rare | Non-Rare
(1) | ✓ |  | S-O Fusion | 33.80 | 29.28 | 35.15
(2) | ✓ |  | S-O w/o bottom-up | 33.40 | 28.69 | 34.81
(3) |  | ✓ | S-O Fusion | 32.77 | 27.09 | 34.46
(4) | ✓ |  | Sum Fusion | 33.01 | 27.83 | 34.55
(5) |  | ✓ | Sum Fusion | 32.23 | 27.49 | 33.65

Table 5: Ablation studies for S-O attention. The ”multi-layer” indicates
feeding the fused verb embeddings to the verb decoder in a multi-layer manner.

Method | Fea. | Backbone | $AP_{role}^{S1}$ | $AP_{role}^{S2}$
---|---|---|---|---
RLIP-ParSe [34] | L | ResNet-50 | 61.9 | 64.2
MSTR [15] | A | ResNet-50 | 62.0 | 65.2
ParSe [34] | A | ResNet-50 | 62.5 | 64.8
GEN-VLKT-M [21] | L | ResNet-101 | 63.3 | 65.6
GEN-VLKT-L [21] | L | ResNet-101 | 63.6 | 65.9
CDN-L [36] | A | ResNet-101 | 63.9 | 65.9
SSRT [12] | A | ResNet-101 | 65.0 | 67.1
SOV-STG-M | A | ResNet-101 | 63.7 | 65.2
SOV-STG-L | A | ResNet-101 | 63.9 | 65.4

Table 2: Comparison on V-COCO. ’A’ and ’L’ indicate the appearance and
language features, respectively.

# | Verb Box | Default
---|---|---
Full | Rare | Non-Rare
(1) | Object Box | 33.16 | 27.21 | 34.94
(2) | Subject Box | 32.78 | 28.01 | 34.21
(3) | MBR | 33.44 | 27.84 | 35.11
(4) | SMBR | 33.41 | 28.22 | 34.97
(5) | ASMBR | 33.80 | 29.28 | 35.15

Table 4: Different designs for the verb box. The ”SMBR” indicates the Shifted
MBR.

Denoising Strategies | Default
---|---
# | Box | Obj | Verb | Full | Rare | Non-Rare
(1) |  |  |  | 32.99 | 28.28 | 34.40
(2) | ✓ |  |  | 33.27 | 29.07 | 34.53
(3) | ✓ |  | ✓ | 33.28 | 28.57 | 34.69
(4) |  | ✓ | ✓ | 33.39 | 28.82 | 34.76
(5) | ✓ | ✓ |  | 33.51 | 29.05 | 34.84
(6) | ✓ | ✓ | ✓ | 33.80 | 29.28 | 35.15

Table 6: Ablation studies for denoising strategies. The symbol of ✓ means
adding nosie to the ground-truth.

# | oDec | sDec | vDec | STG | Default
---|---|---|---|---|---
Full | Rare | Non-Rare
(1) | ✓ |  | ✓ | ✓ | 32.68 | 28.21 | 34.02
(2) | ✓ | ✓ |  | ✓ | 32.35 | 27.64 | 33.63
(3) | ✓ |  |  |  | 30.14 | 22.82 | 32.32
(4) | ✓ | ✓ |  |  | 30.62 | 24.60 | 32.42
(5) | ✓ | ✓ | ✓ |  | 31.90 | 25.92 | 33.69
(6) | ✓ | ✓ | ✓ | ✓ | 33.80 | 29.28 | 35.15

Table 3: Contributions of each module. The ”oDec”, ”sDec”, and ”vDec” denote
the object, subject, and verb decoder, respectively.

##  4 Experiments

We evaluate our proposed SOV-STG on the HICO-DET [2] and V-COCO [10] datasets
to compare with current SOTA methods and conduct extensive ablation studies to
analyze the contributions of each component and show the effectiveness of our
proposed method.

###  4.1 Experimental Settings

Dataset and Metric. The HICO-DET [2] dataset contains 38,118 images for
training and 9,658 images for the test. The 117 verb classes and 80 object
classes in HICO-DET form 600 HOI classes. According to the number of HOI
instances appearing in the dataset, the HOI classes are divided into three
categories: Full, Rare, and Non-Rare. Moreover, considering HOI instances
including or not including the unknown objects, the evaluation of HICO-DET is
divided into two settings: Default and Known Object. The V-COCO [10] dataset
contains 5,400 images for training and 4,946 images for the test. In V-COCO,
80 object classes and 29 verb classes are annotated, and two scenarios are
considered: scenario 1 with 29 verb classes and scenario 2 with 25 verb
classes. We follow the standard evaluation [2] and report the mAP scores.

Implementation Details. We use the DAB-Deformable-DETR trained on COCO [22] to
initialize the weight of the feature extractor, the subject decoder, and the
object decoder. The feature extractor consists of a ResNet-50 [11] backbone
and a 6-layer deformable transformer encoder. Similar to GEN-VLKT [21], we
implement three variants of SOV-STG by adjusting the backbone and the number
of layers in all the decoders, which are denoted as SOV-STG-S with ResNet-50
and 3-layer decoders, SOV-STG-M with ResNet-101 and 3-layer decoders, and SOV-
STG-L with ResNet-101 and 6-layer decoders. The hidden dimension of the
transformer is $D=256$, and the number of the query is set to $N_{q}=64$. For
the DN part, $2N_{p}=6$ groups of noised labels are generated for each ground-
truth HOI instance. The dynamic DN scale is set to $\gamma=\frac{2}{3}$, and
we define the maximum denoising level by setting the noising rate of the box
to $\delta_{b}=0.4$, the object label flipping rate to $\eta_{o}=0.3$, the
verb denoising rate to $\eta_{v}=0.6$, and the verb label flipping rate to
$\lambda_{v}=0.6$. We train the model with the AdamW optimizer [26] with a
learning rate of 2e-4 (except for the backbone, which is 1e-5 for HICO-DET,
2e-6 for V-COCO) and a weight decay of 1e-4. The batch size is set to 32 (4
images per GPU), and the training epochs are 30 (learning rate drops at the
20th epoch), which is one-third of the GEN-VLKT [21], and one-fifth of the
QPIC [31] and QAHOI [3]. All of the experiments are conducted on 8 NVIDIA
A6000 GPUs.

###  4.2 Comparison to State-of-the-Arts

In Table 6, we compare our proposed SOV-STG with the recent SOTA methods on
the HICO-DET dataset. Our SOV-STG-S with ResNet-50 backbone achieves 33.80 mAP
on the Full category of the Default setting. Compared with the transformer-
based one-stage methods, QAHOI and MSTR, which are based on the reference
point, SOV-STG benefits from the anchor box priors and label priors and
achieves 7.62 (29.11%) and 2.63 (8.44%) mAP improvements, respectively. Note
that, without any extra language prior knowledge [30], SOV-STG-M outperforms
GEN-VLKT-M by 0.26% in one-third of the training epochs. Our proposed
framework and learning strategy close the gap in training efficiency between
transformer-based one-stage and two-stage methods. As a result, compared with
UPT, SOV-STG-L achieves 3.42 (10.56%) mAP improvements with only 10 more
training epochs. Since our SOV-STG explicitly makes full use of the ground-
truth information, compared with DOQ, which also uses ground-truth to guide
the training, SOV-STG-S achieves 1.05% mAP improvement with less than half of
the training epochs of DOQ. Furthermore, our best model SOV-STG-Swin-L with
Swin-Large [25] backbone achieves a new SOTA performance of 43.35 mAP, which
outperforms FGAHOI-Swin-L by 14.23%. Similarly, in Table 6, SOV-STG-M achieves
63.7 mAP on $AP_{role}^{S1}$ and surpasses UPT and GEN-VLKT-L by 3.92% and
0.63%, respectively.

###  4.3 Ablation Study

We conduct all the ablation experiments on the HICO-DET dataset with the SOV-
STG-S model, and if not explicitly noticed, the same training setting is used
as the training of our SOTA model.

Contributions of proposed modules. SOV-STG is composed of flexible decoding
architecture and training strategies. To clarify the contributions of each
proposed module, in Table 3, we remove the proposed modules one by one and
conduct ablation studies on the HICO-DET dataset. The row of (5) indicates the
experiment removing the STG strategy and the S-O attention module is degraded
to a sum fusion module which is similar to GEN-VLKT [21]. From the result, the
STG strategy and S-O attention improve the performance by 5.96% on the Full
category. Moreover, without the STG strategy, our framework also achieves a
significant improvement over QPIC (ResNet-50) by 9.74% with one-fifth of the
training epochs. Next, in (4), we remove the verb decoder in (5). As the
result, comparing (4) with (5), without the verb decoder, the performance
drops by 4.01%. Then, in (3), we remove the subject decoder and the sum fusion
module, and update both the subject and object boxes by the object decoder.
Without balancing the decoding burden of the detection, compared with (4), the
performance drops by 1.57%. Furthermore, in (1) and (2), we conduct drop-one-
out experiments on the subject and verb decoder, respectively. Compared with
(1) and (2), the model without the verb decoder is worse than the model
without the subject decoder, which indicates that the verb decoder plays a
more critical role.

S-O attention module. The S-O attention module is the core module of the SOV
model, which is responsible for the fusion of the object and subject features.
To explore the strength of the S-O attention mechanism, different variants of
designs we have attempted are shown in Table 6. The result of (1) indicates
the S-O attention module used in SOV-STG-S. In (2), we remove the bottom-up
path in S-O attention. Since the bottom-up path strengthens the feature
fusion, without the information flow from lower layers, the performance drops
by 1.18%. Our verb decoder uses the fused embeddings from the last layer of
the S-O attention module as the input query, and updates the embeddings layer
by layer. In (3), we attempt to feed all layers’ fused embeddings as query
positional embeddings for the verb decoder. However, compared with our design
in (1), the accuracy drops by 3.05%.

The cross-attention enables the fused embeddings to be enriched by the verb
label embeddings. In (4), we remove the cross-attention of S-O attention, and
the attention module is degraded to a sum fusion module. From the result, the
performance drops by 2.34% compared with (1). Similarly, in (5), we also
attempt the multi-layer design of the (4), which is similar to GEN-VLKT [21],
and the performance also drops. We consider that the multi-layer design is not
suitable for the verb prediction as the deformable transformer attention
mechanism is a local attention mechanism, which focuses on different parts of
the source feature in different layers. Specifically, the sampling points in
the verb decoder are not related to the sampling points in the object and
subject decoders, which focuses on different positions of the global semantic
feature. Consequently, the multi-layer design forces the verb decoder to match
the attention of the object and subject decoders, which leads to the
performance drop.

Denoising Strategies. In Table 6, we investigate the denoising strategies of
three parts of the targets, i.e., the box coordinates, the object labels, and
the verb labels. The result of (6) indicates the result of SOV-STG-S. In (1),
we set the noise rate of box coordinates to $\delta_{b}=0$, the object label
filp rate to $\eta_{o}=0$, and the verb label filp rate to $\eta_{v}=0$, thus,
the ground-truth box coordinates, object labels, and verb labels are directly
fed into the model without any noise. From the result, the accuracy drops by
2.40% compared with the full denoising training in (6). In (3), (4), and (5),
we conduct drop-one-out experiments, and the results show that each part of
the denoising strategy is effective. For the results between (2) and (3), and
(4) and (3), the verb denoising increases the performance while it used with
the object denoising.

Figure 7: The effects of different dynamic DN scales $\gamma$, where
$\gamma=0$ means the DN rate $\gamma\cdot\eta$ is start from 0 and end at
$\eta$, and $\gamma=1$ means do not use the dynamic DN strategy.

Formulations of the verb box. To verify the effectiveness of ASMBR, we use the
verb box degraded from the ASMBR to conduct ablation studies, and the results
are shown in Table 6. From the results of (3) to (5), the adaptive and shift
operations for the MBR promote the performance of the verb box, by 1.08% on
the Full category and 5.17% on the Rare category. Furthermore, in (1) and (2),
we directly use the object or subject box as the verb box, and the results
show that the object box is better for non-rare class recognition, while the
subject box is better for rare class recognition.

Dynamic DN scales. The dynamic DN scale is used to adjust the denoising
training difficulties during the whole training session. In Figure 7, we
adjust the dynamic DN scale ($\gamma$) to reveal the effects of different
dynamic DN scales. Compared with $\gamma=1$, while $\gamma=\frac{2}{3}$, the
best performance is achieved, and the dynamic DN strategy mainly improves the
performance on the Full and Rare categories.

##  5 Conclusion and Future Work

In this paper, we proposes a novel one-stage framework, SOV with HOI split
decoders for target-specific decoding and a specific target guided denoising
strategy, STG, for efficient training. Our framework SOV-STG adopts a new
format to represent HOI instances in boxes and learns HOI-specific priors for
decoding. With the well-designed architecture and efficient training strategy,
our framework achieves state-of-the-art performance with less training cost.
Since our architecture disentangles the HOI detection by specific priors and
decoders, it is easy to improve any one of them. In the future, we are going
to incorporate the knowledge from the language models to improve performance.

## References

  * [1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [2] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In WACV, 2018.
  * [3] Junwen Chen and Keiji Yanai.  Qahoi: Query-based anchors for human-object interaction detection.  arXiv preprint arXiv:2112.08647, 2021.
  * [4] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian.  Reformulating hoi detection as adaptive set prediction.  In CVPR, 2021.
  * [5] Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan, Sheng Zhong, and Xu Zou.  Category-aware transformer network for better human-object interaction detection.  In CVPR, 2022.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [7] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang.  DRG: Dual relation graph for human-object interaction detection.  In ECCV, 2020.
  * [8] Chen Gao, Yuliang Zou, and Jia-Bin Huang.  iCAN: Instance-centric attention network for human-object interaction detection.  In BMVC, 2018.
  * [9] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He.  Detecting and recognizing human-object interactions.  In CVPR, 2018.
  * [10] Saurabh Gupta and Jitendra Malik.  Visual semantic role labeling.  arXiv preprint arXiv:1505.04474, 2015.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [12] ASM Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo.  What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions.  In CVPR, 2022.
  * [13] Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J Kim.  UnionDet: Union-level detector towards real-time human-object interaction detection.  In ECCV, 2020.
  * [14] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim.  HOTR: End-to-end human-object interaction detection with transformers.  In CVPR, 2021.
  * [15] Bumsoo Kim, Jonghwan Mun, Kyoung-Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim.  Mstr: Multi-scale transformer for end-to-end human-object interaction detection.  In CVPR, 2022.
  * [16] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon.  Detecting human-object interactions with action co-occurrence priors.  In ECCV, 2020.
  * [17] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval Res. Logist. Quart, pages 83–97, 1955.
  * [18] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, and Lei Zhang.  Dn-detr: Accelerate detr training by introducing query denoising.  In CVPR, 2022.
  * [19] Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu.  Detailed 2d-3d joint representation for human-object interaction.  In CVPR, 2020.
  * [20] Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng.  PPDM: Parallel point detection and matching for real-time human-object interaction detection.  In CVPR, 2020.
  * [21] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu.  Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection.  In CVPR, 2022.
  * [22] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In ECCV, 2014.
  * [23] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  DAB-DETR: Dynamic anchor boxes are better queries for DETR.  In ICLR, 2022.
  * [24] Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu-Wing Tai, Cewu Lu, and Chi-Keung Tang.  Interactiveness field in human-object interactions.  In CVPR, 2022.
  * [25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [26] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2018.
  * [27] Shuailei Ma, Yuefeng Wang, Shanze Wang, and Ying Wei.  Fgahoi: Fine-grained anchors for human-object interaction detection.  arXiv preprint arXiv:2301.04019, 2023.
  * [28] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu.  Learning human-object interactions by graph parsing neural networks.  In ECCV, 2018.
  * [29] Xian Qu, Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao.  Distillation using oracle queries for transformer-based human-object interaction detection.  In CVPR, 2022.
  * [30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [31] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga.  QPIC: Query-based pairwise human-object interaction detection with image-wide contextual information.  In CVPR, 2021.
  * [32] Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath.  VSGNet: Spatial attention network for detecting human object interactions using graph convolutions.  In CVPR, 2020.
  * [33] Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun.  Learning human-object interaction detection using interaction points.  In CVPR, 2020.
  * [34] Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang.  Rlip: Relational language-image pre-training for human-object interaction detection.  In NeurIPS, 2022.
  * [35] Hangjie Yuan, Mang Wang, Dong Ni, and Liangpeng Xu.  Detecting human-object interactions with object-guided cross-modal calibrated semantics.  In AAAI, 2022.
  * [36] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li.  Mining the benefits of two-stage and one-stage hoi detection.  In NeurIPS, 2021.
  * [37] Frederic Z Zhang, Dylan Campbell, and Stephen Gould.  Spatially conditioned graphs for detecting human-object interactions.  In ICCV, 2021.
  * [38] Frederic Z. Zhang, Dylan Campbell, and Stephen Gould.  Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.  In CVPR, 2022.
  * [39] Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen.  Exploring structure-aware transformer over interaction proposals for human-object interaction detection.  In CVPR, 2022.
  * [40] Xubin Zhong, Changxing Ding, Zijian Li, and Shaoli Huang.  Towards hard-positive query mining for detr-based human-object interaction detection.  In ECCV, 2022.
  * [41] Xubin Zhong, Changxing Ding, Xian Qu, and Dacheng Tao.  Polysemy deciphering network for robust human-object interaction detection.  In IJCV, 2021.
  * [42] Xubin Zhong, Xian Qu, Changxing Ding, and Dacheng Tao.  Glance and Gaze: Inferring action-aware points for one-stage human-object interaction detection.  In CVPR, 2021.
  * [43] Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang.  Human-object interaction detection via disentangled transformer.  In CVPR, 2022.
  * [44] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: Deformable transformers for end-to-end object detection.  In ICLR, 2020.
  * [45] Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al.  End-to-end human object interaction detection with hoi transformer.  In CVPR, 2021.