/(B(\log B)^{r-2}(\log\log B)^{s-1}) \ll_{\varepsilon}
\sum_{\mathbf d\in(\ZZ_{>0})^{s}}\frac{|\mu(d)|}
{\prod_{i=1}^s d_i^{\frac23-\varepsilon}}.

For $\varepsilon=\frac1{12}$, the right hand side converges by <cit.>.

Theorem <ref> follows from Proposition <ref>
and the interpretation of the leading constant that we carry out in the following section.

§.§.§ The leading constant

Here we prove that the leading constant (<ref>) can be written as

\begin{equation}
\label{eq:expected_leading_constant}
\alpha(X, \Delta)\tau(X,\Delta)\prod_{i=1}^s\frac 1{m_i},
\end{equation}

where $\alpha(X,\Delta):=\alpha(-(K_X+\Delta))$ has been introduced in
Lemma <ref>, and

\[
\tau(X,\Delta)=\int_{\overline {\mathscr X(\ZZ)}}H_{\Delta}(x) \delta_{\Delta}(x) \tau_X,
\]

where $\tau_X$ is the Tamagawa measure on the set of adelic points $X(\mathbb A_\QQ)$
defined in <cit.>,
$H_\Delta$ is the height function defined by the divisor $\Delta$ as in <cit.>,
$\overline {\mathscr X(\ZZ)}$ is the closure of the set of rational points $\mathscr X(\ZZ)$
inside the space of adelic points on $X$,
and $\delta_{\Delta}=\prod_{p\in\Omega_f}\delta_{\Delta, p}$,
where $\delta_{\Delta,p}$ is the characteristic function of the set of
Campana points in $\mathscr{X}(\ZZ_p)$ for each finite place $p$.

We observe that by  <cit.>,
the product (<ref>) agrees with the expectation formulated
in <cit.> provided that the domains of integration in the definitions of
$\tau(X,\Delta)$ (i.e., $\{x\in \overline {\mathscr X(\ZZ)} : \delta_{\Delta}(x)=1\}$) coincide.

We have

$$\tau(X,\Delta)=\left(2^{s-r}\sum_{\sigma\in\Sigma_{\max}}\prod_{j\in\ii{\sigma}} m_j\right)
\sum_{\mathbf d\in(\ZZ_{>0})^s}\left( \mu(\mathbf d) \prod_{i=1}^sc_{m_i,d_i} \right)>0.$$

For every prime number $p$, we denote by $\Fr_p$ the geometric Frobenius acting
on $\pic(\mathscr 	X_{\overline{\FF}_p})\otimes_\ZZ\QQ$, and for $t\in\CC$
we define $L_p(t,\pic(X_{\overline \QQ})):=\det(1-p^{-t} \Fr_p)^{-1}$.
Since $X$ is split, and hence $\mathscr{X}_{\overline{\FF}_p}$ is split,
we have $L_p(t,\pic(X_{\overline \QQ}))=(1-p^{-t})^{-r}$ for every prime $p$.
Let $\lambda_p:=L_p(1,\pic(X_{\overline \QQ}))=(1-p^{-1})^{-r}$,
and define $\lambda:=\lim_{t\to 1}(t-1)^{r}\prod_{p\in\Omega_f}L_p(t,\pic(X_{\overline \QQ}))$.
Then $\lambda=\lim_{t\to 1}(t-1)^{r}\zeta(t)^r=1$ by properties of the residue at $1$
of the Riemann zeta function. Let $\lambda_{\infty}:=1$.
By <cit.>, we have
$\tau_X=\lambda \prod_{v\in\Omega_\QQ}\lambda_v^{-1}\tau_{X,p}$, where $\tau_{X,v}$
denotes the local measure on $X(\QQ_v)$ defined in <cit.>.

Since split toric varieties satisfy weak approximation (e.g., <cit.>)
and $\mathscr X$ is projective, we get

\[
\tau(X,\Delta)=\int_{X(\RR)}H_{\Delta, \infty}(x)\tau_{X,\infty}
\prod_{p \in \Omega_f}\lambda_p^{-1}
\int_{ {\mathscr X(\ZZ_p)}}H_{\Delta,p}(x)\delta_{\Delta,p} (x) 	\tau_{X,p},
\]

$H_\Delta=\prod_{v\in\Omega_\QQ} H_{\Delta,v}$ as in <cit.>.
We recall that the global metrization used to define $H_\delta$ in <cit.>
is equivalent to the one determined by <cit.>.

Since $\tau_{X,\infty}$ is the measure used in <cit.> for the real place,
we have

\begin{gather*}
\int_{X(\RR)}H_{\Delta, \infty}(x)\tau_{X,\infty}
=\sum_{\sigma\in\Sigma_{\max}}\left(2^{s-r}\prod_{j \in \ii{\sigma}}
\int_{0\leq x_j\leq 1}x_j^{\frac 1{m_j}-1}\mathrm d x_j\right)\\
=2^{s-r}\sum_{\sigma\in\Sigma_{\max}}\prod_{j\in \ii{\sigma}} m_j.
\end{gather*}

Moreover, if we use the $\GG_m^r$-torsor structure on
$\pi:\mathscr Y(\ZZ_p)\to\mathscr X(\ZZ_p)$ together
with <cit.>  we get

\begin{gather*}
\int_{\mathscr X(\ZZ_p)}H_{\Delta,p}(x)\delta_{\Delta,p} (x) \tau_{X,p}
=\left(\int_{\GG_m^r(\ZZ_p)}\mathrm d z\right)^{-1}
\int_{\mathscr Y(\ZZ_p)}H_{\Delta,p}(\pi(y))\delta_{\Delta,p} (\pi(y)) \prod_{i=1}^s\mathrm d y_i,
\end{gather*}

where $dz$ and $\prod_{i=1}^s\mathrm d y_i$ are the Haar measures on $\QQ_p^r$ and $\QQ_p^s$,
respectively, induced by the Haar measure on $\QQ_p$ normalized such that $\ZZ_p$ has volume $1$,
and $\delta_{\Delta,p}\circ \pi$ is the characteristic function of the property

\[
v_p(y_i)>0 \quad  \Rightarrow \quad v_p(y_i)\geq m_i, \qquad \forall i\in\{1,\dots,s\}.
\]

We have $\int_{\GG_m^r(\ZZ_p)}\mathrm d z=(1-p^{-1})^r$.
Let $\chi$ be the characteristic function of $\mathscr Y$. By <cit.>
we have

\begin{gather*}
\int_{\mathscr Y(\ZZ_p)}H_{\Delta,p}(\pi(\mathbf y))\delta_{\Delta,p} (\pi(\mathbf y))
\prod_{i=1}^s\mathrm d y_i
=\int_{\ZZ_p^N}\chi(\mathbf y)\delta_{\Delta,p} (\pi(\mathbf y))
\prod_{i=1}^s | y_i |_p^{\frac 1{m_i}-1} \prod_{i=1}^s\mathrm d y_i\\
=\sum_{(\mathbf e)\in\{0,1\}^s}\mu((p^{e_1},\dots,p^{e_s}))
\int_{p^{e_i}\mid y_i, 1\leq i\leq s}\delta_{\Delta,p} (\pi(\mathbf y))
\prod_{i=1}^s | y_i |_p^{\frac 1{m_i}-1} \prod_{i=1}^s\mathrm d y_i\\
=\sum_{(\mathbf e)\in\{0,1\}^s}\mu((p^{e_1},\dots,p^{e_s}))
\prod_{i=1}^s \left(
(1-e_i)\int_{\ZZ_p^{\times}}\mathrm d y_i
+ \sum_{j=m_i}^\infty\int_{p^j\ZZ_p^\times}|y_i|_p^{\frac 1{m_i}-1} \mathrm d y_i
\right)\\
=\sum_{(\mathbf e)\in\{0,1\}^s}\mu((p^{e_1},\dots,p^{e_s}))
\prod_{i=1}^s\left(
\left (1-p^{-1} \right)\left(1-e_i +\sum_{j=m_i}^\infty p^{-\frac j{m_i}}\right)
\right).
\end{gather*}

We observe that the product
$\left (1-p^{-1} \right)\left (1-e_i +\sum_{j=m_i}^\infty p^{-\frac j{m_i}}\right)$ equals

\begin{gather*}
\begin{cases}
1+\sum_{j=m_i+1}^{2m_{i}-1}p^{-\frac j{m_i}}
& \text{ if } e_i=0\\
\left(1+\sum_{j=m_i+1}^{2m_{i}-1}p^{-\frac j{m_i}} \right )
\left (1+p-p^{\frac {m_i-1}{m_i}} \right)^{-1}
& \text{ if } e_i=1.
\end{cases}
\end{gather*}

So, remembering (<ref>) and (<ref>), we get

\begin{multline*}
\tau({X,\Delta})
\left(2^{s-r}\sum_{\sigma\in\Sigma_{\max}}\prod_{j\in\ii{\sigma}} m_j\right)^{-1}
\\
\left(\prod_{i=1}^s C_{m_i}\right)
\left (
\prod_{p\in\Omega_f}\sum_{\mathbf e\in\{0,1\}^s} \mu ((p^{e_1},\dots,p^{e_s}))
\prod_{i=1}^s\left (p^{-1}+G_{m_i}\left (p^{-\frac 1{m_i}}\right )\right )^{e_i}
\right).
\end{multline*}

We conclude by <cit.> as
$\prod_{p\mid d_i}(p^{-1}+G_{m_i}(p^{-\frac 1{m_i}}))\ll_\epsilon d_i^{-1+\epsilon}$
by (<ref>) for all $i\in\{1,\dots,s\}$.

To show that $\tau({X,\Delta})$ is positive it suffices to observe that

\int_{\mathscr Y(\ZZ_p)}H_{\Delta,p}(\pi(y))\delta_{\Delta,p} (\pi(y)) \prod_{i=1}^s\mathrm d y_i
\geq
\int_{\mathscr (\ZZ_p^\times)^s} \prod_{i=1}^s\mathrm d y_i =(1-p^{-1})^s >0,

as the integral on the right is the restriction of the integral on the left to the subset
$(\ZZ_p^\times)^s	\subseteq \mathscr Y(\ZZ_p)$.

§ ACKNOWLEDGEMENTS

We thank for their hospitality the organizers of the trimester program
“Reinventing Rational Points” at the Institut Henri Poincaré, where we made significant progress on this project.
The second author is supported by a NWO grant 016.Veni.173.016.

[1]
I. Arzhantsev, U. Derenthal, J. Hausen, and A. Laface.
Cox rings, volume 144 of Cambridge Studies in Advanced
Cambridge University Press, Cambridge, 2015.

[2]
V. Blomer and J. Brüdern.
Rational points on the inner product cone via the hyperbola method.
Mathematika, 63(3):780–796, 2017.

[3]
V. Blomer and J. Brüdern.
Counting in hyperbolic spikes: The diophantine analysis of
multihomogeneous diagonal equations.
J. Reine Angew. Math., 737:255–300, 2018.

[4]
P. T. Bateman and E. Grosswald.
On a theorem of Erdös and Szekeres.
Illinois J. Math., 2:88–98, 1958.

[5]
T. D. Browning and L. Q. Hu.
Counting rational points on biquadratic hypersurfaces.
Adv. Math., 349:920–940, 2019.

[6]
V. V. Batyrev and Y. I. Manin.
Sur le nombre des points rationnels de hauteur borné des
variétés algébriques.
Math. Ann., 286(1-3):27–43, 1990.

[7]
V. V. Batyrev and Yu. Tschinkel.
Rational points of bounded height on compactifications of anisotropic
Internat. Math. Res. Notices, (12):591–635, 1995.

[8]
V. V. Batyrev and Yu. Tschinkel.
Rational points on toric varieties.
In Number theory (Halifax, NS, 1994), volume 15 of CMS
Conf. Proc., pages 39–48. Amer. Math. Soc., Providence, RI, 1995.

[9]
V. Batyrev and Yu. Tschinkel.
Height zeta functions of toric varieties.
volume 82, pages 3220–3239. 1996.
Algebraic geometry, 5.

[10]
V. V. Batyrev and Yu. Tschinkel.
Manin's conjecture for toric varieties.
J. Algebraic Geom., 7(1):15–53, 1998.

[11]
T. D. Browning and K. Van Valckenborgh.
Sums of three squareful numbers.
Exp. Math., 21(2):204–211, 2012.

[12]
T. Browning and S. Yamagishi.
Arithmetic of higher-dimensional orbifolds and a mixed Waring
arXiv:1902.07782, 2019.

[13]
F. Campana.
Orbifolds, special varieties and classification theory.
Ann. Inst. Fourier (Grenoble), 54(3):499–630, 2004.

[14]
F. Campana.
Orbifoldes géométriques spéciales et classification
biméromorphe des variétés kählériennes compactes.
J. Inst. Math. Jussieu, 10(4):809–934, 2011.

[15]
F. Campana.
Special manifolds, arithmetic and hyperbolic aspects: a short survey.
In Rational points, rational curves, and entire holomorphic
curves on projective varieties, volume 654 of Contemp. Math., pages
23–52. Amer. Math. Soc., Providence, RI, 2015.

[16]
D. A. Cox, J. B. Little, and H. K. Schenck.
Toric varieties, volume 124 of Graduate Studies in
American Mathematical Society, Providence, RI, 2011.

[17]
A. Chambert-Loir and Y. Tschinkel.
Igusa integrals and volume asymptotics in analytic and adelic
Confluentes Math., 2(3):351–429, 2010.

[18]
D. A. Cox.
The homogeneous coordinate ring of a toric variety.
J. Algebraic Geom., 4(1):17–50, 1995.

[19]
R. de la Bretèche.
Compter des points d'une variété torique.
J. Number Theory, 87(2):315–331, 2001.

[20]
R. de la Bretèche.
Estimation de sommes multiples de fonctions arithmétiques.
Compositio Math., 128(3):261–298, 2001.

[21]
P. Erdős and G. Szekeres.
Über die Anzahl der Abelschen Gruppen gegebener Ordnung
und über ein verwandtes zahlentheoretisches Problem.
Acta Univ. Szeged. Sect. Sci. Math., 7(2):92 – 102, 1934.

[22]
C. Frei and M. Pieropan.
O-minimality on twisted universal torsors and Manin's conjecture
over number fields.
Ann. Sci. Éc. Norm. Supér. (4), 49(4):757–811, 2016.

[23]
Y. Gongyo, S. Okawa, A. Sannai, and S. Takagi.
Characterization of varieties of Fano type via singularities of
Cox rings.
J. Algebraic Geom., 24(1):159–182, 2015.

[24]
H. W. Gould.
Combinatorial identities, volume 139 of Pure and Applied
Morgantown, W.Va., 1972.
A standardized set of tables listing 500 binomial coefficient

[25]
D. Harari.
Weak approximation on algebraic varieties.
In Arithmetic of higher-dimensional algebraic varieties (Palo
Alto, CA, 2002), volume 226 of Progr. Math., pages 43–60.
Birkhäuser Boston, Boston, MA, 2004.

[26]
T. Mignot.
Points de hauteur bornée sur les hypersurfaces lisses des
variétés toriques.
Acta Arith., 172(1):1–97, 2016.

[27]
M. Pieropan.
Imaginary quadratic points on toric varieties via universal torsors.
Manuscripta Math., 150(3-4):415–439, 2016.

[28]
M. Pieropan, A. Smeets, S. Tanimoto, and A. Várilly-Alvarado.
Campana points of bounded height on vector group compactifications.
arXiv:1908.10263, 2019.

[29]
P. Salberger.
Tamagawa measures on universal torsors and points of bounded height
on Fano varieties.
Astérisque, (251):91–258, 1998.
Nombre et répartition de points de hauteur bornée (Paris, 1996).

[30]
D. Schindler.
Bihomogeneous forms in many variables.
J. Théor. Nombres Bordeaux, 26(2):483–506, 2014.

[31]
D. Schindler.
Manin's conjecture for certain biprojective hypersurfaces.
J. Reine Angew. Math., 714:209–250, 2016.

[32]
K. Van Valckenborgh.
Squareful numbers in hyperplanes.
Algebra Number Theory, 6(5):1019–1041, 2012.


11institutetext: University of Bonn, Germany 22institutetext: Fraunhofer IAIS,
Dresden, Germany

# Training Multimodal Systems for Classification with Multiple Objectives

Jason Armitage 11 Shramana Thakur 11 Rishi Tripathi 11 Jens Lehmann 11 2 2
Maria Maleshkova 11

###### Abstract

We learn about the world from a diverse range of sensory information.
Automated systems lack this ability as investigation has centred on processing
information presented in a single form. Adapting architectures to learn from
multiple modalities creates the potential to learn rich representations of the
world - but current multimodal systems only deliver marginal improvements on
unimodal approaches. Neural networks learn sampling noise during training with
the result that performance on unseen data is degraded. This research
introduces a second objective over the multimodal fusion process learned with
variational inference. Regularisation methods are implemented in the inner
training loop to control variance and the modular structure stabilises
performance as additional neurons are added to layers. This framework is
evaluated on a multilabel classification task with textual and visual inputs
to demonstrate the potential for multiple objectives and probabilistic methods
to lower variance and improve generalisation.

###### Keywords:

Machine Learning Multimodal Data Probabilistic Methods.

##  1 Introduction

00footnotetext: Copyright © 2020 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0).

Human experience of the world is rooted in our ability to process and
integrate information present in diverse perceptual modalities [1]. Multimodal
approaches to machine learning are motivated by this ability and aim to
develop rich representations that combine information from multiple sources
[2]. Consider a machine learning system that models events in the world by
processing inputs from online media sources. Representations of these events
take the form of text, images, video, and audio. Systems that are able to
process signals from a range of these inputs learn models that are more
complete descriptions of the represented events with resulting benefits for
inference and predictions [3]. Researchers have proposed related classifiers
for performing event detection [4], source prediction [5], and activity
recognition [6].

Multimodal machine learning presents a suite of methods for leveraging diverse
data - but the development of systems that generalise to unseen samples leads
to challenges arising both in practice and from the underlying theory of
machine learning. Limited data resources are the most pressing concern in the
first category. Data acquisition for multimodal systems is complicated by the
requirement for combinations of samples in each input modality [7]. In the
absence of large-scale data, neural networks learn sampling noise in the
training data and report low scores on unseen samples [8]. Additional
modalities also inflate parameter counts with the outcome that multimodal
systems report high accuracy during training and low accuracy at test time
[9]. Additional hidden layers can boost performance during training but
introduce the requirement to prevent the interaction of parameters across the
model from slowing convergence [10].

Multimodal fusion combines representations from constituent modalities into a
single embedding. In recent years, the deployment of neural networks to
generate fused embeddings has resulted in state-of-the-art performance on
classification tasks of textual and visual samples. In theory, multimodal
fusion methods capture information present in the input representations and
produce outputs with complimentary information. Comparison with unimodal
classifiers demonstrates that the introduction of additional modalities yields
only modest performance gains [9]. In addition to overfitting, limitations on
available data are acute when tasks require images or video.

Main contributions. We propose and build a novel approach to multimodal
classification that introduces a second objective to learn fused embeddings
trained with variational inference. To our knowledge, this use of multiple
objectives - where one function is learned with a method from inverse
probability - is unique in the research on multimodal representation learning.
We go on to show that the range of methods for calibrating parameter updates
developed within the latter approach offsets the overfitting associated with
multimodal fusion. The benefits of these proposals are demonstrated
empirically by adapting an existing end-to-end architecture to perform
multilabel classification on a dataset of 25k samples of paired images and
text. $F$-scores on classifying unseen samples provide measurement of the
contribution from introducing a second probabilistic objective and related
regularisation methods to multimodal classification tasks.

Structure of the analysis. We start with an outline of the use case identified
for multimodal representation learning followed by a detailed specification of
our proposed framework and related methods. The evaluation section presents
topline results for the system and an ablation on regularisation methods in
variational inference. Section 4 highlights the existing research informing
our work and we conclude by summarising the main findings.

##  2 Use Case

Learning on multiple modalities presents opportunities to generate rich
representations for enhancing performance on existing tasks and enables new
applications [3]. This section introduces a form of classification task where
samples are presented both in natural language and images. Systems that learn
on these modalities are applied to a range of use cases related to archived
and online media [5, 4, 3].

###  2.1 Multilabel Classification on Multiple Modalities

Label prediction underlies approaches to information retrieval [11] and
classification [12] \- and enables the downstream tasks of document retrieval
[13], textual and visual entailment [14, 15], and fact validation [16].
Assigning samples to a potential subset of multiple labels also characterises
challenges in unimodal [17] and multimodal [18] real-world applications.

Multilabel classification on image and text inputs forms a benchmark task in
the research on bimodal learning [18, 19] and is also used here to assess our
proposed approach to multimodal fusion. In creating the MM-IMDb dataset for
movie genre classification, the authors were addressing a shortfall in
training data to conduct multimodal classification [20]. The task in MM-IMDb
is an instance of multilabel prediction over multiple modalities where titles
have an average of 2.48 classes and the system undertakes a series of
independent classifications. Metrics are computed by comparing outputs $Y$
with target labels from the set $D$. The authors propose an architecture
(referenced below as the _GMU baseline_) with gates to control information
learned from modalities to perform classification. We build a version of this
system in PyTorch and include results as a benchmark (see Figure 2 and Table
1).

###  2.2 Dataset

The MM-IMDb dataset constitutes samples for 25,959 movies assigned to one or
more of 23 genre classes. Inputs processed in predicting class labels for each
sample are a text summary averaging 92.5 words and an image poster. An
additional 50 metadata fields of structured text are excluded from the
multilabel prediction task to enable assessment of systems on natural language
and images.

Figure 1: Number of samples attributed to each genre in the MM-IMDb dataset.

Systems are evaluated on a processed version of the dataset available from the
authors’ institution ***http://lisi1.unal.edu.co/mmimdb/multimodal_imdb.hdf5.
We extracted four columns from MM-IMDb: FileID, Genres (in one-hot encoded
format), VGG16 image embeddings, word2vec text embeddings. Text embeddings are
$300-dimension$ word2vec representations and image embeddings are
$4096-dimension$ representations of features extracted by Arevalo et al. [20].
Image embeddings, text embeddings, and one-hot-encoded true labels are stored
as separate tensors ahead of training. Training and cross-validation are
performed with 70% of the data and systems are evaluated on the remaining 30%
of samples.

###  2.3 Variance in Multimodal Classification Systems

High variance is a core challenge to system performance at test time in
multimodal classification tasks [21]. Arevalo et al. [20] note the
improvements that regularisation methods contribute to the architecture
proposed for conducting classification on the MM-IMDb dataset. As a starting
point, we examine the effects of excluding batchnorm [10] and constraining
weight updates to an upper bound (max-norm) [8] on system performance.
Validation accuracy curves in Figure 2 run to one epoch after the maximum
$weighted-F1$ score (see Section 4) observed for different versions of the
baseline system. The negative impact of variance is visible after only a few
epochs when these methods are excluded from the system.

Figure 2: Performance on the validation set of MM-IMDb by $weighted-F1$ for
the GMU baseline system and a version excluding regularisation methods. Curves
are plotted to maximum score +1 epoch.

##  3 Approach

We have examined the importance of regularisation in the use case above and
continue with our proposal to provide additional controls for calibrating
updates to a set of parameters $\Theta$. In this section, we introduce a
classification framework with multimodal fusion comprised of two modules
trained with separate loss functions and a single computation of gradients
w.r.t. inputs. This framework is the basis for our investigation into
mitigating variance with the aim of improving classification performance over
multiple modalities and is referred to as _PM+MO_ below.

###  3.1 Classification Framework with Multimodal Fusion

A multilabel classification framework with bimodal inputs is a function $h(x)$
that takes pairs of samples
$(x_{i}^{t},x_{i}^{i})=((x_{1}^{t},x_{1}^{i}),(x_{2}^{t},x_{2}^{i}),...(x_{n}^{t},x_{n}^{i}))$
where $t$ and $i$ are text and image representations respectively. The
resulting multimodal representations are mapped to a subset of labels
$S\subseteq D$ or $d_{i}$ and each output is classified
$y=(y_{1},y_{2},...y_{n})\in\\{0,1\\}.$ In the proposed framework, the two
objectives in $h(f_{1}(x),f_{2}(z))$ are computed sequentially with a single
step of gradient computations. Module $A$ learns the function $f_{1}(x)$ for
multimodal fusion with a variational inference framework and ELBO as the
objective. Module $C$ conducts multilabel classification $f_{2}(z)$ on the
outputs of $A$ by optimising a standard loss described directly below.

Three wide hidden layers are the basis of the fused embedding module $A$. As
with Arevalo et al. [20], input embeddings $v^{t}$ and $v^{i}$ are assigned to
linear functions and hyperbolic tangent $tanh(u_{i}^{l})$ activations where
$u$ is the hyperbolic angle. In our proposal, weights and biases of hidden
layers $[w_{i,j}^{I,l},b_{i}^{l}]$ are random variables drawn from a Laplace
$L$ distribution $\theta_{j}^{l}\sim L(\mu_{j},\sigma_{j})$ with mean
$\mu_{j}$ and variance $\sigma_{j}$ optimised during training. Outputs from
the unimodal embedding layers are fused using concatenation (1) and mixing (2)
operations:

$v_{j}^{cat}=[v_{j}^{t,o},v_{j}^{i,o}]$ (1)
$z_{j}=v_{j}^{cat}*v_{j}^{i,o}+(1-v_{j}^{cat})*v_{j}^{t,o}$ (2)

Loss on $A$ is computed with stochastic variational inference using the
variants of ELBO detailed in Section 3.2. Multimodal embeddings $v^{m}$ form
the inputs for the classifier module $C$. We align with Arevalo et al. [20] in
implementing a multilayer perceptron with maxout activation
$max_{j}(w_{i,j}^{I,l}x,b_{i}^{l})$ as proposed by Goodfellow et al. [22]. A
wide hidden layer receives $v^{m}$ and the $max$ of parameters for $v_{[}m,o]$
are taken during activation. Binary cross-entropy combines sigmoid activation
with cross-entropy loss to assign a probability for each class to outputs
$(y_{1},y_{2},...y_{n})\in\\{0,1\\}$ and summing the results
$\sum_{c=1}^{M}y_{j,c},log(p_{j,c}).$

Regularisation methods recommended by Arevalo et al. [20] \- and retained in
our framework - consist of batchnorm to learn $\gamma z+\beta$ for $\mu[z]$
and $[z]$ on batches $\beta$, max-norm to constrain weight updates
$||w_{j}^{l}||$, and dropout with maxout activation in $C$. Both modules are
optimised with variants of the Adam algorithm incorporating regularisation.
Adam with gradient clipping in $A$ \- as implemented in Pyro PPL [23] \- is
run on each parameter during steps of variational inference. In the case of
$C$, AdamW was used in place of Adam after initial testing. This algorithm
acts on Loschilov and Hutter’s proposal to replace L2 regularisation with
decoupled weight decay when Adam is the optimiser [24].

###  3.2 Multimodal Fusion with Variational Inference

Multimodal embeddings are learned by inferring $[w_{i,j}^{I,l},b_{i}^{l}]$ for
the layers of $A$ using variational inference. In each case, we assume that
the posterior $p(\theta_{j}^{l}|X,Y)$ is drawn from a family of Laplace
distributions $Q$. Computing the integrals for $p$ is intractable and so we
infer an approximate candidate from $Q$ by selecting $q_{i}$ with the lowest
KL divergence from $p$ [25].The posterior expression is reduced to
$p(\theta_{j}^{l}|x)$ and defined as:

$p(\theta_{j}^{l}|x)=\frac{p(\theta_{j}^{l},x)}{p(x)}.$ (3)

Minimising the KL divergence

$KL(q(\theta_{j}^{l})||p(\theta_{j}^{l}|x))$ (4)

is equivalent to maximising the evidence lower bound (ELBO) [26]

$KL(q(\theta_{j}^{l})||p(\theta_{j}^{l}|x))=-(E_{q}[logp(\theta_{j}^{l},x)]-E_{q}[logq(\theta_{j}^{l})]$
(5)

where $E_{q}$ is the expected value under $q$. In practice the parameters are
stochastic gradients sampled from the optimal variational distribution.

Three variants of ELBO are evaluated in trained versions of the PM+MO
framework. The first implementation (ELBOv1) samples $s$ from $q_{i}$ and
computes the expected value in a basic form:

$E_{q_{s(x)}}[logp(\theta_{j}^{l},x)-logq_{s}(x)].$ (6)

ELBOv2 [23] uses the Rao-Blackwellization strategy proposed by Ranganath et
al. to reduce variance when estimating gradients by replacing random variables
with conditional expectations of the variables [27]. The third variant
($\lambda KL$) also addresses variance in gradient estimation by including a
term to limit the regularising influence of the KL term at initiation - and
then scaling up the level as training progresses [28]. L1 and L2 norms in the
training steps of variational inference present additional controls to
regulate parameter updates.

##  4 Evaluation

An analysis of the PM+MO framework comprises training and testing system
variants on the task of multilabel classification on text and images from the
MM-IMDb dataset. System performance is measured with $micro-F1$, $macro-F1$,
$weighted-F1$, and $samples-F1$. $F1$ is a standard metric for measuring
accuracy in multiclass classification tasks [29] and is computed as the mean
of precision and recall

$f_{1}^{sample}=\frac{1}{N}\sum_{N}^{i=1}\frac{2|\hat{y_{i}}\cap
y_{i}|}{|\hat{y_{i}}|+|y_{i}|}$ (7)

where $N$ is the number of samples and $y_{i}$ is the tuple of predictions.
Each of the four methods is an average of $F$-scores computed in the following
ways: per sample (samples), across all system outputs (micro), by genre
(macro), or by genre and with a weighted average on positive samples for each
label. Performance at system level is reported for all of these metrics and
$weighted-F1$ is referenced in comparisons between systems in the text.

###  4.1 System Configuration

Systems in the evaluation are all trained on a single Tesla K80 GPU and with a
batch size of 512. Priors for the weights and biases of each layer in builds
with variational inference are modeled using Laplace distributions initialised
at $\mu=0.1$ and $\sigma=0.01$. Parameters for these distributions are learned
during gradient steps with the three variants of ELBO detailed above. In the
version with KL scaling ($\lambda KL$), the scaling term is set at
$\lambda=0.2$ following tests in the range $(0.1,1.0)$. L1 and L2 updates to
parameters are set by a different $\lambda=0.1$ in all tests.

###  4.2 Results

Experiments aim to measure the impact of training with multiple objectives,
variational inference, and probabilistic regularisation methods when
conducting multimodal fusion for classification tasks. Assessment starts with
a comparison of the best performing version of the PM+MO framework - PM+MO
($\lambda KL+L2$) - with the GMU baseline. An ablation analysis on several
PM+MO variants provides a granular analysis of regularisation methods
associated with variational inference. Reported numbers are the means of
scores calculated over five complete cycles of training and testing.

Table 1: F-scores for PM+MO and GMU Baseline (mean over 5 cycles) System | F-score
---|---
|  Micro |  Macro |  Weighted |  Samples
PM+MO ($\lambda KL+L2$) |  0.620 |  0.549 |  0.617 |  0.620
PM+MO ($\lambda KL+L2+1024$) |  0.602 |  0.524 |  0.599 |  0.607
GMU Baseline |  0.618 |  0.528 |  0.608 |  0.617

Topline results for our proposed framework and the GMU baseline are presented
in Table 1. Hyperparameter settings were optimised for each system with
differences in learning rate (PM+MO=0.005, GMU=0.001) and dropout (PM+MO=0.9,
GMU=0.7). A version of the PM+MO framework with a fused embedding module
including ELBO+KL scaling and L2 regularisation in the fused embedding model -
and wide layers with 3000 neurons - scored higher on all $F$-scores than the
GMU baseline ($weighted-F1=+0.009$). The baseline combines a Gated Multimodal
Unit and a simple classifier with maxout activation - and is trained end-to-
end with a single binary cross-entropy objective. Regularisation and
hyperparameter settings conform to specifications shared in the publication
[20] and repository. Code conversion from Theano to PyTorch is a contributor
to the difference in scores for this build of the GMU system against those
stated in the original research ($weighted-F1=0.617$).

A version of our framework with 1024 neurons in each linear layer completes
the topline analysis. Lower scores for this approach underline the benefits of
training with wide layers when regularisation offsets variance. As a final
check of the benefits of training with a combination of variational inference
and wide layers, we tested a version of the GMU baseline with wide layers and
observed lower accuracy to the reported system. Training time per epoch on a
single GPU for the best performing PM+MO system is 5.25 secs compared to 1.10
secs for the GMU baseline. Total training time for the former is still low at
2m11s - but extended training times are significant considerations in large-
scale data regimes [30].

Figure 3: PM+MO and GMU Baseline performance on genres by $weighted-F1$ (mean
over 5 cycles).

Measurements of accuracy on individual classes are presented in Figure 3. The
most performant PM+MO system reported higher $weighted-F1$ scores in relation
to the GMU baseline for 15 of the 23 movie genres. Classification accuracy
matched or exceeded the baseline on all genres where $weighted-F1$ for the
latter system was less than 0.5.

Table 2: Ablation for PM+MO with F-scores (mean over 5 cycles) Specification | F-score
---|---
|  Micro |  Macro |  Weighted |  Samples
PM+MO ($\lambda KL+L2$) |  0.620 |  0.549 |  0.617 |  0.620
PM+MO ($\lambda KL+L1$) |  0.612 |  0.545 |  0.613 |  0.611
PM+MO ($\lambda KL$) |  0.612 |  0.544 |  0.611 |  0.612
PM+MO (ELBOv2+L2) |  0.618 |  0.543 |  0.614 |  0.619
PM+MO (ELBOv2) |  0.615 |  0.541 |  0.611 |  0.616
PM+MO (ELBOv1+L2) |  0.617 |  0.544 |  0.614 |  0.618
PM+MO (ELBOv1) |  0.612 |  0.543 |  0.609 |  0.613
M+MO (2 units minus VI) |  0.611 |  0.529 |  0.602 |  0.611

Scores for several versions of the PM+MO framework are presented in Table 2
with the objective of comparing the impact of regularisation strategies.
Hyperparameter settings are uniform across all runs with the exception of the
specific methods noted in rows. ELBO versions incorporating methods for
managing variance outperform basic implementations of ELBO (ie ELBOv1). KL
scaling with L2 regularisation delivers a marginal improvement
($weighted-F1=+0.003$) on the same configuration with Rao-Blackwellization
(ELBOv2+L2). Supplementary L2 norm penalties on parameter updates boost
accuracy on all configurations. A build with multiple objectives and excluding
variational inference (M+MO) returns the lowest $weighted-F1$ ($-0.015$ w.r.t.
PM+MO ($\lambda KL+L2$)).

##  5 Related Work

###  5.1 Multimodal Representation Learning

Researchers have investigated the role of neural networks in combining
representations from multiple modalities to perform end tasks for several
decades [31]. Multimodal fusion is deployed in classification tasks when all
constituent modalities are present during training and inference [32].
Coordinated embedding methods are an alternative method for these tasks [3].
Separation between vectors is retained in these approaches by projecting the
textual and visual representations into a common $d-dimensional$ space and
introducing a constraint [33]. Fusion-based learning results in a single
output vector: one advantage for sticking with this approach in our framework
is to facilitate transfer between modules. Multimodal embeddings are also
learned by Silberer and Lapata to perform word similarity and object
classification [34]. The process proposed for this and related methods [34,
35] differs from the method in our system by including Singular Value
Decomposition (SVD) to integrate constituent embeddings.

###  5.2 Multiple Modules and Objectives

System architectures composed of multiple modules form a foundational area in
the research on neural networks. A primary objective in this literature is the
construction of classification frameworks that generalise to unseen samples
[36]. Auda et al. [37] detailed several approaches to decomposing tasks and
designed a classifier with multiple modules to solve components for separate
sub-tasks. In this case, a voting layer acted as a constraint on outputs from
individual components [38]. Early implementations of modular architectures for
task decomposition were trained with a single objective - or were separated
into distinct models. Secondary losses are implemented during training in the
related areas of representation learning and transfer learning. Our system
approximates Zhang et al’s proposal to introduce an auxiliary objective and
module into a classification framework [36]. The method selection in this
research differs to ours in implementing unsupervised learning for the
auxiliary components. Du et al. [39] proposed a system for measuring cosine
similarity between auxiliary and main losses when the former contributes to
the latter [39]. In contrast to our work, positive transfer with multiple
losses are applied in instances where source and target tasks share related
objectives.

###  5.3 Probabilistic Deep Learning

Probabilistic methods in this research extend an approach to machine learning
where the assessment of architectures is based on inverse probability [40].
Here the plausibility of the model - or in our case, the parameters in each
layer - are computed w.r.t. to the data. Variational inference is a non-
deterministic method that replaces elements in probabilistic inference with
approximations when the computation of integrals is intractable. A family of
distributions is placed over the model parameters and the candidate
distribution with the lowest KL divergence from the true posterior is selected
[27]. ELBO formulates this minimisation as optimisation and rewards candidate
distributions that maximise both $p(z|x)$ and a spread of uncertainty. The
ELBO term in our system extends optimisation with simple operations to
regularise parameter updates. Ma et al. [41] modify ELBO with a supplementary
regularisation term to improve representation learning - although the
objective of this technique is to reward diversity in the selection of
candidate distributions [41]. In contrast to our proposals, enhancements or
substitutes for ELBO in this and other research are centred on variational
autoencoder (VAE) architectures [42, 43]. The selection of Laplace
distributions in our system is informed by the ability of these distributions
to model data with a high level of heterogeneity [44]. Stochastic variational
inference and related methods are implemented in our system using the Pyro PPL
[23].

###  5.4 Regularisation Methods

Our framework retains means for reducing variance when conducting multimodal
fusion proposed by Arevalo et al. [20]. Batch normalisation was introduced to
minimise the impact of changes in parameters on the distributions for
activations [10]. Goodfellow et al. describe the maxout activation function as
an averaging technique in neural network-based architectures that compliments
dropout [22]. In initial testing, we verified the performance gain from
selecting maxout activation in the classifier module and retained it in the
PM+MO architecture. Contributions that describe interactions between
parameters and optimiser algorithms [24, 45] supported our decision to test
different forms of managing weight updates. The context differs as we extend
these techniques to training representations with variational inference.

##  6 Conclusion

In this research, we have demonstrated that a framework of sub-modules trained
with variational inference as one of multiple objectives leads to improvements
in performance on multimodal classification. The proposed framework supports
wide layers and higher learning rates when compared with systems trained with
a single objective. Improvements in generalisation over multiple objective
systems that exclude variational inference are also demonstrated. An
evaluation of regularisation methods associated with variational inference
underlines the advantages of probabilistic approaches in extending options to
calibrate parameter updates during training and offset overfitting in
multimodal systems. We plan to train on additional modalities and extend
probabilistic methods in representation learning as further contributions to
the research on improving generalisation in multimodal systems.

## Acknowledgements

The project leading to this publication has received funding from the European
Union’s Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie grant agreement No 812997.

## References

  * [1] Sepulcre, J., Sabuncu, M.R., Yeo, T.B., Liu, H., Johnson, K.A.: Stepwise Connectivity of the Modal Cortex Reveals the Multimodal Organization of the Human Brain. Journal of Neuroscience 32(31), 10649–10661 (Aug 2012), http://www.jneurosci.org/cgi/doi/10.1523/JNEUROSCI.0759-12.2012
  * [2] Bruni, E., Tran, N.K., Baroni, M.: Multimodal Distributional Semantics. Journal of Artificial Intelligence Research p. 47 (2013)
  * [3] Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal Machine Learning: A Survey and Taxonomy. arXiv:1705.09406 [cs] (Aug 2017), http://arxiv.org/abs/1705.09406, arXiv: 1705.09406
  * [4] Petkos, G., Papadopoulos, S., Kompatsiaris, I.: Social event detection using multimodal clustering and integrating supervisory signals | Proceedings of the 2nd ACM International Conference on Multimedia Retrieval. In: ICMR ’12: Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (Jun 2012), https://dl.acm.org/doi/abs/10.1145/2324796.2324825, archive Location: world Library Catalog: dl.acm.org
  * [5] Ramisa, A.: Multimodal News Article Analysis. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. pp. 5136–5140. International Joint Conferences on Artificial Intelligence Organization, Melbourne, Australia (Aug 2017), https://www.ijcai.org/proceedings/2017/737
  * [6] Yang, X., Ramesh, P., Chitta, R., Madhvanath, S., Bernal, E.A., Luo, J.: Deep Multimodal Representation Learning from Temporal Data. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5066–5074. IEEE, Honolulu, HI (Jul 2017), http://ieeexplore.ieee.org/document/8100021/
  * [7] Amato, G., Behrmann, M., Bimbot, F., Caramiaux, B., Falchi, F., Garcia, A., Geurts, J., Gibert, J., Gravier, G., Holken, H., Koenitz, H., Lefebvre, S., Liutkus, A., Lotte, F., Perkis, A., Redondo, R., Turrin, E., Vieville, T., Vincent, E.: AI in the media and creative industries. arXiv:1905.04175 [cs] (May 2019), http://arxiv.org/abs/1905.04175, arXiv: 1905.04175
  * [8] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A Simple Way to Prevent Neural Networks from Overﬁtting p. 30
  * [9] Wang, W., Tran, D., Feiszli, M.: What Makes Training Multi-Modal Classification Networks Hard? arXiv:1905.12681 [cs] (Dec 2019), http://arxiv.org/abs/1905.12681, arXiv: 1905.12681
  * [10] Ioffe, S., Szegedy, C.: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167 [cs] (Mar 2015), http://arxiv.org/abs/1502.03167, arXiv: 1502.03167
  * [11] Chen, H.: Machine learning for information retrieval: Neural networks, symbolic learning, and genetic algorithms. Journal of the American Society for Information Science 46(3), 194–216 (1995), https://asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291097-4571%28199504%2946%3A3%3C194%3A%3AAID-ASI4%3E3.0.CO%3B2-S
  * [12] Aggarwal, C.C., Zhai, C.: A Survey of Text Classification Algorithms. In: Aggarwal, C.C., Zhai, C. (eds.) Mining Text Data, pp. 163–222. Springer US, Boston, MA (2012), https://doi.org/10.1007/978-1-4614-3223-4_6
  * [13] Hinton, G.E., Salakhutdinov, R.R.: Reducing the Dimensionality of Data with Neural Networks. Science 313(5786), 504–507 (Jul 2006), https://science.sciencemag.org/content/313/5786/504, publisher: American Association for the Advancement of Science Section: Report
  * [14] Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zamparelli, R.: A SICK cure for the evaluation of compositional distributional semantic models p. 8
  * [15] Xie, N., Lai, F., Doran, D., Kadav, A.: Visual Entailment: A Novel Task for Fine-Grained Image Understanding. arXiv:1901.06706 [cs] (Jan 2019), http://arxiv.org/abs/1901.06706, arXiv: 1901.06706
  * [16] Lehmann, J., Gerber, D., Morsey, M., Ngonga Ngomo, A.C.: DeFacto - Deep Fact Validation. In: Cudré-Mauroux, P., Heflin, J., Sirin, E., Tudorache, T., Euzenat, J., Hauswirth, M., Parreira, J.X., Hendler, J., Schreiber, G., Bernstein, A., Blomqvist, E. (eds.) The Semantic Web – ISWC 2012. pp. 312–327. Lecture Notes in Computer Science, Springer, Berlin, Heidelberg (2012)
  * [17] Nam, J., Kim, Y.B., Mencía, E.L., Park, S., Sarikaya, R., Fürnkranz, J.: Learning Context-DependentLabel Permutations for Multi-Label Classification p. 10
  * [18] Kiela, D., Grave, E., Joulin, A., Mikolov, T.: Efficient Large-Scale Multi-Modal Classification. arXiv:1802.02892 [cs] (Feb 2018), http://arxiv.org/abs/1802.02892, arXiv: 1802.02892
  * [19] Pang, Y., Ma, Z., Yuan, Y., Li, X., Wang, K.: Multimodal learning for multi-label image classification. In: 2011 18th IEEE International Conference on Image Processing. pp. 1797–1800. IEEE, Brussels, Belgium (Sep 2011), http://ieeexplore.ieee.org/document/6115811/
  * [20] Arevalo, J., Solorio, T., Montes-y Gómez, M., González, F.A.: Gated Multimodal Units for Information Fusion. arXiv:1702.01992 [cs, stat] (Feb 2017), http://arxiv.org/abs/1702.01992, arXiv: 1702.01992
  * [21] Radu, V., Tong, C., Bhattacharya, S., Lane, N.D., Mascolo, C., Marina, M.K., Kawsar, F.: Multimodal Deep Learning for Activity and Context Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1(4), 1–27 (Jan 2018), http://dl.acm.org/citation.cfm?doid=3178157.3161174
  * [22] Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout Networks. arXiv:1302.4389 [cs, stat] (Sep 2013), http://arxiv.org/abs/1302.4389, arXiv: 1302.4389
  * [23] Bingham, E., Chen, J.P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., Goodman, N.D.: Pyro: Deep Universal Probabilistic Programming. arXiv:1810.09538 [cs, stat] (Oct 2018), http://arxiv.org/abs/1810.09538, arXiv: 1810.09538
  * [24] Loshchilov, I., Hutter, F.: Decoupled Weight Decay Regularization. arXiv:1711.05101 [cs, math] (Jan 2019), http://arxiv.org/abs/1711.05101, arXiv: 1711.05101
  * [25] Wingate, D., Weber, T.: Automated Variational Inference in Probabilistic Programming. arXiv:1301.1299 [cs, stat] (Jan 2013), http://arxiv.org/abs/1301.1299, arXiv: 1301.1299
  * [26] Minka, T.: Divergence Measures and Message Passing (Jan 2005), https://www.microsoft.com/en-us/research/publication/divergence-measures-and-message-passing/
  * [27] Ranganath, R., Gerrish, S., Blei, D.M.: Black Box Variational Inference. arXiv:1401.0118 [cs, stat] (Dec 2013), http://arxiv.org/abs/1401.0118, arXiv: 1401.0118
  * [28] Sønderby, C.K., Raiko, T., Maaløe, L., Sønderby, S.K., Winther, O.: Ladder Variational Autoencoders. arXiv:1602.02282 [cs, stat] (May 2016), http://arxiv.org/abs/1602.02282, arXiv: 1602.02282
  * [29] Madjarov, G., Kocev, D., Gjorgjevikj, D., Dzeroski, S.: An extensive experimental comparison of methods for multi-label learning. Pattern Recognit. 45(9), 3084–3104 (2012), http://dblp.uni-trier.de/db/journals/pr/pr45.html#MadjarovKGD12
  * [30] Ma, M., Pouransari, H., Chao, D., Adya, S., Serrano, S.A., Qin, Y., Gimnicher, D., Walsh, D.: Democratizing Production-Scale Distributed Deep Learning. arXiv:1811.00143 [cs] (Nov 2018), http://arxiv.org/abs/1811.00143, arXiv: 1811.00143
  * [31] Yuhas, B., Goldstein, M., Sejnowski, T.: Integration of acoustic and visual speech signals using neural networks \textbar IEEE Communications Magazine. IEEE Communications Magazine 27 (1989), https://dl.acm.org/doi/10.1109/35.41402
  * [32] Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.: Multimodal Deep Learning (2011), https://people.csail.mit.edu/khosla/papers/icml2011_ngiam.pdf
  * [33] Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M.A., Mikolov, T.: DeViSE: A Deep Visual-Semantic Embedding Model. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26, pp. 2121–2129. Curran Associates, Inc. (2013), http://papers.nips.cc/paper/5204-devise-a-deep-visual-semantic-embedding-model.pdf
  * [34] Silberer, C., Lapata, M.: Learning Grounded Meaning Representations with Autoencoders. In: ACL (1). pp. 721–732. The Association for Computer Linguistics (2014), http://dblp.uni-trier.de/db/conf/acl/acl2014-1.html#SilbererL14
  * [35] Bruni, E., Boleda, G., Baroni, M., Tran, N.K.: Distributional Semantics in Technicolor. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 136–145. Association for Computational Linguistics, Jeju Island, Korea (Jul 2012), https://www.aclweb.org/anthology/P12-1015
  * [36] Zhang, Y., Lee, K., Lee, H.: Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification. arXiv:1606.06582 [cs] (Jun 2016), http://arxiv.org/abs/1606.06582, arXiv: 1606.06582
  * [37] Auda, G., Kamel, M.: Multimodal Deep Learning. Journal of Intelligent and Robotic Systems (1998), https://dl.acm.org/doi/10.1023/A%3A1007925203918
  * [38] Auda, G., Kamel, M., Raafat, H.: A new neural network structure with cooperative modules. In: Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94). vol. 3, pp. 1301–1306 vol.3. Orlando, FL (Jun 1994)
  * [39] Du, Y., Czarnecki, W.M., Jayakumar, S.M., Pascanu, R., Lakshminarayanan, B.: Adapting Auxiliary Losses Using Gradient Similarity. arXiv:1812.02224 [cs, stat] (Dec 2018), http://arxiv.org/abs/1812.02224, arXiv: 1812.02224
  * [40] MacKay, D.J.C.: Bayesian methods for adaptive models. Ph.D. thesis, California Institute of Technology (1992)
  * [41] Ma, X., Zhou, C., Hovy, E.: MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders. arXiv:1901.01498 [cs, stat] (Jan 2019), http://arxiv.org/abs/1901.01498, arXiv: 1901.01498
  * [42] Alemi, A.A., Poole, B., Fischer, I., Dillon, J.V., Saurous, R.A., Murphy, K.: Fixing a Broken ELBO. arXiv:1711.00464 [cs, stat] (Feb 2018), http://arxiv.org/abs/1711.00464, arXiv: 1711.00464
  * [43] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. arXiv:1606.03657 [cs, stat] (Jun 2016), http://arxiv.org/abs/1606.03657, arXiv: 1606.03657
  * [44] Geraci, M., Borja, M.C.: Notebook: The Laplace distribution. Significance 15(5), 10–11 (Oct 2018), https://rss.onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2018.01185.x
  * [45] Hanson, S.J., Pratt, L.Y.: Comparing Biases for Minimal Network Construction with Back-Propagation. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 1, pp. 177–185. Morgan-Kaufmann (1989), http://papers.nips.cc/paper/156-comparing-biases-for-minimal-network-construction-with-back-propagation.pdf


# One Agent To Rule Them All: Towards Multi-agent Conversational AI

Christopher Clarke1 Joseph J. Peper1 Karthik Krishnamurthy2 Walter Talamonti2
Kevin Leach3 Walter Lasecki∗ Yiping Kang1 Lingjia Tang1 Jason Mars1
1University of Michigan, Ann Arbor, MI
2Ford Motor Company, Dearborn, MI
3Vanderbilt University, Nashville, TN
{csclarke, jpeper, ypkang, lingjia<EMAIL_ADDRESS>
{kkrish65<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Work was done while at University of Michigan

###### Abstract

The increasing volume of commercially available conversational agents (CAs) on
the market has resulted in users being burdened with learning and adopting
multiple agents to accomplish their tasks. Though prior work has explored
supporting a multitude of domains within the design of a single agent, the
interaction experience suffers due to the large action space of desired
capabilities. To address these problems, we introduce a new task BBAI: Black-
Box Agent Integration, focusing on combining the capabilities of multiple
black-box CAs at scale. We explore two techniques: _question agent pairing_
and _question response pairing_ aimed at resolving this task. Leveraging these
techniques, we design One For All (OFA), a scalable system that provides a
unified interface to interact with multiple CAs. Additionally, we introduce
MARS: Multi Agent Response Selection, a new encoder model for question
response pairing that jointly encodes user question and agent response pairs.
We demonstrate that OFA is able to automatically and accurately integrate an
ensemble of commercially available CAs spanning disparate domains.
Specifically, using the MARS encoder we achieve the highest accuracy on our
BBAI task, outperforming strong baselines.

##  1 Introduction

Influenced by the popularity of intelligent conversational agents (CAs), such
as Apple Siri and Amazon Alexa, the conversational AI market is growing at an
increasingly rapid pace and is projected to reach a valuation of US $13.9
billion by 2025 Market and Markets (2020). These CAs have already begun to
show great promise when deployed in domain-specific areas such as driver
assistance Lin et al. (2018), home automation Luria et al. (2017), and food
ordering Frangoul (2018) with platforms such as Pandora and Facebook today
hosting more than 300,000 of these agents Chaves and Gerosa (2018); Nealon
(2018).

Figure 1: An example interaction using One For All which integrates multiple
production black-box agents into a unified experience.

Most CAs are designed to be specialized in a single or set of specific
domains. As such, users are required to interact with multiple agents in order
to complete their tasks and answer their queries as shown in figure 1. For
example, a user may use an agent such as Amazon Alexa for online shopping but
engage with Google Assistant for daily news updates. Additionally, a given
agent may be more proficient at a specific domain over another i.e A finance
CA is better suited to answer finance questions. As a result, users are taxed
with the burden of learning and adopting multiple agents leading to an
increase in the cognitive load of interacting with agents, further
discouraging the proliferation of their usage Dubiel et al. (2020); Novick et
al. (2018); Saltsman et al. (2019). This is escalated further as the number of
conversational agents deployed into the market continues to increase.
Therefore, the need arises for unifying multiple independent CAs through one
conversational interface. This need has manifested in the commercial
conversational AI industry with initiatives such as the Amazon Voice
Interoperability Initiative Amazon (2019) which aims to create voice-enabled
products that contain multiple, distinct, interoperable intelligent assistants
on a single device. However, this interaction is still manual, requiring the
user to orchestrate which agent is initiated. In addition, while it is
possible to have distinct agents in a single device, users prefer interacting
with a single agent over multiple Chaves and Gerosa (2018).

Prior work has explored in part combining the strengths of multiple agents in
one system but they rely on direct access to the design and implementation
details of the to-be-integrated agents. Subramaniam et al. (2018) and Cercas
Curry et al. (2018) direct incoming user questions to a specific agent based
on the candidate agents’ internal knowledge graph and NLU architectures,
respectively. However, in practice, the majority of the publicly available CAs
are "black boxes" where their inner-workings contain highly-protected IP that
is not accessible to the public. Additionally, Cercas Curry et al. (2018)
facilitates their bot selection with a manual heuristic preference order that
requires intimate knowledge of the agents to construct, and additional effort
to maintain, thus not scaling well for the adaption of existing agents and
introduction of new agents. Therefore, the task of integrating multiple
production black-box CAs with a unified interface remains an open problem.

In order to explore this problem, we introduce the task BBAI: Black-Box Agent
Integration that focuses on integrating multiple black-boxes CAs. We propose
two techniques to tackle black-box multi-agent integration: (1) Question agent
pairing and (2) Question response pairing. Intuitively, these two approaches
can be viewed as a query-to-agent classification problem in contrast to that
of a response selection problem. This formulation allows us to facilitate
multi-agent integration whilst operating within the black-box constraints of
the agents. Using these techniques we develop _One For All_ , a novel
conversational system that accurately and automatically unifies a set of
black-box CAs spanning disparate domains. Additionally, we introduce MARS:
Multi Agent Response Selection, a new encoder model for question response
pairing that jointly encodes user question and agent response pairs. We
evaluate these techniques on a suite of 19 publicly available agents
consisting of Amazon Alexa111https://developer.amazon.com/en-US/alexa, Google
Assistant222https://assistant.google.com/, SoundHound
Houndify333https://www.houndify.com/, Ford Adasa Lin et al. (2018) and many
more.

Specifically, this paper makes the following contributions:

  * •

Formulation of the BBAI task that focuses on the challenge of integrating
disparate black-box conversational agents into one experience. We construct a
new dataset for this task, comprising of examples from a suite of 19
commercially deployed conversational agents. We publish our model and
datasets. 444https://github.com/ChrisIsKing/black-box-multi-agent-integation

  * •

We design _One For All_ , a novel conversational system that accurately and
automatically unifies a set of black-box CAs and introduce the MARS encoder
model that outperforms strong state-of-art classification and ranking model
baselines on our BBAI task.

  * •

We conduct a thorough evaluation of various agent integration approaches
showing that our MARS encoder outperforms strong baselines. We show that by
facilitating the integration of multiple agents we can alleviate the need for
users to adopt multiple agents whilst facilitating the improvement and growth
of agents over time.

##  2 BBAI: Black-Box Agent Integration Task Formulation

Building a unified interface for production agents spanning different domains
presents several key challenges. First, most commercially available CAs are
black-boxes, providing little to no information on their inner workings. Any
approach for agent integration must operate without relying on the internals
of any given agent. Second, these conversational agents are constantly
improved upon and expanded with new capabilities. The agent integration
approaches need to be flexible and adaptive to these changes with relative
ease. Given these constraints we assume the existence of the following
information sources for the agent integration task:

  1. 1.

User query/utterance: The question that the user asks the agent.

  2. 2.

Agent skill representation: A textual representation that denotes what each
agent is capable of. This can be in the form of example queries or a
description of that agent.

  3. 3.

Agent response: Each agent’s response to the query asked.

Using this information we formulate the task of agent integration as given a
query $Q$, a set of agents $A=\\{a_{1},a_{2},\ldots,a_{n}\\}$ and a set of
agent responses $R=\\{r_{1},r_{2},...,r_{n}\\}$ to query $Q$, determine the
question-agent-response pair $(Q,A_{i},R_{i})$ that resolves the query $Q$.
Further, given the information available, we can taxonomize our approach into
two techniques: (1) Question agent pairing where we preemptively select the
agent for the query and (2) Question response pairing where we evaluate the
set of returned responses as depicted in Figure 2.

Figure 2: Overview of our proposed black-box agent integration techniques. In
QA Pairing, the goal is to select the correct agent using information about
the agent’s capabilities. In QR Pairing, the goal is to select the correct
agent response.

###  2.1 Question Agent Pairing

As shown in Figure 2, the goal of question agent pairing is, given a query $Q$
and a set of agents $A=\\{a_{1},a_{2},\ldots,a_{n}\\}$, determine the
question-agent pair $(Q,A_{i})$ that resolves the query $Q$. At its core, this
can be viewed as a classification problem where the model learns the
respective capabilities of each independent agent in order to predict which
agent to use for a given question.

###  2.2 Question Response Pairing

As shown in Figure 2, the goal of question response pairing is, given a query
$Q$ and a set of agent responses $R=\\{r_{1},r_{2},...,r_{n}\\}$, determine
the question-response pair $(Q,R_{i})$ such that $R_{i}$ resolves the query
$Q$.

##  3 The One For All System

In this section, we present the design of One For All (OFA), a scalable system
that integrates multiple black-box CAs with a unified interface. We explain
the various approaches implemented in One For All, detailing their inputs,
outputs and training methodology.

Figure 3: The transformer-based classification models in the OFA system. The
models are trained on question agent pairs and tasked to predict an agent to
route the given query to. Figure 4: Overview of OFA approaches. (a) Bi-Encoder
which is used for both QA and QR pairing encodes the question and candidate
response/description separately and computes a ranking score via a dot product
calculation. (b) Our MARS encoder jointly encodes the question and response
into a single transformer and performs self-attention between the question and
candidate response. To score a response we reduce the candidate embedding from
a vector to a scalar score between 0…1 Humeau et al. (2020).

###  3.1 Question Agent Pairing

In order to predict the best agent for a given query, knowledge of each
agent’s individual skill-set is required. However, as described in the task
formulation in Section 2, the internal details of the agents are unavailable.
Everyday users of these agents have no insight into the internal specifics of
these agents. However, they are able to use these agents to accomplish tasks
by building a mental model of each agents’ respective capabilities through
usage over time. We draw inspiration from this to determine the information we
can use to represent an agent’s skills without access to its internals.

####  3.1.1 Agent Skills Representation

Following the learning patterns described above, we model an agent’s skill-set
in two distinct ways:

(1) Query examples: Similar to building knowledge overtime via agent
interaction, an agents’ query examples allows the model to learn what type of
queries each agent is capable of resolving. For example, questions such as
“Where is the nearest gas station?" and "Direct me to Starbucks please" will
be amongst the query examples for a “Directions" agent.

(2) Agent descriptions:. These are textual summaries of an agent’s
capabilities. For example, a bank releases a new CA for its customers to use
instead of having to visit the bank. Accompanied with this agent will be a
semi-formal description of what this agent is capable of doing. This
information is often publicly available in the agent’s marketing materials.

Using these query examples and agent descriptions, we explore approaches for
determining the agent best to resolve a given query. We describe in more
detail the dataset collection process in Section 4.

##### Question agent pairing using query examples

QA pairing using query examples seeks to explore how best we can facilitate
agent orchestration in a data constrained environment where only a few
examples of the questions the agents can answer are present. This is similar
to the use of text examples for the training of an intent classifier but at
the agent level instead. Therefore, we treat this as a multi-label
classification problem where a given query $Q$ is mapped to a set of agents
$A$. e.g Q: _‘locate me some good places in Kentucky that serve sushi‘_ maps
to the set of agents $A$: [“Alexa”, "Google"] indicating that this query can
be correctly answered by the agents Alexa and Google. Specifically, as shown
in Figure 3, we build a multi-label classifier on top of state-of-the-art
transformer models, BERT Devlin et al. (2019), RoBerta Liu et al. (2019) and
Electra Clark et al. (2020) to predict an agent $A$ given a query $Q$.

##### Question agent pairing using agent descriptions

While query examples are useful for understanding the capabilities of a given
agent, they may not be readily available. When a new agent is introduced,
users are unsure of the exact questions this agent can answer but they would
typically have access to an explanation of its capabilities. As an
alternative, we explore the use of such a description of the agents. For this
task, we assume a textual description of an agent’s capabilities, e.g. "Our
productivity bot helps you stay productive and organized. From sleep timers
and alarms to reminders, calendar management, and email ….".

In order to map a given query $Q$ to an agent $A$ described by description
$D_{i}$, we treat this as a semantic similarity task. The intuition behind
this is that for a given query $Q$ the agent that is capable of answering a
given question is likely to feature an agent description semantically similar
to the question. We explore a suite of pre-trained and fine-tuned language
models focusing on ranking the relevance of given description $D_{i}$ to a
query $Q$. Additionally, given the length of descriptions and the range of
capabilities that may be described within a single description, we split the
full description at the sentence level and use each sentence to represent a
single skill $S_{i}$ belonging to agent $A$. With this variation, the
question-description similarity score is calculated as the
$\max_{i}{SemSim(Q,S_{i})}$.

For our BBAI task we consider the following state-of-art semantic retrieval-
based approaches whose utility map well to our problem domain:

##### BM25

This classic method measures keyword similarity and uses it to estimate the
relevance of documents to a given search query Robertson and Zaragoza (2009).
We encode the collection of agent descriptions and return the agent whose
description is most relevant to the given query.

##### Universal Sentence Encoder

Cer et al. (2018) A sentence encoding model for encoding sentences into high
dimensional vectors. We use the transformer
model555https://tfhub.dev/google/universal-sentence-encoder/4 for our
experiment. As shown in part (a) of Figure 4, we encode the user query and the
agent description and compute the dot product as a ranking score.

##### Roberta + STS

Reimers and Gurevych (2019) We fine-tune Roberta-base on the STS benchmark
dataset and use this model to encode our agent descriptions and user query. We
compute the cosine similarity between the two vectors to compute a ranking
score for each description as shown in Figure 4.

###  3.2 Question Response Pairing

Contrary to question agent pairing which selects the agent beforehand,
question response pairing assumes that we provide each agent in the ensemble
the opportunity to respond to the query $Q$ and focus on selecting the best
response from the set of returned responses. As such, we treat this as a
response ranking problem of determining which question-response pair
$(Q,R_{i})$ best answers the query $Q$. Prior work has shown strong
performance on sentence pairing tasks such as this through the use of sentence
encoders and language model fine-tuning Henderson et al. (2019); Humeau et al.
(2020); Reimers and Gurevych (2019). We explore the use of these architectures
in the domain of response selection with the goal of learning representations
for correct question answering from diverse conversational agents.

##### BM25

Similar to our use of BM25 for question agent pairing we use it to rank each
of our question response pairs.

##### USE and USE QA

Yang et al. (2019) We apply the USE model from our agent pairing task to rank
agent responses. In addition, we consider USE QA, an extended version of the
USE architecture specifically designed for question-answer retrieval
applications. We use the Bi-Encoder architecture as shown in Figure 4 (a).

##### Roberta + STS

We fine-tune Roberta-base on the STS benchmark dataset and use it to encode
our question response pairs using the bi-encoder architecture in figure 4.

##### MARS encoder

Pre-existing sentence pairing scoring models are tuned to score sentence pairs
deemed semantically similar. However, in the case of conversational systems,
an agent’s response can be semantically similar but still incorrect. e.g Q:
"What is the weather in Santa Clara today?", R: "Weather information is
currently unavailable". These two sentences are semantically similar but the
response does not resolve the query. In the MARS encoder, we focus on learning
representation beyond similarity by also incorporating the correctness of
agent responses. Using the cross-encoder architecture Humeau et al. (2020);
Reimers and Gurevych (2019) shown in part (b) of Figure 4, we train a question
response pair scoring model for the task of ranking responses to a given query
$Q$ generated by conversational agents. We concatenate both the input question
and response performing full self-attention on the entire input sequence. By
passing both the question and agent response through a single transformer, the
agent response is able to attend to the user query and produce a more input
sensitive representation of the question response embedding. Using the
generated question response embedding vector we then convert it to a scalar
score $S(Q,R_{i})$ between 0..1 via a linear layer. Our training objective is
to minimize the Cross-Entropy loss between the correct agent responses and the
negative agent responses to the query $Q$.

Question | Agent Response
---|---
Alexa | Google | Houndify | Adasa
At how many miles will I run out of gas |  "here’s something I found on the web according to freakonomics.com previously when cars got 8 to 12 miles …." |  "on the website post Dash gazette.com they say some popular car models can make it between 30 and 50 miles ….", |  Didn’t get that! |  "With your current fuel economy of 28 MPG, you should be able to cover about 532 miles with the fuel you have."
Is it gonna be warm Friday in Alhambra? |  "here’s something I found on the web according to Wikipedia. Org Cobra is one of the 100 selected cities in India which will be developed …." |  "No, it won’t be hot Friday in Alhambra, California. Expect a high of 21 and a low of 6.", |  "There will be a high of seventy degrees in Alhambra on Friday November twenty-seventh." |  "Out of scope!"

Table 1: Sample question agent responses from the One For All dataset.
Responses highlighted in green represent agent responses voted as correct by
crowd workers.

##  4 Dataset Construction

For the task of BBAI, we construct a new dataset focusing on making it
representative of real-world conversational agents at scale and covering a
broad range of domains.

Using Amazon Mechanical Turk and scenario/paraphrasing-based prompts Kang et
al. (2018); Larson et al. (2019), we crowdsourced utterances across a range of
agent skills/capabilities. These skills were extracted from public information
sources describing each of the agents, in addition to observing their
capabilities. Our dataset is comprised of utterances across 37 broad domain
categories. These include domains such as Weather, Flight Information,
Directions, Automobile, etc. Crowd workers were paid $0.12 for 5 utterances.
These submitted utterances were then vetted by hand to ensure quality. Using
the curated utterances, we then generated question responses by querying each
agent to gather its response to the utterance.

In order to generate ground truth samples on which of the question-response
pairs $(Q,R_{i})$ correctly resolves the query $Q$ we launched a crowdsourcing
task asking workers to indicate the candidate responses that best answer the
question shown. Five workers were assigned to each response selection task and
majority voting (>2) was used to label the gold responses. As such for each
query $Q$ and the set of responses $R$ we were able to gather the necessary
question-agent pairs $(Q,A_{i})$ and question-response pairs $(Q,R_{i})$
needed to evaluate our approaches.

| Agent Breakdown
---|---
| Method | Accuracy (n=4) | Alexa | Google | Houndify | Adasa
Question Agent Pairing (QA Labels) | Bert | 68.31 | 37.98 | 40.93 | 18.49 | 2.6
Electra | 67.86 | 35.28 | 42.01 | 20.11 | 2.6
Roberta | 69.03 | 34.92 | 41.56 | 20.65 | 2.87
Question Agent Pairing (Descriptions) | BM25 | 27.91 | 13.91 | 10.95 | 17.33 | 57.81
USE | 47.84 | 13.20 | 28.82 | 52.42 | 5.56
Roberta+STS | 39.40 | 18.94 | 22.35 | 51.35 | 7.36
Response Selection | BM25 | 51.07 | 28.64 | 24.69 | 14.81 | 31.86
USE | 72.89 | 34.20 | 27.65 | 22.98 | 15.17
USE QA | 75.49 | 41.65 | 36.45 | 17.95 | 3.95
Roberta+STS | 69.83 | 18.94 | 22.35 | 51.35 | 7.36
MARS | 79.70 | 37.34 | 43.9 | 15.71 | 3.05
Individual Agents | Alexa | 49.37 | - | - | - | -
Google | 51.79 | - | - | - | -
Houndify | 34.82 | - | - | - | -
Adasa | 4.12 | - | - | - | -

Table 2: Performance breakdown of QA and QR approaches on our BBAI task when
using our 4 largest agents Alexa, Google, Houndify and Adasa. Note: n = number
of agents.

##### Agent Descriptions

We gather our agent descriptions by scraping the contents of each of the
agent’s public product pages and their built-in feature documentation web
pages. We then manually clean, reformat and merge this data into a single
document per agent. For our experiment, we focus only on extracting
descriptions related to the built-in features of our agents.

Overall our dataset contains 5550 utterances with 19 question-response pairs
per question (one from each of the 19 agents), 105,450 in total. The
utterances are split into 3700 utterances (100 per domain) for the training
set and 1850 (50 per domain) for the test set. The train and test sets
respectively contain 2399 and 1186 utterances with at least one positive
question-response pair. In the remaining examples, none of the agents were
able to achieve annotator agreement (>= 3). A sample dataset example is shown
in table 1 with responses from 4 of the 19 agents.

Method | Accuracy (n=19) | Agents
---|---|---
Question Agent Pairing (QA Labels) | Bert | 59.10 |  Alexa, Google Houndify, Adasa Recipe agent Dictionary agent Task Manager Hotel agent, Stock agent Math agent, Sports agent Wikipedia agent Mobile Account agent Banking agent Coffee shop agent Event Search agent Jokes agent Reminders agent Covid-19 agent
Electra | 52.86
Roberta | 61.88
Question Agent Pairing (Descriptions) | BM25 | 23.69
USE | 43.59
Roberta+STS | 36.67
Response Selection | BM25 | 59.94
USE | 64.42
USE QA | 71.66
Roberta+STS | 56.82
MARS | 83.55
Individual Agents | Alexa | 44.09
Google | 48.06
Houndify | 32.04
Adasa | 3.45

Table 3: Performance breakdown of QA and QR approaches on our BBAI task on all
19 commercial agents we show that the MARS encoder is able to scale and
leverage the capabilities of new agents added to the ensemble without
diminishing performance compared to other approaches.

##  5 Results and Discussion

In this section we present and analyze the results of our experiments,
detailing our insights and discussing the implications of each of our
techniques.

##### Evaluation task:

Similar to standard information retrieval evaluation measures, we denote
accuracy as the metric _precision@1_ and use it to evaluate both our question
agent and question response pairing approaches. For question agent pairing
this metric denotes: Given a set of $N$ agents to the given query, whether the
agent selected ultimately resolves the query successfully. For question
response pairing it denotes: Given a set of $N$ responses to the given query,
whether the top-scoring response resolves the query successfully. For this
evaluation, we test on examples with at least one valid agent response.

###  5.1 Question agent pairing

The results are summarized in tables 2 and 3. We find that for the QA pairing
Roberta yields the best result with an accuracy of 69% in selecting the
correct agent and 61.8% when scaled to 19 agents. Similarly, we see that we
can achieve fair performance in extreme data-scarce environments when using
simple agent descriptions compared to that of query agent examples, with USE
achieving 47.8% accuracy. Using agent descriptions offers greater flexibility
in facilitating the improvement of agents over time compared to query examples
since it only requires an update to the agent description. However, it still
falls short when compared to using a single agent like Google or Alexa. Also,
while consistent in learning to recognize the domain a given agent may be
performant in, QA approaches fall short in a few cases:

(1) Agent overlap \- This is when a given domains’ coverage is split between
various agents. e.g The model learns that both Alexa & Google have proficiency
in handling some weather queries but it remains unclear about which one is
best suited for the current query at hand.

(2) Query variation \- While an agent’s examples or descriptions may allude to
proficiency in a given domain, it may still fail when asked certain query
variations. e.g Figure 1 shows a case where Alexa is capable of handling
weather queries but fails when a condition like humidity is asked for. Another
example is when a similar question is asked in a different or more complex
way. Both Houndify & Alexa are known to be proficient at answering age-related
questions but for questions like "How old I will be on September 28, 1995, if
I was born on March 29, 1967?", Alexa is unable to answer as opposed to
Houndify.

These cases are further highlighted when inspecting QA pairing performance at
the domain level in table 4. We find that the QA approaches struggle with
domains such as _"travel suggestion"_ and _"Directions"_ which are heavily
split in coverage and more diverse in their variation.

Evaluation Performance per Domain (n=19)
---
Domain | MARS (QR) | USE (QA) | Roberta (QA)
Weather | 0.88 | 0.45 | 0.67
Directions | 0.78 | 0.29 | 0.44
Auto | 1.00 | 0.79 | 0.82
Restaurant Suggestion | 0.79 | 0.5 | 0.68
Travel Suggestion | 0.97 | 0.33 | 0.57
Time | 0.81 | 0.54 | 0.76
Flight Info | 0.83 | 0.61 | 0.7
Date | 0.82 | 0.47 | 0.56

Table 4: Further breakdown of the best-performing approaches per technique on
a subset of 8 out of the 37 domains. We find that our MARS encoder generalizes
well across the various agent domains.

###  5.2 Question response pairing

In overall performance we find that our MARS encoder outperforms strong
baselines, achieving 83.55% accuracy on the BBAI task. We note that our MARS
encoder outperforms the best single performing agent (Google Assistant) by
32%. This shows the utility and power of OFA in not only alleviating the need
for users to learn and adopt multiple agents but also validating that multiple
agents working collectively can achieve significantly more than single agents
working in isolation.

When inspecting the performance of MARS at the domain level we see in Table 4
that it is able to maintain its high performance across the varying domains
unlike the QA approaches. This advantage comes from the ability to select an
agent at the response level allowing the system to catch cases in which an
agent once deemed proficient fails or another agent improves.

###  5.3 Agent pairing vs Response pairing

We now describe the trade-offs between agent pairing and response pairing.
Question response pairing greatly outperforms agent pairing in terms of
accuracy, given that it is privy to the final responses from each of the
agents. However, in practice, this comes with additional networking, compute,
and latency costs from having to send the query to each of the agents and
await their response. Given that the querying of agents is done in parallel,
the latency cost is equal to that of the slowest agent. Question response
pairing also better supports agent adaptation. With response pairing, a system
can seamlessly add or remove an agent without diminishing the experience as
shown by MARS in table 3. In addition, as conversational agents are upgraded
to offer a more diverse feature-set such as new domain support or improved
responses, they can instantly be integrated into a response pairing approach.

###  5.4 Scalability

We evaluate our approaches on a suite of 19 commercially deployed agents
spanning 37 broad domain categories. As shown in table 2 we examine
performance when using the 4 largest agents in terms of domain support and
popularity (Alexa, Google Assistant, Houndify and Ford Adasa) showing
improvement upon single-agent use in both QA and QR approaches. When scaled up
to 19 agents, MARS encoder improves even further by leveraging the new
capabilities of the additional agents and is the only approach that does not
decrease in performance as the number of agents and domains scale. This
improvement is due to the input sensitive representations that the MARS
encoder is able to learn by encoding both the question and response in a
single transformer.

Cross-encoding vs Bi-encoding For pairwise sentence scoring tasks such as
response selection which compare question response pairs, it is important to
be mindful of the trade-offs between cross encoder based models such as MARS
in figure 4 (b) and bi-encoder models such as USE in 4 (a). Cross-encoders
perform full self-attention over the pairwise input of the question and
response, thus, producing an encoding representative of the combined input.
This typically leads to much more performative models, especially in pairwise
scoring tasks such as ours. However, given that this encoding isn’t
independent of the question for each question response pair, it is necessary
to produce an encoding for each question label pair. Bi-encoders on the other
hand perform self-attention over the question response pairs separately, map
them to a dense vector space, and score them using an appropriate distance
metric. With this separation, bi-encoders are able to index the question and
compare these representations for each response resulting in faster prediction
times when the numbers of candidate responses to a given question scales.
Given the nature of our BBAI task which focuses on the scoring of responses to
a singular question as opposed to a clustering task which requires an encoding
for every pairwise combination across a set of sentences, cross-encoder based
architectures remain a viable option even at the production scale for our use
case.

##  6 Related Work

Ensemble approaches to solving complex tasks in the context of NLP are widely
used Deng and Platt (2014); Araque et al. (2017). In dialogue systems, recent
attempts at ensemble approaches and multi-agent architectures include Cercas
Curry et al. (2018) and Subramaniam et al. (2018). AlanaV2 Cercas Curry et al.
(2018) demonstrated an ensemble architecture of multiple bots using a
combination of rule-based machine learning systems built to support topic-
based conversations across domains. It was built to be an open domain bot
supporting topic-based conversations. Specifically, AlanaV2’s architecture
utilizes a variety of ontologies and NLU pipelines that draw information from
a variety of web sources such as Reddit. However, its agent selection approach
is guided by a simple priority bot list. Subramaniam et al. Subramaniam et al.
(2018) describe their conversational framework that employs an Orchestrator
Bot to understand the user query and direct them to a domain-specific bot that
handles subsequent dialogue. In our work, we expand up the multi-agent goal by
focusing on the integration of black-box conversational agents at scale.

###  6.1 Response Selection

This is the task of selecting the most appropriate response given context from
a pool of candidates. It is a central component of information retrieval
applications and has become a focal point in the evaluation of dialogue
systems. Sato et al. (2020); Henderson et al. (2019); Wang et al. (2020).
Prior work has shown strong performance on sentence pairing tasks through the
use of sentence encoders and language model fine-tuning Henderson et al.
(2019); Humeau et al. (2020); Reimers and Gurevych (2019). In our work, we
explore the task of response selection using it as one of the bases for
integrating black-box conversation agents.

##  7 Conclusion

The rapid proliferation of conversational agents calls for a unified approach
to interacting with multiple CAs. The key challenge of building such an
interface lie in that most commercial CAs are black-boxes with hidden
internals. This paper introduces BBAI a new task of agent integration that
focuses on unifying black-boxes CAs across varying domains. We explore two
task techniques, question agent pairing and question response pairing and
present One For All, a scalable system that unifies multiple black-box CAs
with a centralized user interface. Using a combination of commercially
available conversational agents, we evaluate a variety of approaches to multi-
agent integration through One For All. Our MARS encoder achieves 83.5%
accuracy on BBAI and outperforms the best single agent configuration by over
32%. These results demonstrate the power of One For All which can leverage
state-of-the-art NLU approaches to enable multiple agents to collectively
achieve more than any single conversational agent in isolation eliminating the
need for users to learn and adopt multiple agents.

This work opens up a wide range of potential future work involving the design
of systems geared towards facilitating more advanced multi-agent interaction.
We foresee a system with even greater response selection performance as the
NLP community continues to produce more state-of-the-art language models with
even greater contextual knowledge of the world. Extensions of this work can
include examining not only the integration of agents but the interoperability
by facilitating the passing of shared conversation knowledge across agents
especially in multi-turn conversational scenarios across multiple agents.

## Acknowledgements

We thank our anonymous reviewers for their feedback and suggestions. We also
thank Yi-Chun Chen who assisted in designing the diagrams and figures shown in
this work. This work was sponsored by Ford Motor Company.

## References

  * Amazon (2019) Amazon. 2019.  Amazon and leading technology companies announce the voice interoperability initiative.
  * Araque et al. (2017) Oscar Araque, Ignacio Corcuera-Platas, J. Fernando Sánchez-Rada, and Carlos A. Iglesias. 2017.  Enhancing deep learning sentiment analysis with ensemble techniques in social applications.  _Expert Systems with Applications_ , 77:236 – 246.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder for English.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 169–174, Brussels, Belgium. Association for Computational Linguistics.
  * Cercas Curry et al. (2018) Amanda Cercas Curry, Ioannis Papaioannou, Alessandro Suglia, Shubham Agarwal, Igor Shalyminov, Xu Xinnuo, Ondrej Dusek, Arash Eshghi, Ioannis Konstas, Verena Rieser, and Oliver Lemon. 2018.  Alana v2: Entertaining and informative open-domain social dialogue using ontologies and entity linking.  In _1st Proceedings of Alexa Prize (Alexa Prize 2018)_.
  * Chaves and Gerosa (2018) Ana Paula Chaves and Marco Aurelio Gerosa. 2018.  Single or multiple conversational agents?: An interactional coherence comparison.  In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ , CHI ’18, pages 191:1–191:13, New York, NY, USA. ACM.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.
  * Deng and Platt (2014) Li Deng and John Platt. 2014.  Ensemble deep learning for speech recognition.  In _Proc. Interspeech_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dubiel et al. (2020) Mateusz Dubiel, Martin Halvey, Leif Azzopardi, and Sylvain Daronnat. 2020.  Interactive evaluation of conversational agents: Reflections on the impact of search task design.  In _Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval_ , ICTIR ’20, page 85–88, New York, NY, USA. Association for Computing Machinery.
  * Frangoul (2018) Anmar Frangoul. 2018.  Here’s how robots are transforming takeout deliveries.
  * Henderson et al. (2019) Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2019.  Convert: Efficient and accurate conversational representations from transformers.
  * Humeau et al. (2020) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020.  Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring.  In _International Conference on Learning Representations_.
  * Kang et al. (2018) Yiping Kang, Yunqi Zhang, Jonathan K. Kummerfeld, Parker Hill, Johann Hauswald, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2018.  Data collection for dialogue system: A startup perspective.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 1311–1316. Association for Computational Linguistics.
  * Lin et al. (2018) Shih-Chieh Lin, Chang-Hong Hsu, Walter Talamonti, Yunqi Zhang, Steve Oney, Jason Mars, and Lingjia Tang. 2018.  Adasa: A conversational in-vehicle digital assistant for advanced driver assistance features.  In _Proceedings of the 31st ACM Symposium on User Interface Software and Technology (UIST 2018)_ , UIST-31, Berlin, Germany. ACM.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Luria et al. (2017) Michal Luria, Guy Hoffman, and Oren Zuckerman. 2017.  Comparing social robot, screen and voice interfaces for smart-home control.  In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ , CHI ’17, pages 580–628, New York, NY, USA. ACM.
  * Market and Markets (2020) Market and Markets. 2020.  Conversational ai market.
  * Nealon (2018) Gary Nealon. 2018.  Using facebook messenger and chatbots to grow your audience.
  * Novick et al. (2018) David Novick, Laura J. Hinojos, Aaron E. Rodriguez, Adriana Camacho, and Mahdokht Afravi. 2018.  Conversational interaction with multiple agents initiated via proxemics and gaze.  In _Proceedings of the 6th International Conference on Human-Agent Interaction_ , HAI ’18, page 356–358, New York, NY, USA. Association for Computing Machinery.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Saltsman et al. (2019) Thomas L. Saltsman, Mark D. Seery, Cheryl L. Kondrak, Veronica M. Lamarche, and Lindsey Streamer. 2019.  Too many fish in the sea: A motivational examination of the choice overload experience.  _Biological Psychology_ , 145:17–30.
  * Sato et al. (2020) Shiki Sato, Reina Akama, Hiroki Ouchi, Jun Suzuki, and Kentaro Inui. 2020.  Evaluating dialogue generation systems via response selection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 593–599, Online. Association for Computational Linguistics.
  * Subramaniam et al. (2018) Sethuramalingam Subramaniam, Pooja Aggarwal, Gargi B. Dasgupta, and Amit Paradkar. 2018.  Cobots - a cognitive multi-bot conversational framework for technical support.  In _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , AAMAS ’18, pages 597–604, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
  * Wang et al. (2020) Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2020.  Response selection for multi-party conversations with dynamic topic tracking.  _CoRR_ , abs/2010.07785.
  * Yang et al. (2019) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Multilingual universal sentence encoder for semantic retrieval.