# Polarization interferometric prism: a versatile tool for generation of
vector fields, measurement of topological charges and implementation of a
spin-orbit Controlled-Not gate

Zhi-Cheng Ren  Zi-Mo Cheng  Xi-Lin Wang  Author to whom correspondence should
be addressed<EMAIL_ADDRESS>Jianping Ding  Hui-Tian Wang
<EMAIL_ADDRESS>National Laboratory of Solid State Microstructures and
School of Physics, Nanjing University, Nanjing 210093, China  Collaborative
Innovation Center of Advanced Microstructures, Nanjing 210093, China

###### Abstract

Optical vortex and vector field are two important types of structured optical
fields. Due to their wide applications and unique features in many scientific
realms, the generation, manipulation and measurement of such fields have
attracted significant interest and become very important topics. However, most
ways to generate vector fields have a trade-off among flexibility, efficiency,
stability, and simplicity. Meanwhile, an easy and direct way to measure the
topological charges, especially for high order optical vortex, is still a
challenge. Here we design and manufacture a prism: polarization
interferometric prism (PIP) as a single-element interferometer, which can
conveniently convert an optical vortex to vector fields with high efficiency
and be utilized to precisely measure the topological charge (both absolute
value and sign) of an arbitrary optical vortex, even with a high order.
Experimentally we generate a variety of vector fields with global fidelity
ranging from 0.963 to 0.993 and measure the topological charge of an optical
vortex by counting the number of petals uniformly distributed over a ring on
the output intensity patterns. As a versatile tool to generate, manipulate and
detect the spin-orbital state of single photons, PIP can also work in single-
photon regime for quantum information processing. In experiment, the PIP is
utilized as a spin-orbit Controlled-Not gate on the generated 28 two-qubit
states, achieving the state fidelities ranging from 0.966 to 0.995 and
demonstrating the feasibility of the PIP for single photons.

††preprint: AIP/123-QED

Optical vortex has a helical phase of $\exp(jm\varphi)$ and carries an orbital
angular momentum (OAM) of $m\hbar$ per photon,Allen1992  where $\varphi$ is
the azimuthal angle and the integer index $m$ is the topological charge. As
one of the most important structured light,Forbes2019 ; Yuan2019  optical
vortices keep attracting great attention, and their generation and detection
develop fast ranging from optical Forbes2016 ; Wang2007  to acoustic Zhang2019
and radio frequency.Mao2018  Till now, optical vortices have already
contributed in quantum information,Alois2001  optical tweezers,He1995  and
microscopy,Willig2006  etc. Meanwhile, optical vortices promote the emergence
and development of optical vector fields with spatially-variant polarization
state Zhan2009 ; Milione2011 , which are another important type of structured
light. The special polarization distributions of vector fields result in some
peculiar properties in the research of photons themselves and light-matter
interaction, for example, shaper focus,Dorn2003  longitudinally polarized
optical needles,Wang2008  polarization Möbius strips,Bauer2015  and so on.
Vector fields have found a wide variety of applications in
microscopy,Kozawa2018  nonlinear optics,Camacho2016  material processing
Weber2011  and even quantum information.Barreiro2008

Many methods have been developed to generate vector fields, such as
interferometric scheme,Wang2007 ; Forbes2016  intra-cavity,Naidoo2016 ;
JLZhao2017  $q$-plate,Marrucci2006  meta-vector-polarizer, Bomzon2002  and
nanostructure.Beresna2011  Laser cavity, $q$-plate, meta-vector-polarizer and
nanostructure are lack of flexibility. With the help of spatial light
modulator (SLM), interferometric schemes are quite flexible, but are
relatively complicated. Considering the close connection between optical
vortex and vector field, a simple and stable device, which can conveniently
and efficiently convert an optical vortex into vector fields, will be of great
value. On the other hand, most common ways to measure the topological charge
of optical vortex is to use its diffraction or/and interference properties,
such as slit diffraction and interference,Ferreira2011 ; Zhou2014
interference with plane waves,Huang2013  Mach–Zehnder or Michelson
interferometer,Li2016 ; Pushin2020  diffraction by an apertureHickmann2010 ;
CSGuo2009  and grating,JLZhao2018  and cylindrical lens.LXChen2016 ;
Alperin2016 ; BGu2020  These methods provide more choices for measuring the
OAM number. A compact and stable scheme will enrich the tools for the OAM
research. In this letter, we design a single-element interferometer with a
compact and stable scheme, which is quite convenient in generating a variety
of vector fields, measuring the topological charge of optical vortex, and
manipulating the spin-orbital state of single photons.

Vector fields could be expressed as the coherent superposition of two
orthogonally polarized scalar fields with opposite phases. The key is to
produce two opposite vortices in two paths, which can be usually achieved by
optical fields in one path being reflected even times and the other odd
times.Mendoza2019  To this end, we design and develop a single-element
interferometer, named as polarization interferometric prism (PIP) in Fig.
1(a). The PIP is composed of two fused silica prisms (A and B) glued together
in Fig. 1(b). The polarization splitting coating on the glued surface (60 mm
in length) of the prisms A and B divides the input beam into two beams with
orthogonal linear polarizations in the lower left corner and combine them in
the upper right corner. As shown in Fig. 1(b), four angles in the lower left
and upper right corners for the prisms A and B are all designed to be
$45^{\circ}$, and the other three corners (A1, A2 and A3) for the prism A have
the angles ($157.5^{\circ}$, $135^{\circ}$ and $157.5^{\circ}$) and the other
four corners (B1, B2, B3 and B4) for the prism B have the angles
($165^{\circ}$, $150^{\circ}$, $150^{\circ}$ and $165^{\circ}$). The PIP has a
thickness of 5 mm, which offers an aperture of $\sim$5 mm. The horizontally
(vertically) polarized component is transmitted (reflected) on the glued
surface of the PIP, and will undergo 2 (3) total internal reflections at the
surface of the prism A (B) before combination. The two optical paths inside
the PIP are carefully designed to be equal to ensure the perfectly coherent
superposition. Obviously, the PIP is immune to air turbulence or/and table
vibration, so the PIP is a stable single-element interferometer even for poor-
coherence lasers.

Figure 1: Generator of vector fields and topological charge measurement of
optical vortices, with a single-element interferometer (PIP). (a) Photo of a
PIP. (b) Converter of optical vortex to vector field and measuring instrument
of topological charges for optical vortices. HWP, half-wave plate is used to
adjust intensity ratio in two paths inside the PIP. WP, wave plate is used to
select the superposition bases for vector field.

To generate optical vortices, many methods have already proposed, such as
using a spiral phase plate (SPP) or computer-generated holograms.Forbes2019
To verify the feasibility of our device, we use a commercial spatial light
modulator (SLM) to experimentally generate optical vortex $\exp(jm\varphi)$
with arbitrary topological charge $m$. As mentioned above, the two beams
propagating inside the PIP undergo even and odd total internal reflections
respectively, so the two orthogonal linearly polarized beams output from the
PIP will have opposite helical phases $\exp(\pm jm\varphi)$. We use a Soleil-
Babinet compensator composed of two wedge-shape wave plates to precisely
control the relative phase delay between the two beams.

When the fast axis of the half-wave plate (HWP) in front of the PIP is
oriented at $22.5^{\circ}$ (all orientation angles are defined with respect to
the horizontal direction), the two beams in the two paths have the balanced
intensity; when the wave plate (WP) behind PIP is a quarter-wave plate (QWP)
and its fast axis is oriented at $45^{\circ}$, the two orthogonal linearly
polarized beams output from the PIP are converted into the orthogonal
circularly polarized beams. Under these conditions, the generated cylindrical
vector optical field can be written as

$\textbf{E}\propto\exp(-jm\varphi)\hat{\textbf{e}}_{R}+\exp(jm\varphi)\hat{\textbf{e}}_{L},$
(1)

where $\hat{\textbf{e}}_{R}$ ($\hat{\textbf{e}}_{L}$) refers to the right-
handed (left-handed) circularly polarized unit vector. Such a kind of vector
fields is locally linearly polarized and can be represented by a point on the
equator of a higher-order Poincaré sphere.Milione2011

When the input optical vortex has the topological charge of $m=1$, the
generated radially polarized vector field is shown in the first column of Fig.
2(a). To make the polarization distributions more obvious and observable, we
expand the input beam to homogeneous intensity in Fig. 2. We can easily adjust
the phase delay between the two components to be $\pi$ and generate the
azimuthally polarized vector field. When the input vortex fields have higher-
order topological charges of $m=3$ and $5$, the generated high-order
cylindrical vector fields are shown in the third and fourth columns of Fig.
2(a). We can use a horizontal polarizer to distinguish the vector fields.
Behind a polarizer, the uniform intensities become into the fan-shaped
patterns with $2|m|$ extinction lines where the local linear polarizations are
orthogonal to the polarizing direction of the polarizer in the second row of
Fig. 2(a).

Our device can also generate hybrid vector fields and arbitrary higher-order
Poincaré sphere states,Milione2011  which can be generated by the setup in
Fig. 1. To fully characterize the polarization state distributions, Stokes
parameters of $S_{1}$, $S_{2}$ and $S_{3}$ are also measured. As shown in the
first row of Fig. 2(b), the hybrid vector field with azimuthal-dependent
Stokes parameters of $S_{1}$, $S_{2}$ and $S_{3}$ is created by orthogonal
elliptically polarized bases of
$\\{\sin\alpha\hat{\textbf{e}}_{H}\\!+\\!\cos\alpha
e^{j\beta}\hat{\textbf{e}}_{V},\cos\alpha\hat{\textbf{e}}_{H}\\!-\\!\sin\alpha
e^{j\beta}\hat{\textbf{e}}_{V}\\}$ with $\alpha\\!=\\!52.2^{\circ}$ and
$\beta=-0.35\pi$, where $\hat{\textbf{e}}_{H}$ ($\hat{\textbf{e}}_{V}$) is the
horizontally (vertically) unit vector. This hybrid vector field can be
generated when the WP behind PIP is a rotatable HWP.

The higher-order Poincaré sphere vector fields can also be generated as

$\textbf{E}\propto
a\exp(-jm\varphi)\hat{\textbf{e}}_{R}+b\exp(+jm\varphi)\hat{\textbf{e}}_{L},$
(2)

where $a$ and $b$ indicate the relative intensities with $a^{2}+b^{2}\equiv
1$, and are easily adjusted by rotating the HWP in front of PIP. When the HWP
is oriented at $43.56^{\circ}$ (i.e. $a=0.358$ and $b=0.934$), the
horizontally and vertically polarized components have an intensity ratio of
$13:87$ for the input beam. As shown in the second row of Fig. 2(b), the
generated higher-order Poincaré sphere vector field with $m=1$ has homogeneous
nonzero $S_{3}$ but azimuthal-dependent $S_{1}$ and $S_{2}$.

Figure 2: (a) Generated four vector fields with local linear polarization; (b)
generated hybrid and higher-order Poincaré sphere vector fields, and
corresponding Stokes parameters.

Local Stokes parameters in Fig. 2(b) are difficult to directly characterize
the global quality of the generated vector fields. It will be useful to
evaluate the global quality of the generated vector field by a single
parameter. Here we refer to the concept of fidelity,Nielsen2010  which can
characterize the global similarity or difference between the experimentally
generated state and the target state. The fidelity is defined as
$F(\rho_{t},\rho_{e})\equiv\mathrm{Tr}(\rho_{t},\rho_{e})$, where
$\rho_{t}=|E\rangle\langle E|$ ($\rho_{e}=|E_{e}\rangle\langle E_{e}|$)
represents the density matrix of the target vector field state (the
experimentally generated vector field state). We can rewrite the density
matrix with the Stokes parameters as follows

$\rho=\frac{1}{2}\left[\begin{array}[]{cc}1+S_{1}&S_{2}+jS_{3}\\\
S_{2}-jS_{3}&1-S_{1}\\\ \end{array}\right].$ (3)

With the Stokes parameters normalized by the total intensity $S_{0}$
(satisfying $S_{1}^{2}+S_{2}^{2}+S_{3}^{2}=1$), the fidelity of the complete
polarized vector field we generated can be rewritten as

$F(\rho_{t},\rho_{e})=(1+S_{1}^{t}S_{1}^{e}+S_{2}^{t}S_{2}^{e}+S_{3}^{t}S_{3}^{e})/2,$
(4)

where $S_{i}^{t}$ and $S_{i}^{e}$ ($i=1,2,3$) refer to the normalized Stokes
parameters of the target state and the experimentally generated vector field
state, respectively. By averaging the local fidelities $F(\rho_{t},\rho_{e})$,
we can obtain the global fidelity as $F_{global}(\rho_{t},\rho_{e})$ to
characterize the quality of the generated vector field. A large number of
vector fields we generated have very high quality, because they have the
global fidelities ranging from 0.963 to 0.993.

Obviously, it can be seen from Fig. 2 that behind a polarizer, the uniform
intensity turns into fan-shaped patterns with $2|m|$ bright petals. This
offers an intuitive way to measure the topological charge of an optical
vortex. By using the same setup in Fig. 1, as examples, Fig. 3 shows the
measured optical vortices with topological charges of $m=+4$, $-16$, $+64$ and
$+65$. One should point out that in order to eliminate the unwanted subsidiary
rings in the focal plane, as shown in Fig. 3, we use the optimal annulus
structure mask for generating the input vortex,Guo2004  which shows the fact
that the optimal annulus width of the input vortex depends on the topological
charge. As shown in the first two columns of Fig. 3, the absolute values of
topological charges are easily determined by counting the bright petals at
either the image plane or the focal plane. In details, the patterns in the
first two columns show 8, 32, 128 and 130 bright petals, respectively.
Correspondingly, we can identify the absolute values of topological charges to
be $|m|=4$, 16, 64 and 65, respectively. However, the sign of topological
charge cannot be directly obtained, due to the same number of petals ($2|m|$)
for the optical vortices with $\pm m$. To solve this problem, an auxiliary
measurement should be carried out: inserting a spiral phase plate (SPP) with a
phase structure of $\exp(jm^{\prime}\varphi)$ into the input port of the
original setup. As an example with $m^{\prime}\\!\\!=\\!+1$, the second
measured patterns are shown in the last two columns of Fig. 3. Comparing the
two measured patterns, the sign of topological charge can be easily
determined. If the number of petals increases by 2, the topological charge is
positive. Otherwise, if it decreases by 2, the topological charge is negative.
By counting the number of petals, it is easy to distinguish $m=+64$ from
$m=+65$, which proves the high accuracy and convenience of our method.

Figure 3: Measurement of topological charges of optical vortices with $m=+4$,
$-16$, $+64$ and $+65$. The first and second columns show results using the
same scheme in Fig. 1, while the third and fourth columns show the results by
inserting a SPP with $m^{\prime}\\!\\!=\\!+1$ into the input port of the
original setup.

The above two applications of PIP, namely generation of optical vector fields
and measurement of topological charges, require to record the whole image of
optical fields by exposing a large number of photons. A question to be asked
is whether this PIP can also work in single-photon regime. Our PIP can
implement a spin-orbit Controlled-NOT (CNOT) gate for single photons, with the
spin or polarization as the control qubit and the OAM as the target qubit. To
verify this functionality, we need to prepare a single-photon 4-dimension
spin-orbit composite state
$a{\ket{H}}\ket{+1}+b{\ket{V}}\ket{+1}+c{\ket{H}}\ket{-1}+d{\ket{V}}\ket{-1}$,
where ${\ket{H}}$ (${\ket{V}}$) denotes horizontal (vertical) polarization and
$\ket{+1}$ ($\ket{-1}$) denotes positive (negative) OAM of $+\hbar$
($-\hbar$). As shown in Fig. 4(a), we generate a heralded single photon, which
is one of a photon pair from spontaneous parametric down conversion (SPDC) by
passing a femtosecond (fs) pulsed laser through a 0.6 mm type-I $\beta$-barium
borate (BBO), and triggered by detecting its twin photon. The heralded single
photon is filtered with a single-mode fiber to obtain fundamental Gaussian
mode (zero OAM mode). A $q$-plate and several WPs are utilized to prepare a
spin-orbit composite state. Then this photon passes through PIP and a spin-
orbit CNOT gate operates on its state. The converted state of two qubits is
carried by a single photon and we measure it in a step-by-step manner, first
polarization measurement then OAM measurementWangXL2018 . One of the most
important functions of a CNOT gate is to transfer two qubit product states to
entangled states and vice versa. Here we convert the 4 Bell states of
$\ket{\Phi^{\pm}}=({\ket{H}}\ket{+1}\pm{\ket{V}}\ket{-1})/\sqrt{2}$ and
$\ket{\Psi^{\pm}}=({\ket{H}}\ket{-1}\pm{\ket{V}}\ket{+1})/\sqrt{2}$ into the 4
product states with very high fidelity and vice versa in Fig. 4(b). Moreover,
we also implement the spin-orbit CNOT gate on all 20 states in 5 groups of
mutually unbiased basis Wiesniak2018  and achieve high fidelity for the
converted state ranging from 0.966 to 0.995, demonstrating the feasibility of
our PIP for single photons.

Figure 4: Experimental setup and results for confirming the feasibility of PIP
for single photons. (a) PIP, as a CNOT gate, operates on a 4-dimension spin-
orbit composite state prepared by a heralded single photon, and the converted
state is firstly measured in polarization basis, then in OAM basis. (b) The
measured fidelities for the converted 4 Bell states and the converted states
from the prepared 4 Bell states.

In summary, we have designed and developed a PIP, which can convert vortex
fields into vector fields with simultaneously high quality, low complexity and
high efficiency. Our single-element PIP enables vector fields more widely
used, especially at high power and high energy. Based on PIP, we also propose
a method to conveniently and precisely measure the topological charge of any
optical vortex. Compared with previous methods, our PIP has the advantages of
compact, high stability, high efficiency, and great convenience. More
importantly, the number of uniformly distributed petals is easier to observe
and count than diffracted stripes or nonuniform spots, especially for large
topological charges. Benefiting from its high stability, the PIP has the
advantage for large-scale applications with dozens of single-photon
interferometers.WangXL2018  Furthermore, our PIP can also work at single-
photon level and can be utilized as a versatile quantum element to generate,
manipulate and detect the spin-orbital state of single photons.

This work was supported by the National Key R&D Program of China
(2017YFA0303800, 2017YFA0303700, 2018YFA0306200); National Natural Science
Foundation of China (11534006, 91750114, 11922406, 91750202).

DATA AVAILABILITY

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * (1) L. Allen, M. W. Beijersbergen, R. J. C. Spreeuw, and J. P. Woerdman, Phys. Rev. A. 45, 8185 (1992).
  * (2) A. Forbes, Laser Photonics Rev. 13, 1900140 (2019).
  * (3) Y. J. Shen, X. J. Wang, Z. W. Xie, C. J. Min, X. Fu, Q. Liu, M. L. Gong and X. C. Yuan, Light Sci. Appl. 8, 90 (2019).
  * (4) A. Forbes, A. Dudley, and M. McLaren, Adv. Opt. Photon. 8, 200 (2016).
  * (5) X. L. Wang, J. P. Ding, W. J. Ni, C. S. Guo, and H. T. Wang, Opt. Lett. 32, 3549 (2007).
  * (6) H. Zhang and Y. Gao, Chin. Phys. Lett. 36, 114302 (2019).
  * (7) F. C. Mao, M. Huang, C. F. Yang, T. H. Li, J. L. Zhang, and S. Y. Chen, Chin. Phys. Lett. 35, 20701 (2018).
  * (8) M. Alois, A. Vaziri, G. Weihs, and A. Zeilinger, Nature. 412, 313 (2001).
  * (9) H. He, M. E. J. Friese, N. R. Heckenberg, and H. Rubinsztein-Dunlop, Phys. Rev. Lett. 75, 826 (1995).
  * (10) K. I. Willig, S. O. Rizzoli, V. Westphal, R. Jahn, and S. W. Hell, Nature 440, 935 (2006).
  * (11) Q. Zhan, Adv. Opt. Photon. 1, 1 (2009).
  * (12) G. Milione, H. I. Sztul, D. A. Nolan, and R. R. Alfano, Phys. Rev. Lett. 107, 053601 (2011).
  * (13) R. Dorn, S. Quabis, and G. Leuchs, Phys. Rev. Lett. 91, 233901 (2003).
  * (14) H. Wang, L. Shi, B. Lukyanchuk, C. Sheppard, and C. T. Chong, Nat. Photonics 2, 501 (2008).
  * (15) T. Bauer, P. Banzer, E. Karimi, S. Orlov, A. Rubano, L. Marrucci, E. Santamato, R. W. Boyd, and G. Leuchs, Science 347, 964 (2015).
  * (16) Y. Kozawa, D. Matsunaga, and S. Sato, Optica 5, 86 (2018).
  * (17) R. Camacho, M. Rahmani, S. Kruk, L. Wang, L. Xu, D. A. Smirnova, A. Solntsev, A. E. Miroshnichenko, H. H. Tan, F. Karouta, S. Naureen, K. Vora, L. Carletti, C. De Angelis, C. Jagadish, Y. S. Kivshar, and D. N. Neshev, Nano Lett. 16, 7191 (2016).
  * (18) R. Weber, A. Michalowski, M. Abdou-Ahmed, V. Onuseit, V. Rominger, M. Kraus, and T. Graf, Phys. Procedia 12, 21 (2011).
  * (19) J. T. Barreiro, T. C. Wei, and P. G. Kwiat, Nature Phys. 4, 282 (2008).
  * (20) D. Naidoo, F. S. Roux, A. Dudley, I. Litvin, B. Piccirillo, L. Marrucci, and A. Forbes, Nat. Photon. 10, 327 (2016).
  * (21) D. Mao, T. X. Feng, W. D. Zhang, H. Lu, Y. J. Jiang, P. Li, B. Q. Jiang, Z. P. Sun, and J. L. Zhao, Appl. Phys. Lett. 110, 021107 (2017).
  * (22) L. Marrucci, C. Manzo, and D. Paparo, Phys. Rev. Lett. 96, 163905 (2006).
  * (23) Z. Bomzon, G. Biener, V. Kleiner, and E. Hasman, Opt. Lett. 27, 285 (2002).
  * (24) M. Beresna, M. Gecevičius, P. G. Kazansky, and T. Gertus, Appl. Phys. Lett. 98, 201101 (2011).
  * (25) Q. S. Ferreira, A. J. Jesus-Silva, E. S. Fonseca, and J. M. Hickmann, Opt. Lett. 36, 3106 (2011).
  * (26) H. L. Zhou, S. Q. Yan, J. J. Dong, and X. L. Zhang, Opt. Lett. 39, 3173 (2014).
  * (27) H. Huang, Y. X. Ren, Y. Yan, N. Ahmed, Y. Yue, A. Bozovich, B. I. Erkmen, K. Birnbaum, S. Dolinar, M. Tur, and A. E. Willner, Opt. Lett. 38, 2348 (2013).
  * (28) P. Y. Li, B. Wang, X. B. Song, and X. D. Zhang, Opt. Lett. 41, 1574 (2016).
  * (29) D. Sarenac, C. Kapahi, A. E. Silva, D. G. Cory, I. Taminiau, B. Thompson, and D. A. Pushin, PNAS 117, 14682 (2020).
  * (30) J. M. Hickmann, E. J. S. Fonseca, W. C. Soares, and S. Chávez-Cerda, Phys. Rev. Lett. 105, 053904 (2010).
  * (31) C. S. Guo, L. L. Lu, and H. T. Wang, Opt. Lett. 34, 3686 (2009).
  * (32) Y. Zhang, P. Li, J. Z. Zhong, S. X. Qi, X. Y. Guo, D. J. Wu, S. Liu, and J. L. Zhao, Appl. Phys. Lett. 113, 221108 (2018).
  * (33) J. Zhou, W. Zhang, and L. Chen, Appl. Phys. Lett. 108, 111108 (2016).
  * (34) S. N. Alperin, R. D. Niederriter, J. T. Gopinath, and M. E. Siemens, Opt. Lett. 21, 5019 (2016).
  * (35) J. Lu, C. Cao, Z. Zhu, and B. Gu, Appl. Phys. Lett. 116, 201105 (2020).
  * (36) J. Mendoza-Hernández, M. F. Ferrer-Garcia, J. A. Rojas-Santana, and D. Lopez-Mago, Opt. Express 27, 31810 (2019).
  * (37) M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information, 10th Anniversary ed. (Cambridge U. Press, 2010).
  * (38) C. S. Guo, X. Liu, J. L. He, and H. T. Wang, Opt. Express 12, 4625 (2004)
  * (39) X. L. Wang, Y. H. Luo, H. H. Huang, M. C. Chen, Z. E. Su, C. Liu, C. Chen, W. Li, Y. Q. Fang, X. Jiang, J. Zhang, L. Li, N. L. Liu, C. Y. Lu, and J. W. Pan, Phys. Rev. Lett. 120, 260502 (2018).
  * (40) M. Wiesniak, T. Paterek, and A. Zeilinger, New J. Phys. 13, 053047 (2011).


# Heterogeneous Multi-task Learning with Expert Diversity

Raquel Aoki  1234-5678-9012 Simon Fraser UniversityBurnabyCanada<EMAIL_ADDRESS>,  Frederick Tung  Borealis AIVancouverCanada<EMAIL_ADDRESS>and  Gabriel L. Oliveira  Borealis AIVancouverCanada
<EMAIL_ADDRESS>
(2021)

###### Abstract.

Predicting multiple heterogeneous biological and medical targets is a
challenge for traditional deep learning models. In contrast to single-task
learning, in which a separate model is trained for each target, multi-task
learning (MTL) optimizes a single model to predict multiple related targets
simultaneously. To address this challenge, we propose the Multi-gate Mixture-
of-Experts with Exclusivity (MMoEEx). Our work aims to tackle the
heterogeneous MTL setting, in which the same model optimizes multiple tasks
with different characteristics. Such a scenario can overwhelm current MTL
approaches due to the challenges in balancing shared and task-specific
representations and the need to optimize tasks with competing optimization
paths. Our method makes two key contributions: first, we introduce an approach
to induce more diversity among experts, thus creating representations more
suitable for highly imbalanced and heterogenous MTL learning; second, we adopt
a two-step optimization (Finn et al., 2017; Lee and Son, 2020) approach to
balancing the tasks at the gradient level. We validate our method on three MTL
benchmark datasets, including Medical Information Mart for Intensive Care
(MIMIC-III) and PubChem BioAssay (PCBA).

multi-task learning; mixture of experts; neural network; task balancing

††copyright: acmcopyright††journalyear: 2021††conference: 20th International
Workshop on Data Mining in Bioinformatics; August 14–18, 2021;
Virtual††booktitle: BIOKDD ‘21: 20th International Workshop on Data Mining in
Bioinformatics - Held virtually in conjunction with SIGKDD 2021, August 14–18,
2021, Virtual††ccs: Computer methodologies Multi-task Learning††ccs: Neural
networks

##  1\. Introduction

Single-task learning (STL) models are the most traditional approach in machine
learning and have been extremely successful in many applications. This
approach assumes that the model is required to output a single prediction
target for a given input sample, such as a class label or a regression value.
If two output targets are associated with the same input data, then two
independent models are trained: one for each target or task (See Figure 1.a).
STL may be suitable for situations in which the tasks are very different from
each other, and in which computational efficiency may be ignored. However,
when the tasks are related, STL models are parameter inefficient (Mallya et
al., 2018; Zhai et al., 2020). In addition, in some applications, the synergy
among tasks can help a jointly trained model better capture shared patterns
that would otherwise be missed by independent training. For example, in
computer vision, the synergy between the dense prediction tasks of semantic
segmentation (the assignment of a semantic class label to each pixel in an
image) and depth estimation (the prediction of real-world depth at each pixel
in an image) can be leveraged to train a single neural network that achieves
higher accuracy on both tasks than independently trained networks (Ye et al.,
2019).

In contrast to STL, multi-task learning (MTL) optimizes a single model to
perform multiple related tasks simultaneously, aiming to improve
generalization and parameter efficiency across tasks. In this case, two or
more output targets are associated with the same input data (See Figures 1.b,
1.c and 1.d). Effective multi-tasking learning typically requires task
balancing to prevent one or more tasks from dominating the optimization, to
decrease negative transfer, and to avoid overfitting. Standard MTL settings
usually assume a homogeneous set of tasks, for example all tasks are
classification or regression tasks, and usually they are non-sequential data.
This scenario can greatly benefit MTL approaches with strong shared
representations. In contrast, heterogeneous multi-task learning is defined by
multiple classes of tasks, such as classification, regression with single or
multi label characteristics and temporal data, being optimized simultaneously.
The latter setting is more realistic but lacks further exploration. Ma et al.
(2018) proposed a Multi-gate Mixture-of-Experts (MMoE), a model that combines
experts using gate functions. In this case, each expert is one or more neural
network layers shared among the tasks. MMoE tends to generalize better than
other models because it leverages several shared bottoms (experts) instead of
using a single architecture. It allows dynamic parameter allocation to shared
and task-specific parts of the network thus improving further the
representation power. In order to take advantage of these characteristics and
extend it to heterogenous MTL problems, we introduced MMoEEx.

The multi-gate mixture-of-experts with exclusivity (MMoEEx) model is a new
mixture-of-experts (MMoE) approach to MTL that boosts the generalization
performance of traditional MMoE via two key contributions:

  * •

The experts in traditional MMoE are homogeneous, which limits the diversity of
the learned representations. Inspired by ensemble learning, we improve the
generalization of traditional MMoE by inducing diversity among experts. We
introduce novel exclusion and exclusivity conditions, under which some experts
only contribute to some tasks, while other experts are shared among all tasks.

  * •

We introduce a two-step task balancing optimization at the gradient level
inspired by MAML(Finn et al., 2017). This enables MMoEEx to support the
learning of unbalanced heterogeneous tasks, in which some tasks may be more
susceptible to overfitting, more challenging to learn, or operate at different
loss scales.

We start our experiments by exploring how MMoEEx behaves in the UCI Census-
income dataset (Ma et al., 2018; Dua and Graff, 2017), a standard MTL
benchmark dataset for low cardinality tasks. We compare it with several state-
of-the-art multi-task models and show that our technique outperforms the other
approaches.

We further evaluated the performance of MMoEEx on two medical and biological
benchmark datasets. Medical Information Mart for Intensive Care (MIMIC-III)
(Johnson et al., 2016; Harutyunyan et al., 2019) is a heterogeneous time
series multi-task learning dataset. The mixture of multi-label and single-
label temporal tasks with non-temporal binary classification makes this
dataset ideal to benchmark our approach. The large scale and high task
imbalance characteristics of the dataset also provide a scenario to exploit
the robustness of our approach to competing tasks. We observe significant
improvements in the AUC metrics against all compared approaches, especially
the MMoE (Ma et al., 2018) technique.

PubChem BioAssay (PCBA) (Wang et al., 2017; Liu et al., 2019b) is a non-
temporal homogeneous (only binary classification) high task cardinality
dataset. PCBA is less challenging than the MIMIC-III but is the only available
dataset with more than a hundred tasks, thus able to better benchmark
scalability and negative transfer aspects of MTL approaches. Our results
confirm the effectiveness of MMoEEx on large task cardinality datasets and
show that our approach has performance on par with the current state-of-the-
art.

##  2\. Related Work

Figure 1. Neural Network Architectures.(a) Single-Task Learning, (b) Multi-
task learning Hard-parameter sharing (Caruana, 1993), (c) Multi-gate Mixture-
of-Experts (MMoE)(Ma et al., 2018), (d) MMoEEx – proposed method.

The recent works in deep learning for multi-task learning (MTL) can be divided
into two groups: the ones focused on the neural network architecture, which
study what, when and how to share information among the tasks; and the works
focused on the optimization, which usually concentrate on how to balance
competing tasks, which are jointly learned. Our work makes contributions to
both fields.

MTL architectures can be divided into two main groups, hard parameter sharing
and soft parameter sharing. One of the first works in MTL uses hard-parameter
sharing. In this type of architecture proposed by Caruana (1993), the bottom
layers of the neural network are shared among all the tasks, and the top
layers are task-specific. Another example of a shared bottom approach is
UberNet (Kokkinos, 2017). UberNet consists of a shared encoder that is
followed by a series of task-specific decoding heads that merge encoder data
from different layers and scales. On one hand, the main advantage of this
class of methods is its scale invariance to a large number of tasks. On the
other hand, with a shared representation the resulting features can become
biased towards the tasks with strong signals.

The second group of MTL topologies have a dedicated set of parameters to each
task. Such methods are called soft parameter sharing (Duong et al., 2015).
They can be interpreted as single networks that have a feature sharing
mechanism between encoders to induce inter branch information crossing.
Methods like Cross-Stitch Network (Misra et al., 2016), Multi-gate Mixture of
Experts (Ma et al., 2018) and Multi-task Attention Network (Liu et al., 2019a)
are examples of soft parameter sharing based on a explicit feature sharing
mechanism, mixture of experts feature fusion and attention based approaches to
cross-task among branches. Misra et al. (Misra et al., 2016) introduced the
concept of soft-parameter sharing in deep multi-task approaches by learning a
linear combination of the input activation maps. The linear combination (soft
feature fusion) is learned at each layer from both tasks. The MMoE method
proposed by Ma et al. (2018) is an attempt to provide a soft parameter sharing
mechanic through a gating mechanism. The gate function selects a set of
experts for each task while re-using it for multiple tasks, consequently
providing feature sharing. A new technique for soft parameter MTL is using
attention mechanisms. The work from Liu et al. (Liu et al., 2019a) called
Multi-Task Attention Network (MTAN) used an attention mechanism to share the
features for each task specific network. The main advantage of soft parameter
sharing approaches is the capability of learning task specific and shared
representations explicitly. Nevertheless, these models suffer from scalability
problems, as the size of the MTL network tends to grow proportionally with the
number of tasks. A full survey on different MTL topologies can be found at
(Ruder, 2017; Vandenhende et al., 2020).

The previously mentioned works focused on better network structures for MTL.
Another significant problem of learning multiple tasks is related to the
optimization procedure. MTL methods need to balance gradients of multiple
tasks to prevent one or more tasks from dominating the network and producing
task biased predictions. The optimization methods can be divided into loss
balancing techniques (Liu et al., 2019b, a), gradient normalization (Chen et
al., 2018) and model-agnostic meta-learning (Finn et al., 2017; Lee and Son,
2020). Liu et al. (Liu et al., 2019b) proposed a loss balance approach based
on loss ratios between the first batch and all subsequent ones in each epoch.
The method called Loss-Balanced Task Weighting (LBTW) showed promising results
reducing the negative transfer on a $128$ task scenario. Another balancing
approach operating in the loss level is the Dynamic Weight Average (DWA)
proposed by Liu et al. (Liu et al., 2019a). The main difference between them
is that DWA will need to compute the average loss per epoch for each task
before start the balancing procedure, thus operating in the epoch level and
not on the batch level like LBTW. Loss balancing approaches have as their main
drawbacks its suboptimality when task gradients are conflicting or when a set
of tasks have gradient magnitudes higher than others. In order to mitigate
these limitations of loss based approaches GradNorm (Chen et al., 2018) and
Model-Agnostic Meta-Learning (MAML) for MTL (Finn et al., 2017; Lee and Son,
2020) were proposed.

Gradient normalization proposed by Chen et al. (Chen et al., 2018) aims to
control the training through a mechanism that encourages all tasks to have
similar magnitude. Additionally to it the model also balances the pace tasks
are learned. More recently, methods based on meta-learning (Finn et al., 2017)
emerged and outperformed previous loss-based approaches and gradient
normalization techniques (Lee and Son, 2020). Lee et al. (Lee and Son, 2020)
proposed a multi-step approach which updates each task in an exclusive
fashion. The method is capable of not only providing a balanced task
optimization but also boosts current MTL architectures. MTL meta-learning
methods, while being the current state-of-the-art class of approaches, can
become impractical for settings with large cardinality based on the
intermediate steps which are needed to task state computation.

The proposed multi-gate mixture-of-experts with exclusivity (MMoEEx) approach
has similarities with the MMoE approach (Ma et al., 2018) and MAML (Finn et
al., 2017). Our technique is an MMoE approach, however we introduce an
exclusivity mechanism that provides an explicit sparse activation of the
network, enabling the method to learn task specific features and a shared
representation simultaneously. We also tackle the scalabilty limitation of
MMoE techniques with our exclusion gates. Our approach is inspired by the MAML
technique in which we aim to introduce a two step approach to balance tasks at
the gradient level for mixture of experts.

##  3\. Methodology

Hard-parameter sharing networks (Caruana, 1993) shown in Figure 1.b are one of
the pillars of multi-task learning. These networks are composed of a shared
bottom and task-specific branches. Ma et al. (2018) suggested that a unique
shared bottom might not be enough to generalize for all tasks in an
application, and proposed to use several shared bottoms, or what they call
experts. The experts are combined using gate functions, and their combination
is forwarded to the towers. The final architecture is called Multi-gate
Mixture-of-Experts (MMoE), and is shown in Figure 1.c. MMoE generalizes better
than its traditional hard-parameter sharing counterpart, but there are two
weaknesses: first, it lacks a task-balancing mechanism; second, the only
source of diversity among the experts is due to the random initialization.
Although the experts can indeed be diverse enough if they specialize in
different tasks, there are no guarantees that this will happen in practice. In
this work, we propose the Multi-gate Mixture-of-Experts with Exclusivity
(MMoEEx) (Figure 1.d), a model that induces more diversity among the experts
and has a task-balancing component.

###  3.1. Structure

The neural network architecture follows the structure proposed by Ma et al.
(2018) and can be divided into three parts: gates, experts, and towers.
Considering an application with $K$ tasks, input data $x\in\mathbb{R}^{d}$,
the gate function $g^{k}()$ is defined as:

(1) $g^{k}(x)=\text{softmax}(W^{k}x),\forall k\in\\{0,...,K\\}$

where $W^{k}\in\mathbb{R}^{E\times d}$ are learnable weights and $E$ is the
number of experts, defined by the user. The gates control the contribution of
each expert to each task.

The experts $f_{e}(),\forall e\in\\{0,...,E\\}$, and our implementation is
very flexible to accept several experts architectures, which is essential to
work with applications with different data types. For example, if working with
temporal data, the experts can be LSTMs, GRUs, RNNs; for non-temporal data,
the experts can be dense layers. The number of experts $E$ is defined by the
user. The experts and gates’ outputs are combined as follows:

(2) $f^{k}(x)=\sum_{e=0}^{E}g^{k}(x)f_{e}(x),\forall k\in\\{0,...,K\\}$

The $f^{k}()$ are input to the towers, the task-specific part of the
architecture. Their design depends on the data type and tasks. The towers
$h^{k}$ output the task predictions as follows:

(3) $y^{k}=h^{k}(f^{k}(x)),\forall k\in\\{0,...,K\\}$

###  3.2. Diversity

In ensemble learning, models with a significant diversity among their learners
tend to generalize better. MMoE (Ma et al., 2018) leverages several experts to
make its final predictions; however, it relies only on random initialization
to create diversity among the experts, and on the expectation that the gate
function will learn how to combine these experts. Here we propose two
mechanisms to induce diversity among the experts, defined as exclusion and
exclusivity:

  * •

Exclusivity: We set $\alpha E$ experts to be exclusively connected to one
task. The value $\alpha\in[0,1]$ controls the proportion of experts that will
be exclusive. If $\alpha=1$, all experts are exclusive, and if $\alpha=0$, all
experts are shared (same as MMoE). An exclusive expert is randomly assigned to
one of the tasks $T_{k}$, but the task $T_{k}$ can still be associated with
other exclusive experts and shared experts.

  * •

Exclusion: We randomly exclude edges/connections between $\alpha E$ experts
and tasks. If $\alpha=1$, all experts will have one connection randomly
removed, and if $\alpha=0$, there is no edge deletion (same as MMoE).

For applications with only two tasks ($K=2$), exclusion and exclusivity
mechanisms are identical. The exclusion mechanism is more scalable than the
exclusivity mechanism because it does not require one expert per task, and
therefore, works well in applications with a large number of tasks. For a
small set of tasks, both approaches have similar results. MMoEEx, similarly to
MMoE, relies on the expectation that gate functions will learn how to combine
the experts. Our approach induces more diversity by forcing some of these
gates to be ‘closed’ to some experts, and the exclusivity and exclusion
mechanisms are used to close part of the gates. The remaining non-closed gates
learn to combine the output of each expert based on the input data, according
to Equation 1.

As Equation 2 shows, the gates are used as experts weights. Therefore, if an
expert $e\in\\{0,...,E\\}$ is exclusive to a task $k\in\\{0,...,K\\}$, then
only the value $g^{k}[e]\neq 0$, and all the other gates for that expert are
‘closed’: $g^{m}[e]=0,m\in\\{0,...,K\\},m\neq k$.

###  3.3. MAML - MTL optimization

The goal of the two-step optimization is to balance the tasks on the gradient
level. Finn et al. (2017) proposed the Model-agnostic Meta-learning (MAML), a
two-step optimization approach originally intend to be used with transfer-
learning and few-shot learning due to its fast convergence. MAML also has a
promising future in MTL. Lee and Son (2020) first adapted MAML for Multi-task
learning applications, showing that MAML can balance the tasks on the gradient
level and yield better results than some existing task-balancing approaches.
The core idea is that MAML’s temporary update yields smoothed losses, which
also smooth the gradients on direction and magnitude.

In our work, we adopt MAML. However, differently from Lee and Son (2020), we
do not freeze task-specific layers during the intermediate/inner update. The
pseudocode of our MAML-MTL approach is shown in Algorithm 1.

Sample batch $X$;

$loss$ = 0;

for  _$T$ in TASKS_ do

Evaluate $\Delta_{\theta}\mathcal{L}_{T}(f_{\theta}(X))$ ;

Temporary Update
$\theta^{\prime}_{T}\leftarrow\theta-\Delta_{\theta}\mathcal{L}_{T}(f_{\theta}(X))$;

Re-evaluate and save
$loss=loss+\Delta_{\theta^{\prime}_{T}}\mathcal{L}_{T}(f_{\theta^{\prime}_{T}}(X))$;

Update $\theta\leftarrow\theta-loss$

Algorithm 1 MAML-MTL

Section 4 shows the results of the experiments using our two-step optimization
strategy. One weakness of this approach is the running time. Temporary updates
are expensive, making infeasible the use of MAML in applications with many
tasks.

##  4\. Experiments

In this section, we develop experiments to answer two main questions to
validate our proposed method:

  1. (1)

Our proposed method MMoEEx has better results than existing MTL baselines,
such as MMoE, Hard-parameter sharing (shared bottom), Multi-Channel Wise LSTMs
(time-series datasets);

  2. (2)

Our proposed method MMoEEx, has better results than Single Task Learning (STL)
methods.

Furthermore, we also explore some secondary results, such as the influence of
the expert complexity and the number of experts on the results, and the
comparison of expert’s diversity in our proposed method and our main baseline.

###  4.1. Datasets

We evaluate the performance of our approach on three datasets. UCI-Census-
income dataset (Ma et al., 2018; Dua and Graff, 2017), Medical Information
Mart for Intensive Care (MIMIC-III) database (Johnson et al., 2016;
Harutyunyan et al., 2019) and PubChem BioAssay (PCBA) dataset (Wang et al.,
2017). A common characteristic among all datasets is the presence of very
unbalanced tasks (few positive examples).

#### UCI - Census-income dataset(Ma et al., 2018; Dua and Graff, 2017)

Extracted from the US 1994 census database, there are 299,285 answers and 40
features, extracted from the respondent’s socioeconomic form. Three binary
classification tasks are explored using this dataset:

  1. (1)

Respondent income exceeds $50K;

  2. (2)

Respondent’s marital status is “ever married”;

  3. (3)

Respondent’s education is at least college;

#### Medical Information Mart for Intensive Care (MIMIC-III) database (Johnson
et al., 2016; Harutyunyan et al., 2019)

This database was proposed by Harutyunyan et al. (2019) to be a benchmark
dataset for MTL in time-series data. It contains metrics of patients from over
40,000 intensive care units (ICU) stays. This dataset has 4 tasks: two binary
tasks, one temporal multi-label task, and one temporal temporal
classification. Figure 2 shows the neural network adopted in our work and
where each task is calculated.

Figure 2. The neural network architecture adopted in our work is the same used
by the Multi-Channel Wise LSTM(Harutyunyan et al., 2019). The input data
$x_{t}$ has 76 features, and the size of the hidden layer $h_{t}$ depends on
the model adopted. There are four tasks: the decompensation $d_{t}$ and LOS
$l_{t}$ calculated at each time step, mortality $m_{48}$, and the phenotype
$p_{T}$, both calculated only once per patient.

Tasks description:

  1. (1)

Phenotype prediction: measured on the end of stay, classify if the patient has
25 acute care conditions ($p_{T}$ in Figure 2). In practice, we have 25
parallel binary classification tasks;

  2. (2)

Length-of-stay (LOS) prediction: the goal is to predict the remaining time
spend in ICU at each hour of stay ($l_{t}$ in Figure 2). Following Harutyunyan
et al. (2019), the remaining time was converted from a regression task to a
multi-label task. We have 10 classes, one class for each one of the first 8
days, between 8-15 days, and +15 days;

  3. (3)

Decompensation prediction: aim to predict if the patient state will rapidly
deteriorate in the next 24 hours. We follow the Harutyunyan et al. (2019)
scheme, where due to lack of gold standard, the task is redefined as mortality
prediction in the next 24 hours at each hour of an ICU stay, in practice, a
temporal binary classification ($d_{t}$ in Figure 2);

  4. (4)

In-hospital mortality prediction: binary classification in the end of the
first 48 hours of a patient in an ICU stay ($m_{48}$ in Figure 2);

#### PubChem BioAssay (PCBA)(Wang et al., 2017; Liu et al., 2019b) Database

We worked with a subset of the PCBA, composed of 128 binary tasks/biological
targets and 439,863 samples. Each sample represents a molecule, pre-processed
using Circular Fingerprint (Liu et al., 2019b) molecule feature extractor,
that creates 1024 features. These features are used to determine whether the
chemical affects a biological target, here defined as our tasks.

###  4.2. Design of experiments

The split between train, validation, and test set was the same used by our
baselines to offer a fair comparison. For the UCI-Census, the split was
66%/17%/17% for training/validation/testing sets, for the MIMIC-III and PCBA
70%/15%/15%. The data pre-processing, loss criterion, optimizers, parameters,
and metrics description of our experiments is shown in Section 4.7. The metric
adopted to compare results is AUC (Area Under The Curve) ROC for the binary
tasks and Kappa Score for the multiclass tasks.

###  4.3. UCI - Census-income Study

In this subsection, we report and discuss experimental results on the census-
income data. We present experiments predicting income, marital status, and
education level.

The census income, marital status and education dataset experiments are
presented at Table 1. We can observe that our approach outperforms all the
baselines with the exception of the Education task where the single-task
method presents a marginal improvement over MMoEEx. We argue that the Census
tasks already present slightly conflicting optimization goals and, in this
situation, our approach is better suited to balance multiple competing tasks.

Table 1. Results on Census Income/Marital/Education dataset. $\Delta$ is the average relative improvement. MMoEEx Exclusivity $\alpha=0.5$. Method | AUC | $\Delta$
---|---|---
Income | Marital Stat | Education
Single-Task | $88.95$ | $97.48$ | 87.23 | -
Shared-Bottom | $91.09$ | $97.98$ | $86.99$ | $+0.85\%$
MMoE | $90.86$ | $96.70$ | $86.33$ | $-0.28\%$
MMoEEx (Ours) | 92.51 | 98.47 | $87.19$ | +1.65%

We can conclude that our MMoEEx approach can better learn multiple tasks when
compared to standard shared bottom approaches and the MMoE baseline, due to
the exclusivity and the multi-step optimization contributions of our work.

###  4.4. MIMIC-III Study

MIMIC-III dataset is the main benchmark for heterogeneous MTL with time
series. The dataset consists of a mixture of multi-label and single-label
temporal tasks and two non-temporal binary classification tasks. Our
experiments will first investigate the best recurrent layers to be selected as
experts to the MMoEEx model. The following sections show an ablation study on
the impact of higher experts cardinality and our full scale baseline
evaluation.

####  4.4.1. Recurrent Modules Ablation Study

One of the main design choices for time series prediction is the type of
recurrent unit to be deployed. The goal of this ablation study is to provide a
thorough analysis on the impact of different recurrent layers to our approach.
The layers taken into consideration range from the standard RNN’s (Rumelhart
et al., 1986), LSTM’s (Hochreiter and Schmidhuber, 1997) and GRU’s (Cho et
al., 2014) to modern recurrent layers like the Simple Recurrent Units (SRU)
(Lei et al., 2018) and Independent Recurrent Networks (IndRNN) (Li et al.,
2018).

Table 2. Results on MIMIC-III recurrent modules ablation study. All the MMoEEx configurations count with $12$ experts based on memory limitations of approaches like IndRNN and LSTM. MMoEEx Exclusivity $\alpha=0.5$. Method | Pheno | LOS | Decomp | Ihm
---|---|---|---|---
MMoEEx - SRU (Lei et al., 2018) | $71.00$ | 57.88 | $96.67$ | $89.95$
MMoEEx - IndRNN (Li et al., 2018) | $67.49$ | $57.11$ | $95.89$ | 91.68
MMoEEx - IndRNNV2 (Li et al., 2018) | $68.15$ | $54.48$ | $96.50$ | $90.58$
MMoEEx - LSTM (Hochreiter and Schmidhuber, 1997) | $73.48$ | $45.99$ | $96.54$ | $90.88$
MMoEEx - RNN (Rumelhart et al., 1986) | $73.40$ | $55.56$ | $96.85$ | $91.08$
MMoEEx - GRU (Cho et al., 2014) | 74.08 | $54.48$ | 97.20 | $91.49$

MIMIC-III recurrent modules ablation study is presented in Table 2. SRU and
IndRNN outperform the other methods from length-of-stay (LOS) task. Besides
that MMoEEx with IndRNN also is the top performer for the in-hospital
mortality (Ihm) task. Besides the good performance of SRU and IndRNN for these
tasks, they present an imbalanced performance over all considered tasks and
also impose a memory and runtime burden, making the scalability of MMoEEx to
higher number of experts infeasible. Taking the overall task performance into
consideration, RNN and GRU outperform the compared recurrent approaches. RNN,
in addition to being a top performer expert, also presented the lowest memory
footprint and consequently is capable of providing MMoEEx with more experts if
needed. From this part on we will have MMoEEx’s with RNN’s or GRU’s as their
recurrent layers.

####  4.4.2. Impact of experts cardinality

During the training of our approach for the MIMIC-III experiments we noticed
that a larger number of experts, when connected with our exclusivity
mechanism, gave better overall results. In order to further explore this
parameter we conducted a series of experiments where we trained our MMoEEx
with RNN’s with a number of experts ranging from $12$ to $64$ experts. We
choose RNN’s as the recurrent layer in this experiment based on its low memory
requirement.

Figure 3 depicts our results for the four tasks on the MIMIC-III dataset. LOS
tasks is the one that take most advantage of a larger number of experts with
an improvement superior to $17$ percentage points or a $38$ percent relative
improvement. The remaining tasks are stable for a higher cardinality of
experts. We believe a higher number of experts allow MMoEEx to have a better
representation to challenging tasks when the shared representation is not been
updated with the same magnitudes due to the other tasks have reached
stability. The number of $32$ experts gave MMoEEx the best overall and LOS
performance. The final results on MIMIC-III are all using $32$ experts.

Figure 3. Impacts of cardinality of experts for MMoEEx model on MIMIC-III. Table 3. Final results MIMIC-III. MMoEEx outperforms all the compared baselines with the exception to Pheno where the work from Johnson et al. (2016) is the best approach. Our approach can provide a relative improvement superior to $40$ percentage points when compared to the Multitask channel wise LSTM for the LOS task. MMoEEx Exclusivity $\alpha=0.5$. Method | Pheno | LOS | Decomp | Ihm | $\Delta$
---|---|---|---|---|---
MCW-LSTM (Johnson et al., 2016) | 77.4 | $45.0$ | $90.5$ | $87.0$ | $+0.28\%$
Single Task (Johnson et al., 2016) | $77.0$ | $45.0$ | $91.0$ | $86.0$ | -
Shared Bottom | $73.36$ | $30.60$ | $94.12$ | $82.71$ | $-9.28\%$
MMoE | 75.09 | $54.48$ | $96.20$ | $90.44$ | $+7.36\%$
MMoEEx - RNN | $72.44$ | 63.45 | $96.82$ | $90.73$ | +11.74%
MMoEEx - GRU | $74.57$ | $60.63$ | 97.03 | 91.03 | +11.00%

####  4.4.3. MiMIC-III Results

The full set of results for MIMIC-III dataset is presented in Table 3. We
compared our approach with the multitask channel wise LSTM (MCW-LSTM) (Johnson
et al., 2016), single task trained network, shared bottom, MMoE (Ma et al.,
2018) and two variations of MMoEEx with RNN’s and GRU’s.

MMoEEx outperforms all the compared approaches except on the Phenotype (Pheno)
task. For both time series tasks (LOS and Decomp) our approach outperforms all
baselines. It is worth noting that for the LOS task, which is the hardest task
on MIMIC-III, we present a relative improvement superior to $40$ percentage
points when compared to multitask channel wise LSTM (Johnson et al., 2016) and
over $16$ percentage points to MMoE for our MMoEEx with RNN’s. MMoEEx with
GRU’s presents a better individual task performance than its RNN counterpart
but with lower LOS task performance.

###  4.5. PubChem BioAssay Dataset Study

Table 4. PCBA’s final results. Our approach has competitive results when compared with the baselines. NT is Negative Transfer, $\Delta$ is Average Relative Improvement.  Method | Average | Std |  CI $(95\%)$ | NT | $\Delta$
---|---|---|---|---|---
AUC | Dev |  |  |
STL(Liu et al., 2019b) | 79.9 | 10.62 | [78.04, 81.72] | - | -
MTL(Liu et al., 2019b) | 85.7 | 8.75 | [84.20, 87.24] | 13 | +8.51%
Fine Tuning (Liu et al., 2019b) | 80.6 | 10.46 | [78.79, 82.42] | 50 | +0.87%
GradNorm(Liu et al., 2019b) | 84.0 | 7.98 | [82.58, 85.35] | 44 | +5.13%
RMTL(Liu et al., 2019b) | 85.2 | 8.77 | [83.71, 86.75] | 11 | +6.63%
LBTW($\alpha=0.1$)(Liu et al., 2019b) | 85.9 | 8.49 | [84.45, 87.39] | 13 | +7.51%
LBTW($\alpha=0.5$)(Liu et al., 2019b) | 86.3 | 8.09 | [84.86, 87.67] | 11 | +8.01%
Shared Bottom | 86.8 | 8.53 | [84.62, 87.58] | 10 | +8.63%
MMoE | 85.8 | 8.83 | [84.10, 87.16] | 15 | +7.38%
MMoEEx (ours) | 85.9 | 8.61 | [84.18, 87.16] | 13 | +7.50%

The PCBA dataset has 128 tasks and is the main benchmark for scalability and
negative transfer. All the 128 tasks are binary classification tasks, and they
are very similar to each other. Our experiments first compare our approach
with existing baselines on the tasks’ average AUC and number of tasks with
negative transfer. Then, we have a second ablation study comparing our MMoEEx
approach with the MMoE on the number of experts and overfitting evaluation.

####  4.5.1. Comparison with existing baselines

We used the work of Liu et al. (2019b), as our main baseline, due to the
variety of tested technique, such as GradNorm (Chen et al., 2018), RMTL (Liu,
2018) and LBTW (Liu et al., 2019b). Additionally we included a shared bottom
and a MMoE techniques to our baselines.

The architecture adopted for baselines and experts is very similar (more
details see Section 4.7) For this application, we did not use MAML-MTL
optimization due to scalability issues. Therefore, the difference between the
MMoE and MMoEEx in this application is the diversity of experts: all MMoE’s
experts are shared among all tasks, versus only a portion of MMoEEx are
shared. Table 4 shows the final results. We adopted four metrics to compare
our results with the baselines: the average ROC AUC of all tasks, Standard
Deviation of the ROC AUC, $\Delta$, and the number of negative transfer (NT).
The NT is calculated using Single Task Learning Models, and counts how many
tasks have a worse result on the multi-task learning approach. Figure 4 shows
the improvement of each model in comparison with the STL model, where tasks
below 0 indicates NT.

Considering all the baselines, the shared bottom fitted in our study has the
best overall result (largest average AUC, smaller NT). Using the tasks AUC, we
constructed 95% confidence intervals, shown in Table 4, from where we conclude
there is no significant difference between RMTL, MTL, LBTW, Shared Bottom,
MMoE, and MMoEEx. Therefore, our proposed method MMoEEx has a competitive
result when compared with other baselines. We highlight that LBTW and GradNorm
are both focused on task balancing. However, the PCBA dataset has very similar
tasks, which almost makes unnecessary the task balancing component. The shared
bottom model, for example, does not have any task balancing approach and has
the best performance overall.

Figure 4. The plot shows change on the AUC for each task $k\in\\{1,...,128\\}$
relative to the Single-task Learning(STL) AUC. Values below 0 indicate
negative transfer. MMoEEx Exclusivity $\alpha=0.5$.

####  4.5.2. Impact of number of experts

Figure 5. Comparison of the ROC AUC versus number of experts. We changed the
MMoEEx exclusivity parameter $\alpha$ by increments of 0.09, starting with
$\alpha=0.42$, so there are always only two shared experts. Figure 6. MMoE
($\bar{d}=0.311$) and MMoEEx($\bar{d}=0.445$) heatmap in the MIMIC-III
dataset. The MMoE has 12 shared experts versus 6 shared and 6 exclusive
experts in the MMoEEx model. Darker colors indicate more dissimilarities
between two experts and, therefore, more diversity. The plot is generated with
12 instead of 32 experts to better visualize the distances; the results also
hold in the setting with 32 experts.

We also make a direct comparison between our method MMoEEx and our main
baseline MMoE. In this dataset, fixing the number of experts, our proposed
method MMoEEx has a better average ROC AUC on the testing set than the MMoE,
as Figure 5 shows. In this study, we fix the number of shared experts in the
MMoEEx as 2. With three experts, we adopted $\alpha=0.42$, and to each new
expert added, we increment the value of $\alpha$ by 0.09. Therefore, with
eight experts, we have two shared experts and $\alpha=0.87$. The plot shows
that the inclusion of more diversity on the experts through expert exclusivity
helped the model to generalize better on the testing set and decreased
overfitting.

###  4.6. Diversity Score Study

We propose a diversity measurement to support our claim that our method MMoEEX
induced more diversity among the experts than our baseline MMoE. The diversity
among the experts can be measured through the distance between the experts’
outputs $f_{e},\forall e\in\\{0,...,E\\}$. Considering a pair of experts $i$
and $j$, the distance between them is defined as:

$d_{i,j}=\sqrt{\sum_{n=0}^{N}(f_{i}(x_{n})-f_{j}(x_{n}))^{2}}$

where $N$ is the number of samples in the dataset, $d_{i,j}=d_{j,i}$, and a
matrix $D\in\mathbb{R}^{E\times E}$ is used to keep all the distances. To
scale the distances into $d_{i,j}\in[0,1]$, we divide the raw entries in the
distance matrix $D$ by the maximum distance observed, $\text{max}(D)$. A pair
of experts $i,j$ with $d_{i,j}=0$ are considered identical, and experts
distances $d_{i,j}$ close to 0 are considered very similar; analogously,
experts with $d_{i,j}$ close to 1 are considered very dissimilar. To compare
the overall distance between the experts of a model, we define the diversity
score $\bar{d}$ as the mean entry in $D$.

Figure 7. MMoE ($\bar{d}=0.557$) and MMoEEx($\bar{d}=0.600$) heatmap for the
PCBA dataset. The MMoEEx model has 2 shared experts and 2 experts with
exclusion.

In this section, we analyze the diversity score of the MMoE and MMoEEx in our
benchmark datasets. The MMoE and MMoEEx models compared using the same dataset
have the same neural network structure, but the MMoEEx uses the MAML - MTL
optimization and has the diversity enforced. The MMoEEx models in Figures 6
and 7 were created with $\alpha=0.5$ and exclusivity. In other words, half of
the experts in the MMoEEx models were randomly assigned to be exclusive to one
of the tasks, while the MMoE results have $\alpha=0$ (all experts shared among
all tasks). Figure 6 shows a heatmap of the distances $D^{MMoE}$ and
$D^{MMoEEx}$ calculated on the MIMIC-III testing set with 12 experts. The
MMoE’s heatmap has, overall, lighter colors and a smaller diversity score than
the MMoEEx. Figure 7 shows the MMoE and MMoEx heatmap for the PCBA dataset,
with 128 tasks and 4 experts. Our proposed method MMoEEx also has a larger
diversity score $\bar{d}$ and darker colors.

In summary, our approach works extremely well on the heterogeneous dataset,
MIMIC-III, increasing the diversity score by $43.0$%. The PCBA is a
homogeneous dataset, but the diversity component still positively impacts and
increases the diversity score by $7.7$%. Finally, as the most homogeneous and
simplest dataset adopted in our study, the Census dataset is the only one that
does not take full advantage of the experts’ diversity. MMoE’s diversity score
was $0.410$ versus $0.433$ for the MMoEEx’s model, which is only $5.6$%
improvement.

These results show that our approach indeed increased the experts’ diversity
while keeping the same or better tasks’ AUC (See Tables 1, 4 and 3), which was
one of our main goals.

###  4.7. Experiments Reproducibility

Table 5. Models’ architecture, training information, and dataset pre-processing’s references for experiment reproducibility. Dataset |  Pre-processing | Epochs | Experts |  Loss |  Layers
---|---|---|---|---|---
UCI-Census |  Ma et al. (2018) | 200 | 12 |  $BCEWithLogitsLoss$ |  Experts: $Linear(4)+ReLU$, Towers: $Linear(4)+Linear(1)$
MIMIC-III |  Harutyunyan et al. (2019); Johnson et al. (2016) | 50 | 12 and 32 |  $BCEWithLogitsLoss$, $CrossEntropyLoss$ (multilabel task), $pos\\_weight$: $pheno=5$, $LOS=1$, $Decomp=25$, $Ihm=5$ |  Experts: RNN(128) or GRU(128); Towers: $Linear(16)+Linear(output)$, where the output depends on the task. Three towers had time-series data, and one had only the first 24 observations of the time-series.
PCBA |  Liu et al. (2019b) | 100 | 2 or 4 |  $BCEWithLogitsLoss$, $pos\\_weight=100$ |  $Linear(2000)+Dropout(0.25)+ReLU+Linear(2000)+Sigmoid+Linear(2000)$. The tower had one $Linear(1)$ layer per task.

We used PyTorch in our implementation, and the code will be made available at
https://github.com/BorealisAI/MMoEEx-MTL. We used Adam optimizer with learning
rate $0.001$, weight decay $0.001$, and learning rate decreased by a factor of
$0.9$ every ten epochs. The metric adopted to compare the models was ROC AUC,
with the exception of the task LOS on MIMIC-III dataset, which was Cohen’s
kappa Score, a statistic that measures the agreement between the observed
values and the predicted. We trained the models using the training set, and we
used the task’s AUC sum in the validation set to define the best model, where
the largest sum indicates the best epoch, and consequently, the best model.
Table 5 shows a summary of the models adopted for future reference.

##  5\. Conclusion

We proposed a novel multi-task learning approach called Multi-gate Mixture-of-
Experts with Exclusivity (MMoEEx), which extends MMoE methods by introducing
an exclusivity and exclusion mechanism that induces more diversity among
experts, allowing the network to learn representations that are more effective
for heterogeneous MTL. We also introduce a two-step optimization approach
called MAML-MTL, which balances tasks at the gradient level and enhances
MMoEEx’s capability to optimize imbalanced tasks. We show that our method has
better results than baselines in MTL settings with heterogenous tasks, which
are frequently found in biological applications. Experiments on biological and
clinical benchmark datasets demonstrate the success of our proposed method in
homogeneous and heterogeneous settings, where we outperform several state-of-
the-art baselines.

Besides the contributions above, we believe the ablation analysis on MMoEEx
can open the opportunity to further explore heterogeneous multi-task learning.
The substantial improvement on the hardest task of the MIMIC-III dataset is an
indication of this possible investigation direction. The MAML-MTL optimization
is also in its infancy, and more research on meta-learning task balancing can
greatly benefit MTL research. We hope this work inspires other researchers to
further investigate multi-task learning at the network architecture and
optimization levels.

## References

  * (1)
  * Caruana (1993) Rich Caruana. 1993\.  Multitask Learning: A Knowledge-Based Source of Inductive Bias.
  * Chen et al. (2018) Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  , 794–803 pages.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.  , 1724–1734 pages.
  * Dua and Graff (2017) Dheeru Dua and Casey Graff. 2017.  UCI Machine Learning Repository.  http://archive.ics.uci.edu/ml
  * Duong et al. (2015) Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015\.  Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser.  , 845–850 pages.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-agnostic meta-learning for fast adaptation of deep networks.  , 1126–1135 pages.
  * Harutyunyan et al. (2019) Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. 2019\.  Multitask learning and benchmarking with clinical time series data.  _Scientific data_ 6, 1 (2019), 1–18.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ 9, 8 (Nov. 1997), 1735–1780.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016\.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ 3, 1 (2016), 1–9.
  * Kokkinos (2017) Iasonas Kokkinos. 2017\.  UberNet: Training a ‘Universal’ Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory.
  * Lee and Son (2020) Sungjae Lee and Youngdoo Son. 2020.  Multitask Learning with Single Gradient Step Update for Task Balancing.
  * Lei et al. (2018) Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. 2018.  Simple Recurrent Units for Highly Parallelizable Recurrence.
  * Li et al. (2018) Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. 2018.  Independently recurrent neural network (indrnn): Building A longer and deeper RNN.  , 5457–5466 pages.
  * Liu (2018) S. Liu. 2018.  Exploration on Deep Drug Discovery: Representation and Learning.
  * Liu et al. (2019a) Shikun Liu, Edward Johns, and Andrew J Davison. 2019a.  End-to-end multi-task learning with attention.  , 1871–1880 pages.
  * Liu et al. (2019b) Shengchao Liu, Yingyu Liang, and Anthony Gitter. 2019b.  Loss-balanced task weighting to reduce negative transfer in multi-task learning.  , 9977–9978 pages.
  * Ma et al. (2018) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018\.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  , 1930–1939 pages.
  * Mallya et al. (2018) Arun Mallya, Dillon Davis, and Svetlana Lazebnik. 2018\.  Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights.
  * Misra et al. (2016) Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016.  Cross-stitch networks for multi-task learning.  , 3994–4003 pages.
  * Ruder (2017) Sebastian Ruder. 2017\.  An overview of multi-task learning in deep neural networks.
  * Rumelhart et al. (1986) D. Rumelhart, Geoffrey E. Hinton, and R. J. Williams. 1986\.  Learning internal representations by error propagation.
  * Vandenhende et al. (2020) Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, Dengxin Dai, and Luc Van Gool. 2020\.  Revisiting Multi-Task Learning in the Deep Learning Era.
  * Wang et al. (2017) Yanli Wang, Stephen H Bryant, Tiejun Cheng, Jiyao Wang, Asta Gindulyte, Benjamin A Shoemaker, Paul A Thiessen, Siqian He, and Jian Zhang. 2017.  Pubchem bioassay: 2017 update.  _Nucleic acids research_ 45, D1 (2017), D955–D963.
  * Ye et al. (2019) Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song. 2019\.  Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More.
  * Zhai et al. (2020) Mengyao Zhai, Lei Chen, Jiawei He, Megha Nawhal, Frederick Tung, and Greg Mori. 2020\.  Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation.


* Zinnecker & Yorke (2007) Zinnecker H., Yorke H. W., 2007, ARA&A, 45, 481

##  Appendix A Literature cores and fragmentation analysis

In the section, we present the complete core and homogenised literature core
catalogues used throughout this work, which can be found in full, machine-
readable format online. Moreover, we present the fragmentation structure
determined within each cloud, which has been summarised in Figure 15 of the
main text (section 4.3).

Table 4 presents the observed properties of each core determined from the the
dendrogram analysis (section 3.1), their spectroscopic properties determined
from the N2H+ Gaussian fits (section 3.3), and host millimetre (MM) core
(Rathborne et al., 2006). Table 5 presents the physical properties of the
identified cores (sections 3.2, 3.3). Table 6 presents the literature core
catalogue (Rathborne et al., 2006; Henshaw et al., 2016a, 2017; Liu et al.,
2018b). Here the spatial (effective) radii and masses of each core have been
re-calculated from the observed angular (effective) radii and continuum fluxes
using the same set of assumptions for the dust opacity, dust-to-gas ratio, and
dust temperature (section 4.1).

Table 4: Observational properties of the core population (sections 3.1, 3.3
and 4.3). Shown in columns are the results from the dendrogram analysis of the
core ID and name, the host cloud, the centre RA and Declination, the effective
radius ($R_{\mathrm{eff}}$ in units of arcsec), the total continuum flux
density ($S_{\nu}$) and background subtracted flux density
($S^{\mathrm{b}}_{\nu}$), and the peak continuum intensity ($I_{\nu}$(max)).
Also given are the results of the N2H+ Gaussian fits of peak brightness
temperature ($T_{\mathrm{max}}$), centroid velocity ($\mathrm{v}_{0}$) and
velocity dispersion ($\sigma_{\mathrm{v}}$). We show the millimetre (MM) core
in which each core is contained (Rathborne et al., 2006). Finally, we show if
the core contains an embedded (Spitzer or Herschel 70 µm) infrared point
source, and the flux density of any associated 70 µm point sources
($S_{\mathrm{70\mu m}}$; Molinari et al., 2016; Marton et al., 2017).

ID | Name | Cloud | RA | Dec. | $R_{\mathrm{eff}}$ | $S_{\nu}$ | $S^{\mathrm{b}}_{\nu}$ |  $I_{\nu}$(max) | $T_{\mathrm{max}}$ | $\mathrm{v}_{0}$ | $\sigma_{\mathrm{v}}$ | MM core | SF | $S_{\mathrm{70\mu m}}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  | (J2000) ∘ | (J2000) ∘ | ′′ | $\mathrm{mJy}$ | $\mathrm{mJy}$ | $\mathrm{mJy}$ | $\mathrm{K}$ | $\mathrm{km\,s^{-1}}$ | $\mathrm{km\,s^{-1}}$ | $\mathrm{Jy}$ |  |
1 | A1c1/2 | cloudA | 276.564 | -12.694 | 4.1 | 3.97 | 2.28 | 2.00 | 2.1 | 64.7 | 0.7 | MM4 | y | -
2 | A1c3 | cloudA | 276.566 | -12.692 | 2.3 | 0.84 | 0.34 | 0.72 | 2.0 | 65.7 | 0.3 | MM4 | n | -
3 | A1c4 | cloudA | 276.564 | -12.692 | 1.8 | 0.41 | 0.06 | 0.46 | 1.9 | 65.0 | 0.5 | MM4 | y | -
4 | A1c5 | cloudA | 276.569 | -12.690 | 2.2 | 0.59 | 0.17 | 0.48 | 2.4 | 66.3 | 0.4 | MM4 | y | 3.05
5 | A2c2 | cloudA | 276.580 | -12.688 | 1.9 | 0.45 | 0.17 | 0.58 | 2.4 | 66.3 | 0.4 | MM6 | n | -
… | … | … | … | … | … | … | … | … | … | … | … | … |  |

Note: The full, machine-readable version of this Table can be obtained from
the supplementary online material.

Table 5: Physical properties of the core population (sections 3.2, 3.3). Shown
in columns are the core name (Table 4), the effective radius (in units of
parsec; $R_{\mathrm{eff}}$), the minimum separation or nearest neighbour
distance ($S_{\mathrm{min}}$), the mean dust temperature
($T_{\mathrm{dust}}$), the mass determined using the mean dust temperature
($M$), the background subtracted mass using the mean dust temperature
($M^{\mathrm{b}}$), the mass determined using a constant temperature of 18 K
($M_{\mathrm{18K}}$), the mass determined from the near- and mid- infrared
extinction maps ($M_{\mathrm{ext}}$; Kainulainen & Tan, 2013), the mass
estimates using a temperature determined from 70µm emission ($M_{\mathrm{70\mu
m}}$), the mean density ($n_{\mathrm{H_{2}}}$), the free-fall time
($t_{\mathrm{ff}}$), virial parameter ($\alpha_{\mathrm{vir}}$), background
subtracted virial parameter ($\alpha_{\mathrm{vir}}^{\mathrm{b}}$), the non-
thermal velocity dispersion ($\sigma_{\mathrm{NT}}$), the sonic Mach number
($\mathcal{M}_{\mathrm{s}}$), the Alfvén Mach number
($\mathcal{M}_{\mathrm{A}}$) and magnetic field strength to reach
gravitational equilibrium ($B$; section 3.4.3).

Name | $R_{\mathrm{eff}}$ | $S_{\mathrm{min}}$ | $T_{\mathrm{dust}}$ | $M$ | $M^{\mathrm{b}}$ | $M_{\mathrm{18K}}$ | $M_{\mathrm{ext}}$ | $M_{\mathrm{70\mu m}}$ | $n_{\mathrm{H_{2}}}$ | $t_{\mathrm{ff}}$ | $\alpha_{\mathrm{vir}}$ | $\alpha_{\mathrm{vir}}^{\mathrm{b}}$ | $\sigma_{\mathrm{NT}}$ | $\mathcal{M}_{\mathrm{s}}$ | $\mathcal{M}_{\mathrm{A}}$ | $B$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| $\mathrm{pc}$ | $\mathrm{pc}$ | $\mathrm{K}$ | $\mathrm{M_{\odot}}$ | $\mathrm{M_{\odot}}$ | $\mathrm{M_{\odot}}$ | $\mathrm{M_{\odot}}$ | $\mathrm{M_{\odot}}$ | $\mathrm{10^{5}\,cm^{3}}$ | $\mathrm{10^{4}\,yr}$ |  |  | $\mathrm{km\,s^{-1}}$ |  |  | $\mathrm{\mu G}$
A1c1/2 | 0.095 | 0.2 | 17.6 | 87.6 | 50.2 | 85.3 | 45.0 | - | 3.5 | 5.2 | 0.6 | 1.1 | 0.7 | 2.8 | 6.1 | 665
A1c3 | 0.055 | 0.2 | 17.4 | 18.6 | 7.7 | 17.9 | 14.3 | - | 4.0 | 4.9 | 0.2 | 0.5 | 0.1 | 0.3 | 4.3 | 492
A1c4 | 0.043 | 0.2 | 17.5 | 9.1 | 1.4 | 8.8 | 4.2 | - | 4.1 | 4.8 | 1.3 | 8.3 | 0.4 | 1.7 | 2.1 | 251
A1c5 | 0.051 | 0.3 | 18.0 | 12.6 | 3.5 | 12.6 | - | 8.3 | 3.2 | 5.4 | 0.6 | 2.1 | 0.2 | 1.0 | 3.2 | 335
A2c2 | 0.044 | 0.1 | 17.3 | 10.1 | 3.7 | 9.7 | 9.9 | - | 4.2 | 4.7 | 0.6 | 1.7 | 0.3 | 1.1 | 3.1 | 368
… | … | … | … | … | … | … | … | … | … | … | … | … | … |  |  |

Note: The full, machine-readable version of this Table can be obtained from
the supplementary online material. Were applicable, the online table also
includes all the properties determined with and without background subtraction
for the flux density, and using the measured mean and constant 18 K dust
temperature.

Table 6: Properties of the homogenised literature core sample (section 4.1).
Shown in columns is the host cloud name and ID, the centre RA and Declination,
the effective radius in arcsec and parsec ($R_{\mathrm{eff}}$), the total flux
density at the observed frequency ($S_{\lambda}$), the mass assuming a
constant temperature of 18 K, the wavelength of the observations (see Table
3), and the reference (Rathborne et al., 2006; Henshaw et al., 2016a, 2017;
Liu et al., 2018b).

Cloud | ID | Core | RA | Dec. | $R_{\mathrm{eff}}$ | $R_{\mathrm{eff}}$ | $S_{\lambda}$ | $M_{\mathrm{18K}}$ | $\lambda$ | Reference
---|---|---|---|---|---|---|---|---|---|---
|  |  | ∘ | ∘ | ′′ | $\mathrm{pc}$ | $\mathrm{Jy}$ | $\mathrm{M_{\odot}}$ | $\mathrm{mm}$ |
cloudA | G018.82-00.28 | A3c4 | 276.5900 | -12.6863 | 0.72 | 0.017 | 3.79 | 3.09 | 1.3 | Liu et al. (2018b)
cloudA | G018.82-00.28 | MM5 | 276.5875 | -12.6864 | 15.00 | 0.320 | 234.60 | 144.62 | 1.3 | Rathborne et al. (2006)
cloudA | G018.82-00.28 | A1c3 | 276.5662 | -12.6921 | 2.34 | 0.055 | 0.84 | 17.95 | 3.0 | This work
cloudA | G018.82-00.28 | A2c1 | 276.5777 | -12.6883 | 1.06 | 0.025 | 4.91 | 4.01 | 1.3 | Liu et al. (2018b)
cloudA | G018.82-00.28 | A1c2 | 276.5640 | -12.6937 | 2.26 | 0.052 | 32.38 | 26.43 | 1.3 | Liu et al. (2018b)
cloudA | G018.82-00.28 | A1c5 | 276.5692 | -12.6900 | 2.20 | 0.051 | 0.59 | 12.61 | 3.0 | This work
… | … | … | … | … | … | … | … | … | … |

Note: The full, machine-readable version of this Table can be obtained from
the supplementary online material. This is a literature compilation, please
ensure to cite each individual study when making use of the contents of this
table.

Figure 15: Mass fragmentation of the cores within each IRDC as a function of
the size-scale. The circles, crosses and plus sign markers represent the cores
from the homogenised sample identified by Rathborne et al. (2006), this work
and Liu et al. (2018b) (section 4.1). The straight lines connect the symbols
for each core and the larger, host core of which it is a part (section 4.3.
The lines and symbols have been coloured by host “MM” Rathborne et al. (2006)
core, as indicated in the legend located in the upper left (see Table 6). Also
highlighted with labels are the clumps associated within bright star-forming
regions, which limits the mass determination from the lower resolution
observations from Rathborne et al. (2006). The resultant large uncertainty on
the mass estimate can cause these Rathborne et al. (2006) structures to appear
to have masses smaller than is determined in this work. Their hierarchical
structure should, therefore, be taken with caution. Cloud G is not shown due
to the lack of continuum cores determined in the 3 mm ALMA continuum
observations presented as part of this work (section 3.1). The shaded region
shows the 8 ⪝ mass threshold for a high-mass star. In the lower right corner
of the upper left panel, we show a representative uncertainty range $\sim$ 15
and $\sim$ 50 per cent on the radius and mass, respectively.

##  Appendix B Comparison of feathering and uv-combination

The combination of multiple datasets is typical for interferometric
observations, where the range of recoverable spatial scales is limited by the
baselines included in each observation. There is still, however, much debate
within the literature for the best practises for this combination, and
particularly for the case where single-dish observations are used with
interferometric datasets to recover the zero-spacing. Throughout this work, we
make use of ALMA observations taken with both the 12m and 7m arrays, which
were reduced using the casa-pipeline (version: 5.4.0-70) and combined using
the feather (CASA version: 4.7.0). For the follow-up work investigating the
nitrogen fractionation, we have also combined the single-dish observations
with the 12m and 7m images using the same procedure (Fontani et al., 2021). We
chose to feather the images produced directly by the pipeline for convenience.
However, an alternative, commonly used approach, is to combine the array
configurations in the uv-plane and to then image them together using the e.g.
clean function.

In light of the above, in this section, we conduct a comparison between the
continuum map and cube within Cloud C produced by combining the 12 m and 7 m
datasets in the feather and tclean functions. Firstly, for the continuum, we
use the calibrated measurement sets produced by the pipeline that have not
been continuum subtracted and use the line-free parts of the bandwidth
identified from the hif$\\_$findcont task in the casa-pipeline. When imaging
with tclean (version: 5.6.0), we use natural weighting, a multi-term
(multiscale) multi-frequency synthesis deconvolver (mtmfs option), and set a
high number of iterations to achieve a noise threshold of 0.5 mJy per beam
within a mask that is automatically determined after each minor cycle (see the
auto-multithresh option for the usemask parameter in tclean; Kepley et al.,
2020). Second, for the cubes, we make use of the continuum subtracted
measurement set produced by the pipeline. When imaging, we again use natural
weighting, a multiscale deconvolver (multiscale option), set a high number of
interactions to achieve a noise threshold of 21 mJy per beam per channel
within an automatically determined mask.

Figure 16 shows the feathered and uv-combined, cleaned continuum maps and
integrated intensity maps for Cloud C, where the integrated intensity has been
determined using the same mask and velocity range for both cubes. Here we
match the colour bar scales for both maps for ease of comparison, and overlay
contours at signal-to-noise levels of 3 and 5$\sigma$ (see table 2). We see
here that both qualitatively and quantitatively the maps produced with the
feather and tclean functions are very similar, and only minor differences can
be seen on close inspection. To further quantify this, in Figure 16 we also
present maps of the absolute difference between the feathered and cleaned
images. We find absolute differences of up to $10$ per cent between the two
methods; note the colour scale range used to show the difference maps is 1/10
of the maximum of the cleaned and feathered maps. This difference is, however,
small compared to the underlying systematic uncertainties inherent in the
physical properties that these maps are used to calculate within this work
(e.g. for the mass, where the combined uncertainty from the temperature and
distance will be factors of a few higher).

Along with examining the two-dimensional distributions for , we can also make
use of the cubes to compare the feather and tclean 12m and 7m combination for
each velocity slice. Figure 17 shows the spectra of the isolated component of
averaged over the core C2c1, which is the closest core to the largest
difference in the integrated intensity maps (Figure 16). These spectra both
show profiles that contain only two Gaussian profiles, which are separated by
$\sim$2 . However, we see that the feathered spectrum has systematically lower
intensities than the cleaned spectrum, which is highlighted by the difference
profile. Where present, we find that this difference ranges from $20-50$ per
cent of the feathered spectrum intensity, and is, therefore, larger than
observed within the integrated intensity map. That said, it is worth keeping
in mind that in this work we only use the line-width for the dynamical
analysis of the cores, which will be less sensitive to systematic differences
within the intensity.

In summary, in this section, we find that there are some differences between
the map produced when feathering the pipeline imaged 12 m and 7 m datasets,
and when imaging the 12 m and 7 m datasets together in the clean function.
These differences are of the order $10$ per cent, but can be more substantial
when inspecting the individual slices of a datacube. However, ultimately, the
differences between the two methods are small compared to the systematic
uncertainty inherent within the physical properties calculated within this
work. The analysis presented here then validates our choice of using the
feathered, pipeline reduced images, throughout this work.

Figure 16: A comparison between the images produced when feathering the
pipeline imaged 12 m and 7 m datasets, and when imaging the 12 m and 7 m
datasets together in the uv-plane with the clean function for Cloud C (see
Figure 2). The left two panels show the feathered, and uv-combined, cleaned
continuum maps, overlaid with a solid black contour of the 0.48 mJy beam-1.
The centre-left shows a map of the difference between these two continuum
images. The fourth and fifth panels show the feathered and cleaned integrated
intensity maps. In both cases the integrated intensity has been determined
over the isolated hyperfine component of , using the same mask and velocity
range. Overlaid are solid black contours at signal-to-noise levels of 3 and
5$\sigma$. The rightmost panel shows the difference between these integrated
intensity maps. Note that the feathered and cleaned continuum and integrated
intensity maps have the same colour scale, and the respective difference map
has a range of 1/10 this colour scale. Figure 17: A comparison between the
cubes produced when feathering the pipeline imaged 12 m and 7 m datasets, and
when imaging the 12 m and 7 m datasets together in the uv-plane with the clean
function for Cloud C (see Figure 2). Shown in blue and red are mean spectra of
the isolated hyperfine component of across the core C2c1. Shown in black and
centred on -0.15 K is the difference between these two spectra.


# Improving agnostic searches of Gravitational Waves from Neutron Star
instabilities using image filtering

Lorenzo Pierini  Dipartimento di Fisica, Università degli Studi di Roma La
Sapienza,
Istituto Nazionale di Fisica Nucleare, Sezione Roma 1,
Piazzale Aldo Moro 2, 00185, Rome, Italy
email<EMAIL_ADDRESS>
(2021)

###### Abstract

In this paper I present a method to enhance the search sensitivity for long
transient Gravitational Waves produced by Neutron Star instabilities. This
method consists in a selective image filter, called Triangular Filter, to be
applied to data spectrograms. It is shown that thanks to this implementation a
$\sim 20\%$ gain in sensitivity is achievable.

###### keywords:

Gravitational Waves, Neutron Stars, Data Analysis, Image Filtering

††volume: 363††journal: Neutron Star Astrophysics at the Crossroads:
Magnetars and the Multimessenger Revolution††editors: E. Troja & M. Baring,
eds.

##  1 Introduction

The search of Gravitational Waves (GW) emitted by isolated or binary Neutron
Stars (NS) is one of the biggest challenges in GW astronomy. In this work I
focus on transient GWs - with typical timescales of $\mathcal{O}$($10^{3}$)s -
produced by NS instabilities, see [Lasky (2015)] for a review. Such
instabilities can develop in young supernova remnants or in the post-merger
remnant of binary NS inspirals, for which an electromagnetic counterpart is
expected. We can then assume to know the source position fairly well, while
rotational parameters are largely unknown. The resulting huge parameter space
makes matched filter searches computationally unfeasible and different
strategies have to be used. One example is the Generalized Frequency Hough
(GFH), described in [Miller (2018)] and used the first time in [Abbott
(2019)]. This procedure works on spectrograms of the detectors data: it
searches for time-frequency patterns of the expected GW signal, assumed to
follow a power law equation. This is done transforming data into the signal
parameter space though the GFH transform. This procedure is computationally
feasible and is also more robust with respect to deviations from theoretical
models, but it pays a price in terms of sensitivity loss with respect to
matched filter techniques. The goal of my work is to reduce this sensitivity
gap.

##  2 Method: Triangular Filter on spectrograms

The method I developed works directly on the spectrograms, where we search for
GW time-frequency patterns, and consists of an image filter to enhance the
signal-to-noise ratio (SNR). This filter is applied in the 2D Fourier domain,
and it is suited to filter out signals with decreasing frequency patterns. Its
computational cost is negligible, since it is applied just one time before
appying the multiple GFH transforms. It is called Triangular Filter because of
its shape in the 2D Fourier space. A decreasing time-frequency pattern is
characterized by negative slope tangents: their information is stored in a
well defined region of the 2D Fourier space, so the filter selects only the
regions of the Fourier Transform of the spectrogram that correspond to signals
slowing down within a given range. A visual proof of the filtering effect is
shown in Figure 1, where the color scale reveals the enhanced SNR after the
filter application. Repeating the simulation with randomly localized sources
and varying signal amplitudes, it is possible to compute the efficiency curves
in the standard case of GFH and after applying the Triangular Filter. If we
set a fiduciary efficiency level at $95\%$, it turns out that we can obtain a
sensitivity gain of $\sim 20\%$.

Figure 1: Effect of the Triangular Filter application on spectrograms. On the
left: whitened spectrogram of LIGO Livingston O2 data containing a simulated
long transient GW starting at 750 Hz. The signal is visible but its energy
does not exceed the mean noise level. On the right: the same map after
filtering. Here the signal has become an outlier with respect to the noise.
Figure 2: Efficiency curves obtained through multiple injections of simulated
GW r-mode signals - see [Lasky (2015)] \- on LIGO Livingston O2 data starting
from GPS=1253179440s and with initial frequency in the range [700-770]Hz. The
blue line represents the curve obtained with the ordinary GFH procedure,
whereas the green one is obtained using the Triangular Filter.

##  3 Implications and future developments

The results obtained show clearly the sensitivity gain that can be achieved
thanks to the Triangular Filter technique. The same filter can be implemented
to search for signals with growing or wandering frequency, just restricting to
frequency derivatives not greater than the optimal resolution of the
spectrogram. In the future, this filter will be implemented as a crucial step
of a new search procedure, in order to enhance the detection probability.
Another interesting search topic will be to evaluate the impact of this filter
when it is applied as a pre-processing step on deep learning based methods.

## References

  * [Lasky (2015)] Lasky, Paul D. 2015, PASA, 32, E034
  * [Miller (2018)] Miller, A., et al. 2018, Phys. Rev. D, 98.10: 102004
  * [Abbott (2019)] Abbott, B.P., et al. 2019, ApJ, 875 160